Fix integer overflow, implicit integer sign change and truncation

Fix integer overflow in cabac psy-trellis and cavcl trellis.

And other implicit integer sign change and truncation reported by clang's UBSan: -fsanitize=signed-integer-overflow,implicit-integer-sign-change,implicit-signed-integer-truncation

Continuation of !79 (merged)