libmp4: explicitly use negative value for unsigned return value

This is the same format used to check the returned value.

(cherry picked from commit 1bdc69ccf75dd117ccba3e7cbad514666e336242)