[3.0] lua: don't use | operator on VLC errors

This could end up with unknown/invalid error values.

(cherry picked from commit 586b28f4)