arm: cdef: Special case for pri/sec taps being zero

This gives a 15-25% speedup on the checkasm numbers, full numbers in the commit message.