Skip to content
Snippets Groups Projects

x86: Fix overflows in 12bpc AVX2 identity itx asm

Merged Henrik Gramner requested to merge gramner/dav1d:identity_12bpc_overflow into master

12bpc has 20-bit coefficients, which can't be multiplied by factors larger than 4096 without int32_t overflows. The exception is for 1:2 and 2:1 sizes, which can handle multiplies up to 5793 after the initial sqrt(2) scaling.

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply
Loading