AArch64: Add DotProd support for convolutions
Add an Armv8.4-A DotProd code path for standard bitdepth convolutions. Only horizontal-vertical (HV) convolutions have 6-tap specialisations of their vertical passes. All other convolutions are 4- or 8-tap filters which fit well with the 4-element SDOT instruction.
Benchmarks show up-to 7-29% FPS increase depending on the input video and the CPU used.
This patch will increase the .text by around 6.5 KiB.
Performance highly depends on the SDOT and MLA throughput ratio, this can be seen on the vertical filter cases. Small cores are also affected by the TBL execution latencies.
Some benchmark results:
- AWS Graviton 2: 93.02 fps -> 109.09 fps ( +17.3 % )
- AWS Graviton 3: 160.42 fps -> 166.74 fps ( +3.94 % ) - AWS Graviton 2: 88.24 fps -> 101.35 fps ( +14.8 % )
- AWS Graviton 3: 146.79 fps -> 155.11 fps ( +5.67 % ) - AWS Graviton 2: 78.28 fps -> 87.65 fps ( +12.0 % )
- AWS Graviton 3: 123.37 fps -> 126.45 fps ( +2.50 % ) - AWS Graviton 2: 76.13 fps -> 86.69 fps ( +13.9 % )
- AWS Graviton 3: 118.54 fps -> 124.45 fps ( +4.99 % ) - AWS Graviton 2: 104.68 fps -> 116.39 fps ( +11.2 % )
- AWS Graviton 3: 172.47 fps -> 179.72 fps ( +4.20 % ) - AWS Graviton 2: 98.27 fps -> 127.10 fps ( +29.3 % )
- AWS Graviton 3: 183.91 fps -> 198.03 fps ( +7.67 % )Bosphorus 1080p was encoded by aomenc (3.7.1+):
aomenc --good --cpu-used=5 -w 1920 -h 1080 --bit-depth=8 --ivf -o Bosphorus_1080p_8bit.ivf Bosphorus_1920x1080_120fps_420_8bit_YUV.y4mRelative performance to the C reference on some CPUs:
Horizontal-vertical micro benchmarks
A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc
regular_w2_hv_8bpc_neon: 3.89x 6.39x 5.36x
regular_w2_hv_8bpc_dotprod: 3.95x 7.53x 4.46x
sharp_w2_hv_8bpc_neon: 3.46x 5.83x 5.20x
sharp_w2_hv_8bpc_dotprod: 3.06x 6.06x 3.81x
regular_w4_hv_8bpc_neon: 5.52x 4.21x 5.78x 5.06x 10.75x 8.93x 8.27x 8.24x
regular_w4_hv_8bpc_dotprod: 7.94x 4.57x 8.49x 7.53x 16.84x 14.10x 8.09x 7.88x
sharp_w4_hv_8bpc_neon: 5.27x 4.12x 5.22x 4.72x 9.06x 7.58x 7.87x 7.78x
sharp_w4_hv_8bpc_dotprod: 6.61x 3.91x 6.73x 6.36x 12.64x 11.27x 6.89x 6.72x
regular_w8_hv_8bpc_neon: 1.95x 5.18x 2.19x 2.43x 2.56x 2.69x 3.16x 3.23x
regular_w8_hv_8bpc_dotprod: 3.23x 7.86x 2.81x 3.10x 3.20x 3.61x 3.26x 3.31x
sharp_w8_hv_8bpc_neon: 1.61x 4.87x 1.79x 1.93x 2.05x 2.17x 2.72x 2.77x
sharp_w8_hv_8bpc_dotprod: 2.72x 6.58x 2.29x 2.56x 2.66x 2.96x 2.76x 2.74x
regular_w16_hv_8bpc_neon: 1.63x 1.99x 2.04x 2.27x 2.16x 2.21x 2.73x 2.67x
regular_w16_hv_8bpc_dotprod: 2.72x 3.38x 2.57x 2.84x 2.67x 2.92x 2.80x 2.76x
sharp_w16_hv_8bpc_neon: 1.33x 1.65x 1.67x 1.79x 1.74x 1.78x 2.34x 2.31x
sharp_w16_hv_8bpc_dotprod: 2.31x 2.86x 2.14x 2.36x 2.26x 2.44x 2.39x 2.31x
regular_w32_hv_8bpc_neon: 1.48x 1.66x 1.92x 2.14x 1.94x 2.00x 2.51x 2.48x
regular_w32_hv_8bpc_dotprod: 2.49x 2.86x 2.40x 2.64x 2.33x 2.59x 2.58x 2.54x
sharp_w32_hv_8bpc_neon: 1.21x 1.37x 1.56x 1.68x 1.53x 1.59x 2.14x 2.13x
sharp_w32_hv_8bpc_dotprod: 2.12x 2.43x 2.02x 2.21x 2.00x 2.18x 2.22x 2.14x
regular_w64_hv_8bpc_neon: 1.42x 1.54x 1.87x 2.06x 1.85x 1.87x 2.40x 2.36x
regular_w64_hv_8bpc_dotprod: 2.40x 2.69x 2.32x 2.53x 2.21x 2.41x 2.46x 2.43x
sharp_w64_hv_8bpc_neon: 1.16x 1.28x 1.52x 1.61x 1.46x 1.48x 2.04x 2.03x
sharp_w64_hv_8bpc_dotprod: 2.02x 2.28x 1.96x 2.13x 1.90x 2.04x 2.11x 2.06x
regular_w128_hv_8bpc_neon: 1.39x 1.46x 1.84x 2.04x 1.80x 1.83x 2.27x 2.27x
regular_w128_hv_8bpc_dotprod: 2.33x 2.56x 2.28x 2.49x 2.14x 2.33x 2.35x 2.33x
sharp_w128_hv_8bpc_neon: 1.14x 1.22x 1.50x 1.59x 1.42x 1.44x 1.94x 1.96x
sharp_w128_hv_8bpc_dotprod: 1.98x 2.19x 1.93x 2.10x 1.84x 1.98x 2.03x 1.99x
Horizontal micro benchmarks
A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc
regular_w2_h_8bpc_neon: 3.31x 3.64x 5.90x 5.04x
regular_w2_h_8bpc_dotprod: 6.05x 8.83x 16.68x 5.15x
sharp_w2_h_8bpc_neon: 3.33x 3.67x 5.73x 5.04x
sharp_w2_h_8bpc_dotprod: 6.09x 8.83x 16.61x 5.14x
regular_w4_h_8bpc_neon: 5.43x 5.17x 7.05x 4.72x 10.09x 7.73x 8.06x 8.00x
regular_w4_h_8bpc_dotprod: 11.20x 11.30x N/A N/A N/A N/A 11.43x 9.05x
sharp_w4_h_8bpc_neon: 5.33x 5.21x 7.10x 4.79x 10.03x 7.76x 8.02x 8.00x
sharp_w4_h_8bpc_dotprod: 10.99x 11.19x N/A N/A N/A N/A 11.64x 9.06x
regular_w8_h_8bpc_neon: 2.61x 2.71x 3.20x 3.06x 3.51x 3.67x 3.55x 3.25x
regular_w8_h_8bpc_dotprod: 4.43x 4.61x 5.17x 5.00x 6.26x 6.31x 4.30x 4.05x
sharp_w8_h_8bpc_neon: 2.01x 2.17x 2.80x 2.71x 2.89x 3.04x 3.12x 2.92x
sharp_w8_h_8bpc_dotprod: 4.42x 4.59x 5.16x 5.08x 6.27x 6.21x 4.28x 4.01x
regular_w16_h_8bpc_neon: 2.17x 1.93x 3.13x 2.92x 2.92x 2.80x 3.35x 3.34x
regular_w16_h_8bpc_dotprod: 4.38x 4.51x 4.27x 4.27x 4.53x 4.63x 3.90x 3.74x
sharp_w16_h_8bpc_neon: 1.74x 1.66x 2.65x 2.52x 2.48x 2.31x 2.92x 2.90x
sharp_w16_h_8bpc_dotprod: 4.33x 4.53x 4.27x 4.29x 4.53x 4.65x 3.91x 3.73x
regular_w32_h_8bpc_neon: 2.01x 2.03x 2.91x 2.78x 2.56x 2.49x 3.07x 3.12x
regular_w32_h_8bpc_dotprod: 3.85x 3.94x 3.79x 3.74x 3.67x 3.79x 3.35x 3.31x
sharp_w32_h_8bpc_neon: 1.56x 1.60x 2.43x 2.38x 2.18x 2.08x 2.66x 2.70x
sharp_w32_h_8bpc_dotprod: 3.85x 3.92x 3.78x 3.72x 3.65x 3.78x 3.35x 3.31x
regular_w64_h_8bpc_neon: 1.92x 1.97x 2.82x 2.68x 2.39x 2.35x 2.96x 3.00x
regular_w64_h_8bpc_dotprod: 3.68x 3.83x 3.60x 3.54x 3.40x 3.51x 3.18x 3.19x
sharp_w64_h_8bpc_neon: 1.47x 1.53x 2.33x 2.22x 2.05x 1.99x 2.54x 2.59x
sharp_w64_h_8bpc_dotprod: 3.68x 3.82x 3.60x 3.52x 3.40x 3.50x 3.17x 3.19x
regular_w128_h_8bpc_neon: 1.86x 1.94x 2.78x 2.63x 2.31x 2.28x 2.85x 2.91x
regular_w128_h_8bpc_dotprod: 3.59x 3.74x 3.50x 3.47x 3.30x 3.39x 3.08x 3.11x
sharp_w128_h_8bpc_neon: 1.41x 1.49x 2.27x 2.20x 1.99x 1.93x 2.46x 2.52x
sharp_w128_h_8bpc_dotprod: 3.59x 3.74x 3.50x 3.46x 3.29x 3.39x 3.09x 3.12x
Vertical micro benchmarks
A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc
regular_w2_v_8bpc_neon: 3.64x 4.09x 5.68x 7.05x
regular_w2_v_8bpc_dotprod: 4.86x 6.02x 7.30x 4.94x
sharp_w2_v_8bpc_neon: 3.45x 3.77x 5.03x 6.84x
sharp_w2_v_8bpc_dotprod: 4.88x 6.04x 7.42x 4.93x
regular_w4_v_8bpc_neon: 5.39x 5.60x 7.38x 6.77x 10.27x 10.60x 11.41x 12.09x
regular_w4_v_8bpc_dotprod: 9.46x 8.61x 14.15x 10.23x 18.72x 13.50x 9.84x 8.16x
sharp_w4_v_8bpc_neon: 4.51x 4.62x 6.39x 5.74x 8.17x 8.39x 10.70x 11.34x
sharp_w4_v_8bpc_dotprod: 9.35x 8.20x 14.20x 10.26x 18.63x 13.52x 9.78x 8.15x
regular_w8_v_8bpc_neon: 3.77x 3.63x 6.07x 6.15x 8.18x 8.24x 8.16x 7.74x
regular_w8_v_8bpc_dotprod: 5.70x 5.22x 5.39x 5.04x 7.91x 7.43x 5.04x 4.69x
sharp_w8_v_8bpc_neon: 2.90x 2.79x 4.56x 4.65x 5.92x 5.98x 7.26x 6.87x
sharp_w8_v_8bpc_dotprod: 5.64x 5.17x 5.40x 5.02x 7.85x 7.53x 5.04x 4.58x
regular_w16_v_8bpc_neon: 3.03x 2.93x 4.03x 4.27x 4.65x 4.75x 6.28x 5.81x
regular_w16_v_8bpc_dotprod: 4.64x 4.38x 3.75x 3.77x 4.78x 4.82x 3.89x 3.75x
sharp_w16_v_8bpc_neon: 2.29x 2.22x 3.09x 3.30x 3.44x 3.54x 5.52x 5.11x
sharp_w16_v_8bpc_dotprod: 4.62x 4.40x 3.74x 3.77x 4.77x 4.89x 3.89x 3.73x
regular_w32_v_8bpc_neon: 2.46x 2.41x 3.36x 3.50x 3.58x 3.79x 5.11x 4.86x
regular_w32_v_8bpc_dotprod: 3.87x 3.74x 3.20x 3.16x 3.71x 3.93x 3.13x 3.11x
sharp_w32_v_8bpc_neon: 1.84x 1.81x 2.57x 2.69x 2.63x 2.81x 4.44x 4.25x
sharp_w32_v_8bpc_dotprod: 3.86x 3.74x 3.18x 3.15x 3.68x 3.94x 3.13x 3.10x
regular_w64_v_8bpc_neon: 2.17x 2.17x 3.14x 3.29x 3.19x 3.45x 4.46x 4.46x
regular_w64_v_8bpc_dotprod: 3.43x 3.41x 3.00x 2.98x 3.31x 3.58x 2.74x 2.79x
sharp_w64_v_8bpc_neon: 1.61x 1.63x 2.42x 2.53x 2.34x 2.54x 3.89x 3.89x
sharp_w64_v_8bpc_dotprod: 3.38x 3.41x 3.00x 2.98x 3.29x 3.58x 2.73x 2.79x
regular_w128_v_8bpc_neon: 1.99x 2.08x 3.03x 3.19x 3.00x 3.27x 3.93x 4.13x
regular_w128_v_8bpc_dotprod: 3.15x 3.30x 2.90x 2.90x 3.11x 3.39x 2.48x 2.59x
sharp_w128_v_8bpc_neon: 1.49x 1.54x 2.34x 2.46x 2.21x 2.41x 3.46x 3.61x
sharp_w128_v_8bpc_dotprod: 3.15x 3.26x 2.90x 2.91x 3.11x 3.39x 2.49x 2.60x
Copy/widening-copy micro benchmarks (will be a different MR)
A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc
regular_w2_0_8bpc_neon: 6.36x 5.13x
regular_w2_0_8bpc_dotprod: 7.74x 5.87x
regular_w4_0_8bpc_neon: 4.87x 5.60x 5.88x 5.53x
regular_w4_0_8bpc_dotprod: 5.89x 9.08x 9.78x 6.50x
regular_w8_0_8bpc_neon: 4.71x 6.94x 3.45x 4.71x
regular_w8_0_8bpc_dotprod: 5.37x 7.45x 4.16x 5.22x
regular_w16_0_8bpc_neon: 2.41x 5.23x 3.27x 6.32x 6.08x 7.60x 2.17x 3.33x
regular_w16_0_8bpc_dotprod: 2.61x 5.86x 3.41x 6.83x 5.51x 8.49x 2.38x 4.02x
regular_w32_0_8bpc_neon: 2.16x 3.35x 1.45x 1.87x 2.21x 1.90x 1.64x 2.08x
regular_w32_0_8bpc_dotprod: 2.51x 3.73x 1.45x 2.15x 2.29x 2.35x 2.04x 2.72x
regular_w64_0_8bpc_neon: 1.73x 2.02x 1.06x 1.11x 1.68x 1.03x 1.71x 1.43x
regular_w64_0_8bpc_dotprod: 1.46x 2.22x 1.05x 1.10x 1.87x 1.00x 1.67x 1.71x
regular_w128_0_8bpc_neon: 1.28x 1.37x 1.02x 1.09x 1.33x 1.05x 1.57x 1.41x
regular_w128_0_8bpc_dotprod: 1.15x 1.36x 1.02x 1.09x 1.40x 1.07x 1.57x 1.42x