Skip to content

AArch64: Add DotProd support for convolutions

Arpad Panyik requested to merge arpadpanyik-arm/dav1d:mc_sbd_dotprod into master

Add an Armv8.4-A DotProd code path for standard bitdepth convolutions. Only horizontal-vertical (HV) convolutions have 6-tap specialisations of their vertical passes. All other convolutions are 4- or 8-tap filters which fit well with the 4-element SDOT instruction.

Benchmarks show up-to 7-29% FPS increase depending on the input video and the CPU used.

This patch will increase the .text by around 6.5 KiB.

Performance highly depends on the SDOT and MLA throughput ratio, this can be seen on the vertical filter cases. Small cores are also affected by the TBL execution latencies.


Some benchmark results:

Models 1080p:

 - AWS Graviton 2:   93.02 fps  ->  109.09 fps ( +17.3 % )
 - AWS Graviton 3:  160.42 fps  ->  166.74 fps ( +3.94 % )

Balloons 1080p:

 - AWS Graviton 2:   88.24 fps  ->  101.35 fps ( +14.8 % )
 - AWS Graviton 3:  146.79 fps  ->  155.11 fps ( +5.67 % )

Mountain Bike 1080p:

 - AWS Graviton 2:   78.28 fps  ->   87.65 fps ( +12.0 % )
 - AWS Graviton 3:  123.37 fps  ->  126.45 fps ( +2.50 % )

Nature 1080p:

 - AWS Graviton 2:   76.13 fps  ->   86.69 fps ( +13.9 % )
 - AWS Graviton 3:  118.54 fps  ->  124.45 fps ( +4.99 % )

Vision Pro 1080p:

 - AWS Graviton 2:  104.68 fps  ->  116.39 fps ( +11.2 % )
 - AWS Graviton 3:  172.47 fps  ->  179.72 fps ( +4.20 % )

Bosphorus 1080p:

 - AWS Graviton 2:   98.27 fps  ->  127.10 fps ( +29.3 % )
 - AWS Graviton 3:  183.91 fps  ->  198.03 fps ( +7.67 % )

Bosphorus 1080p was encoded by aomenc (3.7.1+):

aomenc --good --cpu-used=5 -w 1920 -h 1080 --bit-depth=8 --ivf -o Bosphorus_1080p_8bit.ivf Bosphorus_1920x1080_120fps_420_8bit_YUV.y4m

Relative performance to the C reference on some CPUs:

Horizontal-vertical micro benchmarks A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc

regular_w2_hv_8bpc_neon: 3.89x 6.39x 5.36x regular_w2_hv_8bpc_dotprod: 3.95x 7.53x 4.46x

sharp_w2_hv_8bpc_neon: 3.46x 5.83x 5.20x sharp_w2_hv_8bpc_dotprod: 3.06x 6.06x 3.81x

regular_w4_hv_8bpc_neon: 5.52x 4.21x 5.78x 5.06x 10.75x 8.93x 8.27x 8.24x regular_w4_hv_8bpc_dotprod: 7.94x 4.57x 8.49x 7.53x 16.84x 14.10x 8.09x 7.88x

sharp_w4_hv_8bpc_neon: 5.27x 4.12x 5.22x 4.72x 9.06x 7.58x 7.87x 7.78x sharp_w4_hv_8bpc_dotprod: 6.61x 3.91x 6.73x 6.36x 12.64x 11.27x 6.89x 6.72x

regular_w8_hv_8bpc_neon: 1.95x 5.18x 2.19x 2.43x 2.56x 2.69x 3.16x 3.23x regular_w8_hv_8bpc_dotprod: 3.23x 7.86x 2.81x 3.10x 3.20x 3.61x 3.26x 3.31x

sharp_w8_hv_8bpc_neon: 1.61x 4.87x 1.79x 1.93x 2.05x 2.17x 2.72x 2.77x sharp_w8_hv_8bpc_dotprod: 2.72x 6.58x 2.29x 2.56x 2.66x 2.96x 2.76x 2.74x

regular_w16_hv_8bpc_neon: 1.63x 1.99x 2.04x 2.27x 2.16x 2.21x 2.73x 2.67x regular_w16_hv_8bpc_dotprod: 2.72x 3.38x 2.57x 2.84x 2.67x 2.92x 2.80x 2.76x

sharp_w16_hv_8bpc_neon: 1.33x 1.65x 1.67x 1.79x 1.74x 1.78x 2.34x 2.31x sharp_w16_hv_8bpc_dotprod: 2.31x 2.86x 2.14x 2.36x 2.26x 2.44x 2.39x 2.31x

regular_w32_hv_8bpc_neon: 1.48x 1.66x 1.92x 2.14x 1.94x 2.00x 2.51x 2.48x regular_w32_hv_8bpc_dotprod: 2.49x 2.86x 2.40x 2.64x 2.33x 2.59x 2.58x 2.54x

sharp_w32_hv_8bpc_neon: 1.21x 1.37x 1.56x 1.68x 1.53x 1.59x 2.14x 2.13x sharp_w32_hv_8bpc_dotprod: 2.12x 2.43x 2.02x 2.21x 2.00x 2.18x 2.22x 2.14x

regular_w64_hv_8bpc_neon: 1.42x 1.54x 1.87x 2.06x 1.85x 1.87x 2.40x 2.36x regular_w64_hv_8bpc_dotprod: 2.40x 2.69x 2.32x 2.53x 2.21x 2.41x 2.46x 2.43x

sharp_w64_hv_8bpc_neon: 1.16x 1.28x 1.52x 1.61x 1.46x 1.48x 2.04x 2.03x sharp_w64_hv_8bpc_dotprod: 2.02x 2.28x 1.96x 2.13x 1.90x 2.04x 2.11x 2.06x

regular_w128_hv_8bpc_neon: 1.39x 1.46x 1.84x 2.04x 1.80x 1.83x 2.27x 2.27x regular_w128_hv_8bpc_dotprod: 2.33x 2.56x 2.28x 2.49x 2.14x 2.33x 2.35x 2.33x

sharp_w128_hv_8bpc_neon: 1.14x 1.22x 1.50x 1.59x 1.42x 1.44x 1.94x 1.96x sharp_w128_hv_8bpc_dotprod: 1.98x 2.19x 1.93x 2.10x 1.84x 1.98x 2.03x 1.99x

Horizontal micro benchmarks A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc

regular_w2_h_8bpc_neon: 3.31x 3.64x 5.90x 5.04x regular_w2_h_8bpc_dotprod: 6.05x 8.83x 16.68x 5.15x

sharp_w2_h_8bpc_neon: 3.33x 3.67x 5.73x 5.04x sharp_w2_h_8bpc_dotprod: 6.09x 8.83x 16.61x 5.14x

regular_w4_h_8bpc_neon: 5.43x 5.17x 7.05x 4.72x 10.09x 7.73x 8.06x 8.00x regular_w4_h_8bpc_dotprod: 11.20x 11.30x N/A N/A N/A N/A 11.43x 9.05x

sharp_w4_h_8bpc_neon: 5.33x 5.21x 7.10x 4.79x 10.03x 7.76x 8.02x 8.00x sharp_w4_h_8bpc_dotprod: 10.99x 11.19x N/A N/A N/A N/A 11.64x 9.06x

regular_w8_h_8bpc_neon: 2.61x 2.71x 3.20x 3.06x 3.51x 3.67x 3.55x 3.25x regular_w8_h_8bpc_dotprod: 4.43x 4.61x 5.17x 5.00x 6.26x 6.31x 4.30x 4.05x

sharp_w8_h_8bpc_neon: 2.01x 2.17x 2.80x 2.71x 2.89x 3.04x 3.12x 2.92x sharp_w8_h_8bpc_dotprod: 4.42x 4.59x 5.16x 5.08x 6.27x 6.21x 4.28x 4.01x

regular_w16_h_8bpc_neon: 2.17x 1.93x 3.13x 2.92x 2.92x 2.80x 3.35x 3.34x regular_w16_h_8bpc_dotprod: 4.38x 4.51x 4.27x 4.27x 4.53x 4.63x 3.90x 3.74x

sharp_w16_h_8bpc_neon: 1.74x 1.66x 2.65x 2.52x 2.48x 2.31x 2.92x 2.90x sharp_w16_h_8bpc_dotprod: 4.33x 4.53x 4.27x 4.29x 4.53x 4.65x 3.91x 3.73x

regular_w32_h_8bpc_neon: 2.01x 2.03x 2.91x 2.78x 2.56x 2.49x 3.07x 3.12x regular_w32_h_8bpc_dotprod: 3.85x 3.94x 3.79x 3.74x 3.67x 3.79x 3.35x 3.31x

sharp_w32_h_8bpc_neon: 1.56x 1.60x 2.43x 2.38x 2.18x 2.08x 2.66x 2.70x sharp_w32_h_8bpc_dotprod: 3.85x 3.92x 3.78x 3.72x 3.65x 3.78x 3.35x 3.31x

regular_w64_h_8bpc_neon: 1.92x 1.97x 2.82x 2.68x 2.39x 2.35x 2.96x 3.00x regular_w64_h_8bpc_dotprod: 3.68x 3.83x 3.60x 3.54x 3.40x 3.51x 3.18x 3.19x

sharp_w64_h_8bpc_neon: 1.47x 1.53x 2.33x 2.22x 2.05x 1.99x 2.54x 2.59x sharp_w64_h_8bpc_dotprod: 3.68x 3.82x 3.60x 3.52x 3.40x 3.50x 3.17x 3.19x

regular_w128_h_8bpc_neon: 1.86x 1.94x 2.78x 2.63x 2.31x 2.28x 2.85x 2.91x regular_w128_h_8bpc_dotprod: 3.59x 3.74x 3.50x 3.47x 3.30x 3.39x 3.08x 3.11x

sharp_w128_h_8bpc_neon: 1.41x 1.49x 2.27x 2.20x 1.99x 1.93x 2.46x 2.52x sharp_w128_h_8bpc_dotprod: 3.59x 3.74x 3.50x 3.46x 3.29x 3.39x 3.09x 3.12x

Vertical micro benchmarks A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc

regular_w2_v_8bpc_neon: 3.64x 4.09x 5.68x 7.05x regular_w2_v_8bpc_dotprod: 4.86x 6.02x 7.30x 4.94x

sharp_w2_v_8bpc_neon: 3.45x 3.77x 5.03x 6.84x sharp_w2_v_8bpc_dotprod: 4.88x 6.04x 7.42x 4.93x

regular_w4_v_8bpc_neon: 5.39x 5.60x 7.38x 6.77x 10.27x 10.60x 11.41x 12.09x regular_w4_v_8bpc_dotprod: 9.46x 8.61x 14.15x 10.23x 18.72x 13.50x 9.84x 8.16x

sharp_w4_v_8bpc_neon: 4.51x 4.62x 6.39x 5.74x 8.17x 8.39x 10.70x 11.34x sharp_w4_v_8bpc_dotprod: 9.35x 8.20x 14.20x 10.26x 18.63x 13.52x 9.78x 8.15x

regular_w8_v_8bpc_neon: 3.77x 3.63x 6.07x 6.15x 8.18x 8.24x 8.16x 7.74x regular_w8_v_8bpc_dotprod: 5.70x 5.22x 5.39x 5.04x 7.91x 7.43x 5.04x 4.69x

sharp_w8_v_8bpc_neon: 2.90x 2.79x 4.56x 4.65x 5.92x 5.98x 7.26x 6.87x sharp_w8_v_8bpc_dotprod: 5.64x 5.17x 5.40x 5.02x 7.85x 7.53x 5.04x 4.58x

regular_w16_v_8bpc_neon: 3.03x 2.93x 4.03x 4.27x 4.65x 4.75x 6.28x 5.81x regular_w16_v_8bpc_dotprod: 4.64x 4.38x 3.75x 3.77x 4.78x 4.82x 3.89x 3.75x

sharp_w16_v_8bpc_neon: 2.29x 2.22x 3.09x 3.30x 3.44x 3.54x 5.52x 5.11x sharp_w16_v_8bpc_dotprod: 4.62x 4.40x 3.74x 3.77x 4.77x 4.89x 3.89x 3.73x

regular_w32_v_8bpc_neon: 2.46x 2.41x 3.36x 3.50x 3.58x 3.79x 5.11x 4.86x regular_w32_v_8bpc_dotprod: 3.87x 3.74x 3.20x 3.16x 3.71x 3.93x 3.13x 3.11x

sharp_w32_v_8bpc_neon: 1.84x 1.81x 2.57x 2.69x 2.63x 2.81x 4.44x 4.25x sharp_w32_v_8bpc_dotprod: 3.86x 3.74x 3.18x 3.15x 3.68x 3.94x 3.13x 3.10x

regular_w64_v_8bpc_neon: 2.17x 2.17x 3.14x 3.29x 3.19x 3.45x 4.46x 4.46x regular_w64_v_8bpc_dotprod: 3.43x 3.41x 3.00x 2.98x 3.31x 3.58x 2.74x 2.79x

sharp_w64_v_8bpc_neon: 1.61x 1.63x 2.42x 2.53x 2.34x 2.54x 3.89x 3.89x sharp_w64_v_8bpc_dotprod: 3.38x 3.41x 3.00x 2.98x 3.29x 3.58x 2.73x 2.79x

regular_w128_v_8bpc_neon: 1.99x 2.08x 3.03x 3.19x 3.00x 3.27x 3.93x 4.13x regular_w128_v_8bpc_dotprod: 3.15x 3.30x 2.90x 2.90x 3.11x 3.39x 2.48x 2.59x

sharp_w128_v_8bpc_neon: 1.49x 1.54x 2.34x 2.46x 2.21x 2.41x 3.46x 3.61x sharp_w128_v_8bpc_dotprod: 3.15x 3.26x 2.90x 2.91x 3.11x 3.39x 2.49x 2.60x

Copy/widening-copy micro benchmarks (will be a different MR) A76-mct A76-mc A78-mct A78-mc X1-mct X1-mc A55-mct A55-mc

regular_w2_0_8bpc_neon: 6.36x 5.13x regular_w2_0_8bpc_dotprod: 7.74x 5.87x

regular_w4_0_8bpc_neon: 4.87x 5.60x 5.88x 5.53x regular_w4_0_8bpc_dotprod: 5.89x 9.08x 9.78x 6.50x

regular_w8_0_8bpc_neon: 4.71x 6.94x 3.45x 4.71x regular_w8_0_8bpc_dotprod: 5.37x 7.45x 4.16x 5.22x

regular_w16_0_8bpc_neon: 2.41x 5.23x 3.27x 6.32x 6.08x 7.60x 2.17x 3.33x regular_w16_0_8bpc_dotprod: 2.61x 5.86x 3.41x 6.83x 5.51x 8.49x 2.38x 4.02x

regular_w32_0_8bpc_neon: 2.16x 3.35x 1.45x 1.87x 2.21x 1.90x 1.64x 2.08x regular_w32_0_8bpc_dotprod: 2.51x 3.73x 1.45x 2.15x 2.29x 2.35x 2.04x 2.72x

regular_w64_0_8bpc_neon: 1.73x 2.02x 1.06x 1.11x 1.68x 1.03x 1.71x 1.43x regular_w64_0_8bpc_dotprod: 1.46x 2.22x 1.05x 1.10x 1.87x 1.00x 1.67x 1.71x

regular_w128_0_8bpc_neon: 1.28x 1.37x 1.02x 1.09x 1.33x 1.05x 1.57x 1.41x regular_w128_0_8bpc_dotprod: 1.15x 1.36x 1.02x 1.09x 1.40x 1.07x 1.57x 1.42x

Edited by Arpad Panyik

Merge request reports