Setup automated performance testing (and integrate GitLab CI)
Since improving performance is the whole reason dav1d is being build automated performance testing and monitoring should be usefull. I wish I would be able to implement this myself, but here is the general direction in which I'm thinking.
- Determine a number of test we want to run automated. Probably a general performance test on different platforms (arm64, x86) with different SIMD extensions (depends somewhat on #198 (closed)), a scaling test (performance over 1 to n threads) and a profiling test (to determine which functions improved/degraded). Please let know what (else) would be usefull.
- Encode or get files to perform these tests.
- Get a very consistent hardware platform. This could be dedicated on existing systems (tests will be run overnight), but it could also be cloud based. Since Google, Amazon and Microsoft are all AOM members they might be willing to help. Fixing clock-speeds is extremely important since we have had enough Turbo woes (#101 (closed)).
- Get a stable test software configuration. Stable Linux, ivf files on a (persistent) SSD, the needed test packages installed.
- Write some test instructions/code/commands to run tests and output/save usefull data.
- Trigger this platform daily to run these performance test. Also enable a option to trigger it manually for Merge Request that you suspect will impact performance.
- Upload the results to some database. Send push notifications if performance differs more than some margin of error (2% or so).
- Build some fancy graphs and statistics from these database
- If possible, integrate with Gitlab CI.