Loosely defined TODO list, intended both as a roadmap as well as an entry point for developers wishing to contribute but not knowing where to start.

Missing software features:

• tile_ext (aomenc --large-scale-tile=?).
• error resilience (drop a frame but don't die; #307).

SIMD:

• simd for any function already in a ${anything}DSPContext, for any platform (AVX512: #316, AVX2: #78, Neon: #215, SSSE3: #216, VSX: #281);
• move dequant from decode_coeffs() to itx;
• order_palette() to dsp for simd;
• change coef contexting (hi/lo_ctx) to be diagonal-oriented for dsp/simd;
• project_motion_field in ref_mvs.c can maybe be SIMD'ed;
• a specifically optimized version for mc.put/prep_scaled() for super_res, since then my is always 0, so there is only horizontal scaling, not vertical.

Multi-threading:

• postfilter and film-grain threading;
• in first-pass of frame threading with tile threading enabled, it may make sense (assuming no temporal interference from ref_mvs or seg_id) to first parse the tile marked as the one used to update the output CDF, since that would unblock the subsequent thread's pass 1. This is only true if use_ref_mvs=0 and segmentation.temporal_update=0;
• threading can become a generic worker queue (one tile_sbrow symbol parsing/recon, one sbrow postfilter(s)) and then use a generic single threadpool instead of separate tile/frame[/postfilter?] ones (see also #206);
• by not adding invisible frames to out_delayed[] and/or growing it so it can be bigger than the number of frame threads (and thus making the indexing between out_delayed[] and the actual frame thread doing the decoding independent), we could grow concurrency and scalability on typical sequences with frame-multithreading enabled.

Removing redundancies:

• it may make sense to copy one row (8px+2x2px edges) of pre-cdef data in uint16_t at a time so we don't need to extend buffers or add edge data inside the SIMD (!657). This may make the code both simpler and faster. Same is true for looprestoration also;
• backup_lpf() in lr_apply_tmpl.c backs up 4 lines per 64 pixels per plane, and copies bottom to top per superblock (each 128 or 64 pixels). Most of this is unnecessary. Using a flippable index means we don't need the second copy, and using 64-pixel instead of sb (64 or 128) pixel cdef runs (and then running LR, and then optionally the second cdef and second LR) means we only need to copy the pre-cdef top pixels, not the bottom ones, saving 50% copies. CDEF backup already does all of this;
• bonus points for merging the CDEF backup and LR backup together so LR backs up nothing at all;
• for inter, test if prediction at 128x128 should be done at 64x64 subblocks to improve cache efficiency (and also simplify SIMD);
• the identity_* inverse transforms are stored transposed (as are all other coefficient tables). In all other cases, this saves a transpose in assembly, but for those, it actually means we have to transpose, even though in theory we wouldn't have to at all. Therefore, a potential optimization would be to have a special untransposed zigzag coefficient table and remove the transpose from the assembly, which would make those inverse transforms slightly faster.

Other speed optimizations:

• film-grain GL shader (like placebo);
• get rid of memset(0) of seq_hdr and frame_hdr after allocation.
• Reuse per-frame buffers when possible instead of freeing and reallocating them all the time. Applicable to (in order of importance) picture_alloc_with_edges(), dav1d_submit_frame(), dav1d_cdf_thread_alloc(), and dav1d_parse_obus().

Cleanups:

• palette buffers are always 16-bit, even if content is 8-bit (remaining item in #257);
• LR/MC intermediate 2d buffers in C dsp can be reduced by doing windowed like in SIMD;
• cdef: noskip_mask resolution can be 8x8;
• ref_mvs: non-cur frame MVs can be at 8x8 resolution, only direct neighbours need to be 4x4;
• lfmask and l/a ctx zero can be done in tile instead of frame context for better distribution.
• show_existing_frame will be placed in the frame output queue as something keeping a frame thread busy, meaning for such cases, the frame thread will momentarily stall. This is partially required to prevent overflows of the output queue, or growing it to possibly infinite size on garbage input. But for the regular use case, we can make the output buffer queue twice as big, so that each invisible frame can have one matching show_existing_frame, allowing all frame-threads to be active for the worst-"real"-case while still never overflowing on pathological conditions;
• the output queue handling is duplicated in decode.c, lib.c and obu.c, so merge this in one common place.

Reimplement:

• ref_mvs.c(#217).