Add a general "machine learning" / neural network API
This API should be designed in a way that's reusable. I'm thinking of an API that has the following, as discrete function calls:
- Convolution layers (signature:
none -> color
), particularly fractionally strided convolutions - Pooling layers (e.g. max pool)
- Various nonlinear functions (signature:
color -> color
), so the user can choose which one to use
I doubt we'll implement fully connected layers any time soon.
Design plans / notes
- It may be beneficial to implement support for multiple color variables (i.e. extend the signature list by
color[]
, perhaps for a small range of fixed colors, i.e. COLOR_ARRAY_0 up to COLOR_ARRAY_16), and augment libplacebo with support for multiple render targets - Alternatively, we could almost certainly get away with treating each output layer as a separate shader invocation, since I'm not sure if MRT support would actually improve performance
- Users would provide the weights upfront, alongside the configuration data for the step (i.e. stride), and attach a sufficiently sized array of textures.
- Can get speedups from
textureGather
, also compute shaders. See mpv-prescalers for code examples, although note the LGPLv3 licensing of this repo. A more liberally licensed example is igv's FSRCNN-TensorFlow and the repo it was forked from. - It may be beneficial to also attach a
pl_shader_obj
so we can reuse the shader body. But, this needs testing. Most likely the shader invocation is so slow that the bit of CPU time we save by caching the shader is not worth it.