Draft: OpenGL providers v2: move OpenGL execution and provide async providers
Draft status:
-
Missing more documentation. -
With more assertions, OpenGL tests are failing. -
!6412 is needed for this MR. -
Macosx.m output is not completely functional. (see TODO commits). I might re-do the same work as caopengllayer before. -
No tests for the async opengl runtime. -
Hasardeous locking for the render() request, need some discussion on whether we should wait for the frame to be rendered or not. -
request_changes() need some test and work to make it work correctly before the whole change iirc. -
destroy() as callback (ie. context_lost) is not implemented yet, and mandatory for this patchset to work.
As per workshop from last week, I'll wait for 5.0 to add a capability to be able to implement opengl sync providers (most of those being the current providers) without having the sync function appearing on the vlc_gl_t API and simplify the code in the core. I'll detail this point in a separate issue.
I'm eager to have some feedback on the current design and naming, I hope this draft is clearer than the previous one (!1316). The modules caopengllayer and the emscripten webgl implementation from VLC.js have successfully been implemented within this new architecture, as well as opengl filters, opengl display, some tests and visualization. This also provide a framework to implement !6446 properly by not rendering any frame before the backing layer is ready, which allows the window module to be completely async and solve the issues from !457.
The current OpenGL implementation is focused on its ability to be run from different threads, by making the provider current and then releasing it when it's not used anymore.
Time |----------------------------------------------------->
Thread 1 MakeCurrent ---- Release current
Thread 2 --- Make Current ...
---
Fig.1: Current expectation on the APIIn particular, Thread 1 tends to be the decoder thread signalling the
new video output format, leading to the creation of the video output,
and Thread 2 tends to be the Prepare/Display of the vout display
module running inside the vout thread.
However, some platforms like emscripten won't allow a provider to be shared from different threads, because the underyling resources can only exist in a single thread (=worker) at once and is transferred from a global event loop which requires a dedicated thread lifecycle. Even though the concept of Transferable object exists, objects like an OpenGL context are not Transferable and there's no way to have data shared between contexts at the OpenGL level. In more specific terms, only textures can be shared through ImageBitmap "external" texture, but shaders, programs, vertex buffer objects, etc, cannot be shared. A proposal was submitted1 a while ago but was abandoned and replaced by the non-OpenGL concept of OffscreenCanvas2 and ImageBitmapRenderingContext3.
+--------------------------+
| |
| SHARED MEMORY |
| |
+--------------------------+
↑ ↑
+----------+ +----------+
| | | |
| Thread 1 | | Thread 2 |
| | | |
+----------+ +----------+
↑
| Only exist
| in this
| thread
|
| Canvas WebGL | <--------- | Offscreen |
| context | | canvas |
---
Fig.2: HTML5 Offscreen canvas limitationsSome others like macOS can allow the context to be shared to multiple
threads but won't provide a complete framebuffer if the OpenGL drawcalls
are not done from the callback of the underlying implementation. This
currently enforce the macOS providers to be vout display instead of
opengl providers, which makes them incompatible with filters and
visualization modules like glspectrum. It's still possible to make them
work like an OpenGL implementation by copying or by removing part of
their features regarding resizing though.
+--------------------------+
| |
| SHARED CONTEXT |
| |
+--------------------------+
↑ ↑
+----------+ +----------+
| | | |
| Thread 1 | | Thread 2 |
| | | |
+----------+ +----------+
↑
| Only valid
| in this
| thread
|
| Valid default | <----------- | CALayer backing |
| Framebuffer | | buffer |
---
Fig.3: MacOS CALayer-based context limitationsFinally, mobile implementation and some desktop implementation can signal that the context has been lost, which means that the OpenGL display can avoid trying to render pictures and re-initialize the objects in such cases. A browser WebGL2 environment will typically kill OpenGL contexts after more than 16 contexts are created and running simultaneously.
OpenGL client OpenGL implementation
------------- ---------------------
| |
| Create provider |
| -----------------------> |
| |
| Making current |
| -----------------------> |
| |
| Calling OpenGL drawcalls |
| -----------------------> |
| ... |
| Reporting context loss | ]
| <----------------------- | ] Currently no way to achieve
| ??? | ] this signalling
| | ]
| |
| Calling OpenGL drawcalls |
| -----------------------> |
| (results in errors) |
---
Fig.4: OpenGL implementation reporting a context-lossBeing able to call make current from different threads made sense when a single context was used to generate intermediate pictures, but the offscreen provider implementation is already able to fullfill this requirement without suffering from most of the OpenGL defects when it comes to multi-thread. Except from that fact, it was mainly used because the thread the context was run on wasn't known nor stable, which is what was blocking the new implementation from being implemented.
+-----------+ +-----------+ +--------------+
| Filter | +-→| Filter | +-| Vout display |
+-----------+ | +-----------+ | +--------------+
↓ | ↓ |
+-----------+ | +-----------+ |
| Offscreen | | | Offscreen | |
| OpenGL | | | OpenGL | |
+-----------+ | +-----------+ |
↓ | ↓ |
swap() | swap() |
↓ | ↓ |
picture_t ------+ picture_t ------+
---
Fig.5: Multiple OpenGL providers instead of shared contextThis commit provides the tools to allow the three use cases mentioned above. The opengl provider user can provide an additional owner structure to specify how to re-init the resources, how to release these resources when the context is lost, and how to render the next frame.
OpenGL client OpenGL implementation
------------- ---------------------
| |
| Create provider |
| -----------------------> |
| |
| Request renderer init |
| -----------------------> |
| <---------------------- |
| Initialize renderer |
| |
| Request render |
| -----------------------> |
| <---------------------- |
| Use the renderer callback |
| to render the frame. |
| ... |
| ... |
| Report context loss | ]
| <----------------------- | ] The context loss is now
| ... | ] handled so the display or
| Request renderer init | ] filter can stop its job
| -----------------------> | ] until the renderer is
| ... | ] initialized again.
| |
| ... |
| |
| Request renderer change | ] This allows implementation
| -----------------------> | ] of update_format callbacks
| <---------------------- | ] or alike, when a synchronous
| Change renderer in | ] change is needed from the
| correct thread | ] implementation.
---
Fig.6: New OpenGL provider APIThis solution is a bit heavier on the client side, since the commands for initializing the OpenGL state, rendering a new frame, and mutating the OpenGL state after initialization need to be explicitely defined. But enforcing the client to define how the graphics resources are initialized and how the rendering should be done allows to easily pause the rendering and re-initialize the renderer without dedicated support.
The other solution would have been to keep the model with makeCurrent and releaseCurrent, but use indirect rendering by proxying each calls or storing them in a commandQueue before sending it to the real OpenGL implementation on the real thread that is supposed to execute the call. However, this has drawbacks:
-
It has a huge allocation cost, synchronization cost, and complexity cost since the whole OpenGL API needs to be wrapped behind the layer.
-
It also wouldn't have solved the context loss handling since the client wouldn't know when to re-create the resources, and how.
-
Finally, we have no need to be able to move OpenGL contexts from threads to threads, so making current and release implies a lot of context syncing that were not necessary.