LibVLCSharp 4 - MediaPlayer.SetOutputCallbacks Marshal issue?
Summary
LibVLCSharp 4's new MediaPlayer.SetOutputCallbacks appears to have an issue with marshalling the callbacks that a) use a pointer and b) contain a bool type because the bool types are not blittable.
Minimal project and steps to reproduce
Call MediaPlayer.SetOutputCallbacks with the required D3D11 callbacks on Windows using .NET framework (I'd imagine the same would work for OpenGL). For example:
m_MediaPlayer.SetOutputCallbacks(
VideoEngine.D3D11,
m_OutputSetup,
m_OutputCleanup,
m_OutputSetResize,
m_UpdateOutput,
m_Swap,
m_MakeCurrent,
null,
m_FrameMetadata,
m_OutputSelectPlane);
What is the current bug behavior?
Compilation works but at runtime an exception occurs: System.Runtime.InteropServices.MarshalDirectiveException: 'Cannot marshal 'parameter #2': Pointers cannot reference marshaled structures. Use ByRef instead.'
See Possible Fixes for what I think might be going on.
What is the expected correct behavior?
No runtime exception. At least until it reaches my other broken code.
Does it work on other plaforms? Does it work with the official VLC apps?
Given that I saw references to things matching an official sample when the structs were added, I've wondered if maybe e.g. Mono handles the marshalling just a bit differently and it works on other platforms.
Relevant LibVLC logs and/or screenshots
Let me know if logs make a difference here, but I think it's just a P/Invoke thing.
Environment
OS: Win 11 Device: Laptop LibVLC version/arch: x64 LibVLCSharp 4 preview 4.0.0-alpha-20230324-6630, problem believed to exist on current master 3/29/2023.
Possible fixes
It is believed that this is actually sneakily caused by the presence of the bool fields on SetupDeviceConfig and RenderConfig. Changing these to ref
parameters on the callback seems to not blow up. For example, trying to monkeypatch in the following no longer throws the exception:
[DllImport(/*Constants.LibraryName - inaccessible; relatively safe to hardcode on Windows*/ "libvlc",
CallingConvention = CallingConvention.Cdecl,
EntryPoint = "libvlc_video_set_output_callbacks")]
static extern bool LibVLCVideoSetOutputCallbacksPatched(IntPtr mediaplayer, VideoEngine engine, OutputSetupPatchedCallback outputSetup,
OutputCleanup outputCleanup, OutputSetResize resize, UpdateOutputPatchedCallback updateOutput, Swap swap, MakeCurrent makeCurrent,
GetProcAddress getProcAddress, FrameMetadata metadata, OutputSelectPlane selectPlane, IntPtr opaque);
delegate bool OutputSetupPatchedCallback(ref IntPtr opaque, ref SetupDeviceConfig config, ref SetupDeviceInfo setup);
delegate bool UpdateOutputPatchedCallback(IntPtr opaque, ref RenderConfig config, ref OutputConfig output);
LibVLCVideoSetOutputCallbacksPatched(
m_MediaPlayer.NativeReference,
VideoEngine.D3D11,
m_OutputSetup,
m_OutputCleanup,
m_OutputSetResize,
m_UpdateOutputPatched,
m_Swap,
m_MakeCurrent,
null,
m_FrameMetadata,
m_OutputSelectPlane,
GCHandle.ToIntPtr(mediaPlayerGCHandle));
My gut says there's a different way to make the bool blittable etc. - not sure on any performance penalties for the ref
approach; usually I'd not be worried but these calls are likely a bit hotter than normal so the performance may be a consideration.