Skip to content

MediaPlayer.Size() Has Trouble WIth Some Portrait Videos

Summary

MediaPlayer.Size() sometimes returns the wrong size.

Minimal project and steps to reproduce

  1. Make a new Unity project and import VLCUnity from the Package Manager. (Any unity version should do, but I am using 2020.3.6f1.)
  2. Replace MinimalPlayback.cs with attached modified version. MinimalPlayback.cs
  3. Download the attached example videos. Example_1_Working

Example_2_Broken 4. Open the MinimalPlayback scene.

  1. Select the VLC Screen 1 object.

  2. In the inspector, set the path to point to Example1Working.mp4. This video is 640 pixels wide and 360 pixels tall.

  3. Press Play.

  4. Double click the output texture (Tex) in the inspector. It's 640 by 360 like it's supposed to be, the video is flipped, but not stretched or squashed. (Being flipped is a different issue with a different cause, so I made it it's own ticket here #112 (closed)) Unity_2021-07-01_20-48-11_1625197691

  5. Press Play again to exit play mode.

  6. Reselect the VLC Screen 1 Object.

  7. In the inspector, set the path to point to Example2Broken.mov. This video is 720 pixels wide and 960 pixels tall, but thinks it is 960 wide and 720 tall. Unity_2021-07-01_20-54-37_1625198077

  8. Press Play.

  9. Double click the output texture (Tex) in the inspector. It's 960 by 720, and the video is squashed vertically and stretched horizontally.

What is the current bug behavior?

Example2Broken.mov is stretched and squashed.

What is the expected correct behavior?

Example2Broken.mov is undistorted.

Does it work on other plaforms? Does it work with the official VLC apps?

Right now VLC for Unity only supports one platform.

It works fine, more or less, on the official VLC apps. If you play this video in VLC, it appears unstretched in portrait format like it should. If you turn off scaling and count the pixels, it's rendering at the right size. However, if you press ctrl-J, VLC thinks it's 960 wide and 720 tall. One potential insight here is that their orientation is different. This video is Bottom Right, while Example1Working.mp4 is Top Left. Interestingly, if you take a snapshot from VLC, the image it outputs is not stretched, but is rotated 90 degrees. VLC's behavior on this video is not without quirks, but it is much better than VLC for Unity's current behavior.

ShareX_2021-07-01_19-23-00_1625192580

(This is a screenshot of the VLC desktop app, showing the video undistorted and outputting at the correct size. There's a thin black bar at the top of the video, but I think that's just a part of the video. It is possible that bar will turn out to be important, but for right now, I think it isn't.)

Environment

  • OS: Windows
  • Version 10 LTSC
  • Device: PC
  • LibVLC version and architecture 4.0.0 x64
  • LibVLCSharp version Not quite sure how to check this, sorry.
  • VLC Unity plugin version 0.1.3
  • Scripting backend used Mono

Possible fixes

I don't know anything about the vlc-unity code yet, but mfkl on discord suggested that this line might be a place to fix this problem.

https://code.videolan.org/videolan/vlc-unity/-/blob/master/PluginSource/RenderAPI_D3D11.cpp#L497

As far as I understand it, what's happening here is that m_outputTexture (which VLC is generating) is copied onto m_textureUnity (which is a Unity texture created in C# code in MinimalPlayback.cs). This function doesn't care if the two textures are the same size -- you can set the Unity texture to any size you want, and CopySubresourceRegion will stretch m_outputTexture to fit. So the problem is actually in MinimalPlayback.cs! We're creating the unity texture at the wrong size. But how do we create it at the right size? Is there a way to get it right automatically for both of these videos?

Right now it's being set from MediaPlayer.Size(). Ideally, MediaPlayer.Size() would return 720 by 960 for Example2Broken.mov. Perhaps this could involve handling the size differently depending on this Orientation property.

Another option could be to find a way to expose the size of m_outputTexture to the MediaPlayer object. I think m_outputTexture probably is rendering at the actual resolution of the video, not stretched, so if I'm right about that and we could get those dimensions and use them to create the Unity texture at exactly the same size, we'd be good. (My only reason to think this is that it's right in the official VLC app, but that's very encouraging I think!)

Failing that, if MediaPlayer exposed this Orientation property you can see if you press ctrl-J in VLC, and assuming I'm right that that's what makes the difference, we could check that property in MinimalPlayback.cs and create the texture differently for different Orientations. This makes the boilerplate you need to write on the Unity side to get VLC Unity up and running a few lines longer, which isn't ideal, but it would probably work.

Edited by LL

VideoLAN code repository instance