Frames are out of order when receiving RTSP
Summary
I am attempting to receive and display an RTSP video stream (H.264) in real-time on an iPad. However, under poor network conditions, the frames are received out of order.
When setting the option network-caching=0, which lowers network stability, the issue occurs more frequently.
Minimal project and steps to reproduce
RTSP Client (Unity)
Use the provided "VLC Canvas Example" in VLCPlayerExample.cs and replace the media URL with the RTSP server URL.
RTSP Server
Requirements:Raspberry Pi or Linux PC connected to the same network as the client,A USB webcam.
- Build and install GStreamer and gst-rtsp-server.
- Fix the PC's IP to the RTSP URL's IP address and connect the webcam.
- In the gst-rtsp-server build directory, execute the following command to stream the webcam feed via RTSP:
./test-launch '( v4l2src device=/dev/video0 ! 'image/jpeg,width=1280,height=720,framerate=30/1' ! jpegdec ! videoconvert ! x264enc ! video/x-h264,level=(string)4 ! h264parse ! rtph264pay name=pay0 pt=96 )'
If setting up an RTSP server is difficult, you can use the following test RTSP URLs: rtsp://rtspstream:d3c0c9e2be8e05ffbae227207e124eca@zephyr.rtsp.stream/movie
(If this link stops working, you can either let me know or create new links at rtsp.stream. They have a free tier that gives you two test RTSP links and lets you stream up to 2GB/month for free.)
What is the current bug behavior?
Under poor network conditions, the frames are received out of order.
What is the expected correct behavior?
Even under poor network conditions, the frames be displayed in the correct chronological order.
Does it work on other plaforms? Does it work with the official VLC apps?
- VLC Player (PC): This issue does not occur.
- Windows 11 x64(VLC for Unity): The same issue occurs.
Relevant logs and/or screenshots
The following image shows the RTSP stream with timestamps, recorded on the screen received by Unity and arranged frame by frame. (Windows 11 x64)
The time sequence of frames is reversed between frames 191 and 192. Also, frame 192 is the same as frame 190.
Environment
- Device: iPad Air Gen5
- OS: iPadOS 17.5.1
- Unity version: 2021.3.11f1
- RTSP source: Raspberry Pi 4
