MP4 ffmpeg mux broken with movflags=empty_moov

Unrelated to the issue at hand that VLC is unable to play it back - you'll probably get better results with the fragmented mp4 if you set an option controlling how to fragment it as well, such as movflags=empty_moov+frag_keyframe or so, possibly together with min_frag_duration=n (in microseconds). Right now (when testing the same setup directly via avconv) this produces a file consisting of a moov atom describing no samples at all (as intended by the empty_moov flag), followed by one moof+mdat containing the whole file.

The libavformat mp4 muxer should have a fallback to add the frag_keyframe option if no other fragmentation method is enabled but fragmentation is enabled (implicitly via the empty_moov option), but this fallback only seems to kick in if using it in ISM/smoothstreaming mode. I'll fix this detail shortly.

I'm a bit unsure of the terminology here (I don't know the MP4 mux too well); is fragmenting something different from segmenting? I specifically don't want a bunch of small files (HLS-style).

I should mention that ffmpeg with -movflags empty_moov works fine for me; I would believe there's something about the pts/dts here. If I specify mov_timebase=48000 (same as my sample rate), so that the video and audio come on the same timescale, I think I get something that plays marginally better (I need to re-test this when I get back home), although still really bad.

Yes, fragmenting (at least in this context) is different from segmenting. Normally mp4 files aren't streamable whatsoever, but if you fragment them (and use the empty_moov flag), they will be streamable (among those few mp4 demuxers that support fragmented mp4 and an empty moov atom) - I'm not too familiar with webm/mkv but I think the cluster concept there is the same as a fragment within mp4.

If the file is only one fragment, you can't really stream it since the fragment is never output from the muxer until it's finished. frag_keyframe tells the muxer to cut to a new fragment at each keyframe, and with the min_frag_duration you can tell it to only do this if the fragment is over some minimum duration.

Ok, so if it works with ffmpeg but not from VLC there might indeed be something in the wrapping inbetween that goes wrong. Or something in the libavformat muxer that fails with valid but different parameters set by VLC. I'll see if I can look into it or if somebody else beats me to it.

OK, so that makes sense; I'd probably want keyframe-sized fragments there.

I still think a key difference here is the huge pts/dts (CLOCK_MONOTONIC-based) that VLC likes to sneak in, but that's just a hunch based on prior experience. The /usr/bin/ffmpeg encode starts the dts at 0, where it should be.

Ah, right, that's a good clue, I'll see if I can reproduce issues with plain avconv/libavformat if I make it use such timestamps.

I tried again now with just frag_keyframe,mov_timescale=48000. Then I tried with empty_moov,frag_keyframe... and it works! (At least for a video-only stream, haven't tried audio yet.) But I get negative timestamps in mplayer (it starts at -91325.9), so it seems that there is indeed an overflow somewhere that's dependent on pts/dts, and then obviously also dependent on mov_timescale (whether it's the default of 12800 or given externally).

Ok, interesting. I tried manually adding some offset within libavformat, but that didn't trigger any issue so perhaps my offset was too small. I'll test it within the context of VLC later.

FWIW, once the rest of it is in place and you want to stream this (which I guess is what you're aiming towards), you'll need to add the omit_tfhd_offset flag to movflags as well (you'll need a version of libavformat that includes the commit e7bf085b, i.e. after September 18 last year), otherwise you can't skip fragments in the stream.

Yes, the goal is streaming to if that's possible, or if not, figure out why it's not. (I wonder if it's possible without huge delays, but I guess that depends on fragment size somehow.) I already fixed a bug in ffmpeg wrt. writing to pipes; I'm not sure if I actually need to be able to skip fragments in the stream (WebM can't, so I've already made sure I can serve a large backlog).

I ran a test with movflags=empty_moov+frag_keyframe+omit_tfhd_offset. Seemingly I can connect MPlayer to the stream and get something viewable if I do not watch the first fragment. The audio and video pts are completely different (apart by days), but it's a fixed offset, so they don't drift from each other:

A:89483.5 V:335549.4 A-V:-246065.891 ct: -0.001 0/ 0 1% 1% 0.3% 0 0 47%

After maybe 20-30 seconds, however, the stream dies. Chrome plays audio but not video; same with VLC.

Ok, so I've looked at the issue more in detail now, and it seems that the issue with dts not starting from 0 only lies in the ffmpeg side of things - on the libav side (where this feature originally was developed by me), your setup works just fine. The "feature" in ffmpeg which breaks it is unfortunately not easy to disable at runtime, your easiest bet is to comment out the "trk->cluster[trk->entry].dts = trk->start_dts = 0;" line in ff_mov_write_packet. (See commit 537ef8be for details on the feature that breaks it.)

As for the other options I mentioned earlier; you might not need to add frag_keyframe since VLC seems to flush fragments manually already (commit afac3fec in VLC, by you) - instead it might make sense to add frag_custom to movflags instead, telling the muxer that you will handle the flushing manually and the muxer need not try to do it on its own. Although as long as both try to do it on keyframes, the extra attempted flush by the muxer internally won't do anything.

Commenting out that line doesn't seem to change anything for me, either way. All the previous symptoms still apply.

That's strange then. I guess the issue I found and looked at is separate from your issue then, or your new runs didn't pick up the updated libavformat properly.

Can you provide a sample of an output.mp4 produced with your builds, and/or try building using the libav fork instead of ffmpeg, since the issue I identified yesterday isn't present in the libav side at all?

The issue should be fixed in current FFmpeg, thank you for the report!

Yes, this is now much better with current ffmpeg. It's still not ideal, though. In particular, the skewed PTS are still skewed:

A:89486.3 V:335552.2 A-V:-246065.906 ct: -0.002 0/ 0 1% 0% 0.4% 0 0

I've actually managed to stream to Chrome now (with a patch), but it works very spottily. Streaming to MPlayer works, but it just breaks the connection after ~30 seconds or so.

It looks like if I set a specific keyframe interval, streaming to MPlayer is stable. But Chrome seems to play nothing, then ~20 frames at a time, then nothing, and so on. I wonder, how is audio and video muxed together within a fragment?

Within a fragment, all samples for one track are batched up together, so the fragment would first have all audio packets, then all video packets for that track. This is mainly since the moof/trun structure gives a much larger overhead if they are interleaved sample by sample (you basically need a complete separate trun atom for each sample in that case). I think I've got some WIP code somewhere that actually does that, I'll try to dig it up and see if it would help.

I see now that Chrome actually manages to play back fragmented mp4 files - in my earlier tests it wasn't able to play back anything past the first fragment at all so apparently there's been some progress there.

OK, I suspected something like this. Single-frame fragments would help with this, of course, but seemingly setting frag_duration=1 breaks once the video resolution becomes too big.

Chrome isn't the only player having issues with these streams; VLC with -vv also complains about having to increase and decrease the sample rate because of way early audio.

Curiously enough, if I wget the stream to a file and then let Chrome play it over HTTP, it works just fine. It seemingly expects to do some seeking (with range queries) but not during normal playing, so I don't understand fully what's going on.

So, I had some more time to look a bit closer at this.

I do have some old code for writing samples interleaved within a fragment - have a look at https://github.com/mstorsjo/libav/commits/movenc-frag-interleaving (the topmost 3 commits) if you want to try it out - add frag_interleaved to movflags to enable it. This adds quite a bit of muxing overhead though, and if you want the fragments to be position independent within the stream (so that you can cut out fragments and just assemble moov+any later fragments) the overhead grows even larger.

However this didn't seem to help at all for making playback any smoother in Chrome.

FWIW, my test streams play back just fine with VLC (and I don't see any warnings about having to adjust sample rate), but in Chrome the audio is continuous while the video is quite choppy. It seems like it doesn't allow buffering enough video to play it back smoothly (while it manages to play the audio just fine).

I tried a separate test setup as well - I've got a premade fragmented MP4 file with interleaved samples within the fragments, and I use a CGI script to feed this as a plain byte stream at roughly realtime rate, and Chrome plays it back choppily. http://albin.abo.fi/~mstorsjo/mp4-frag-interleaved.mp4 is the source file and http://albin.abo.fi/~mstorsjo/cgi-bin/stream.cgi is the throttling CGI script.

Actually, I think I found one explanation to the issue there. Instead of having my CGI script feed the data at realtime rates, I tuned it up to feed data faster than realtime. The audio track plays back as it should at realtime rate, while the video plays as fast as the data is received - much faster than realtime, completely regardless of the actual timestamps of the video packets. So something seems to be funky with how Chrome plays back these non-seekable streams. To try this out for yourself, try http://albin.abo.fi/~mstorsjo/cgi-bin/stream-fast.cgi.

FWIW, it seems that the latest Chrome (40) on OS X (at least, haven't tested others) finally manages to play back fragmented MP4 streams with audio and video properly in sync. http://albin.abo.fi/~mstorsjo/mp4-stream.html is an example of this, that seems to work fine now - previously the audio and video were severely out of sync. (This is not a live stream, just a static file that is served, throttled.)

So - Sesse, if you still have time and interest, you may want to retry this now. As long as the output fragmented MP4 is ok in general, I'd hope that Chrome would play it back nicely now as well.

I can confirm Chrome 40 on Linux also gets this right. I have to say, this sounds very interesting; unfortunately I'm not doing a lot of streaming these days, and browsing the backlog, all of these details are long gone from my head. :-/ But MP4 streaming to would still be cool to have, indeed.

Can you perhaps make a summary of what patches (if any) and command-line flags are needed to get this to work? I'm sure it would be useful to whoever comes after me, at the very least. :-)

MP4 ffmpeg mux broken with movflags=empty_moov

Child items 0

Activity