QuickTime AVC-Intra 50/100 and MXF AVC-Intra 50 without SPS-PPS not playing

Seriously, AVC without SPS-PPS? How do we guess it?

So, Avid Media Composer has a bug, and the solution chosen is to violate the spec?

Who is the genius who decided that?

ffmpeg uses static tables in that case https://ffmpeg.org/doxygen/2.1/libavformat_2utils_8c_source.html#l04194

Hello Jean-Baptiste and François. Thanks for your usual quick responses.

Obviously the AVC(-Intra) stream before going into a decoder needs at least one SPS-PPS set.

But the situation lies a bit more complicated. For broadcast AVC-Intra (HD) it started somewhere around 2008 or so with Panasonic P2 AVC-Intra MXF OPAtom OP1b files with SPS-PPS only in first frame.

Then we got QuickTime files (Apple, Panasonic, ...) with various Codec ID's including a generic (Panasonic) 'AVCI' and specific ones for each AVC-Intra format (Apple) 'ai1x' and 'ai5x'. QuickTime .mov using these Codec ID's are strongly related/overlapping with ISO/IEC 14496-15, in which there is provision for a 'avcC' AVC Configuration atom, which extends the Image Description atom in the 'moov' (metadata, not samples). The samples itself may or may not contain SPS-PPS (see next paragraph).

Begin 2013 we asked Panasonic Japan (at that time leading in broadcast AVC-Intra) for their recommendations and they advised, that MXF OPAtom should have only 'first' frame (their P2 standard), MXF OP1a 'all' frames and QuickTime 'no' frames (following Apple FCP implementation). They noted however that some large vendors (amongst them Omneon/Harmonic) changed from 'all' to 'no' frames for MXF OP1a (perhaps because of the Avid bug ?).

Later for MXF came SMPTE ST 381-3-2013 which gives the option for SPS/PPS Flag in the MXF AVC Sub Descriptor, either: Unknown/NoSpecific Location, First Access Unit, Every Access Unit or Periodically placed for each GOP.

Somewhere in the middle of this (around 2012-2013) our company needed to deliver workflows including AVC-Intra QuickTime and MXF files, and some customers were using Avid Media Composer. As mentioned we discovered a bug in Media Composer that (invisible) might corrupt frames at the end. So we needed to decide to wait for some fix from Avid or see if 'no' frames in QuickTime and MXF was 'inside the spec'.

After careful considerations taking into account specifications, the conversations with Panasonic, testing and the industry trend (what other companies would read/write) we decided to make our (FORK) software flexible, such that at each point in Ingest/Import/Archive/Edit/Rewrap/Playout/Export it is configurable to use either no/first/all frames for either QuickTime/MXF. As mentioned we still believe all combinations are 'within' (QT/MXF wrapper) specifications and are read-write supported on most players (QT Player, MXF4mac, ...), editors (Premiere, FCP, Avid, ...) and SDK's (BMX, OpenCube, ...).

For some customers facing essence corruption in combination with Avid and having tested the workflow chain we (our company and the customer) choose to go for no-frames MXF files, which has been running successfully for the past few years. So if you want, you may blame me (amongst a group of others) for being that 'genius' ;-)

All this is for AVC-Intra 50/100 HD 720/1080 only, I've seen the newer Sony/Panasonic/Canon/... specifications for 2K/4K incl. Long GOP and the tendency seems to be to more include/keep the SPS-PPS sets inside the essence (Long GOP being a difficult case). But for the 'older' AVC-Intra 50/100 Mbps some SPS-PPS re-creation (from codec ID's, frame sizes and/or other header metadata) might be needed to playback 'no frame' QuickTime/MXF variants. Looking at François's ffmpeg link it already is being done in some cases...

We happened to attempt playback in VLC of some files in mentioned workflows and raised this ticket to make VLC support the broadest number of formats. Will be happy to support (check/advise/add code) any development as needed...

Thanks and keep up the good work, it is appreciated !

Henk Demper

Replying to [comment:4 HenkDemper]:

We happened to attempt playback in VLC of some files in mentioned workflows and raised this ticket to make VLC support the broadest number of formats. Will be happy to support (check/advise/add code) any development as needed...

Yeah, don't worry... It's just the usual insane decisions.

What's weird is that technically, we should already support those "things".

Could you, by any chance, try the latest ZIP version of VLC 3.0.0 win32 on our nightly builds servers and test?

Thanks

3.0.0 won't work as the avcC atom is missing. extradata needs to be passed a rebuilt AnnexB sps/pps, like ffmpeg does.

Replying to [comment:6 fcartegnie]:

3.0.0 won't work as the avcC atom is missing. extradata needs to be passed a rebuilt AnnexB sps/pps, like ffmpeg does.

But it should work for MXF, no?

Just tried both nightly win32/vlc-3.0.0-20151027-0002 and win64/vlc-3.0.0-20151027-0402 with all 12 test files and basically same result as VLC 2.2.1: all files playback Ok, apart from these 'no-frame' SPS-PPS files:

QuickTime AVC-Intra 50/100 either crashes VLC 3.0.0 or gives no image
MXF AVC-Intra 50 gives distorted picture

François: extradata needs to be passed a rebuilt AnnexB sps/pps, like ffmpeg does.

...the strange thing is that MXF AVC-Intra 100 'no-frame' does playback in both VLC 2.2.1 and 3.0.0. I reconfirmed with H264 parser that there is really no SPS-PPS present in that file. Maybe just the default codec setup just happened to match the file (100 Mbps, 1080i50)

I understand that VLC is currently not using the ffmpeg ff_generate_avci_extradata() function, but own code (likely without any static tables) ?

For what it is worth:

In contrast with ffmpeg, you will actually need 20 (not 4) different static SPS tables and (only) 3 different PPS tables. These 20 are there for: 50/100 Mbps, 1080i/1080p/720p and also framerate 24/25/30/50/60 (ffmpeg doesn't take latter into account).
The correct table entry index to be used as said can be taken either from QuickTime/MXF metadata (width/height, QT 'fiel' if available, QT codec ID if not generic 'AVCI', ...) but in general (when insufficient info from metadata) will also need to check the framesize for being either a known fixed framesize with/without 512 SPS-PPS bytes.
Need to check if SPS-PPS is already present (by frame size or NAL parsing) and if SPS-PPS is added, will need to adjust AUD to assure the decoder will see exactly one Access Unit (sample): if AUD is present it needs to be changed to Filler and added SPS-PPS must start with AUD.

We received those 20+3 static tables from Panasonic Japan, I might reach out to them if I may pass them on to VLC (and hence make them public). I would judge it would be beneficial for them as well to have playback of all AVC-Intra files. Obviously only useful if this ticket would be progressed, please advise...

--demux avformat works?

vlc --demux avformat AvcIntra50NoFrames.mov

(so the QuickTime version) indeed works: playing back the correct video :-)

vlc AvcIntra50NoFrames.mov

QuickTime version without --demux avformat fails (no image playback) with:

[h264 @ 0x102805600] non-existing PPS 0 referenced
[h264 @ 0x102805600] decode_slice_header error
[h264 @ 0x102805600] non-existing PPS 0 referenced
[h264 @ 0x102805600] decode_slice_header error
...
[h264 @ 0x102805600] no frame!
[h264 @ 0x102805600] non-existing PPS 0 referenced
[h264 @ 0x102805600] decode_slice_header error
...

Both

vlc --demux avformat AvcIntra50NoFrames.mxf

and

vlc AvcIntra50NoFrames.mxf

(so the MXF version) will identically fail (showing distorted decoded images) with:

[h264 @ 0x102807000] out of range intra chroma pred mode at 0 0
[h264 @ 0x102807000] error while decoding MB 0 0
[h264 @ 0x102807000] negative number of zero coeffs at 66 4
[h264 @ 0x102807000] error while decoding MB 66 4
[h264 @ 0x102807000] top block unavailable for requested intra mode at 12 10
[h264 @ 0x102807000] error while decoding MB 12 10
[h264 @ 0x102807000] top block unavailable for requested intra mode at 78 14
[h264 @ 0x102807000] error while decoding MB 78 14
...

Replying to [comment:10 HenkDemper]:

Both {{{ vlc --demux avformat AvcIntra50NoFrames.mxf }}} and {{{ vlc AvcIntra50NoFrames.mxf }}} (so the MXF version) will identically fail (showing distorted decoded images) with:

Please provide this sample, I don't see it on http://ge.tt/7DYellQ2

Sorry, for some reason 7 out of 12 sample files where not uploaded correctly, ge.tt doesn't seem to like them... I have just updated the complete set on http://ge.tt/7DYellQ2, it took another 3 attempts to upload them all. Obviously for AvcIntra50NoFrames.mxf you should use 'Train (AVC-Intra, 50 Mbps, No Frames SPS-PPS Header).mxf'.

Replying to [comment:12 HenkDemper]:

for AvcIntra50NoFrames.mxf you should use 'Train (AVC-Intra, 50 Mbps, No Frames SPS-PPS Header).mxf'.

I created FFmpeg ticket 5029, thank you for the sample!

This would be easy to fix by forcing a width of 1440 in the demuxer but unfortunately, I don't see how the demuxer can distinguish between AVCI50 and AVCI100 (that needs width 1920) for the files you provided.

Which static PPS (out of 3 different) and SPS (out of 20 different) preset to use depends on 50 or 100 Mbps, format 720p/1080i/p and framerate 24/25/29.97/50/59.94. For the 'No Frames SPS-PPS Header' file variants these need to be derived from the MXF Header Metadata (Picture Descriptor & Track EditRate) or QuickTime 'moov' atom data (Image Description & Media/Track Timescale) and/or frame size in bytes (but some variants have equal framesize)... Not sure if all that info is available at the demuxer (if that is where the SPS/PPS will be added) in some form ?

Replying to [comment:14 HenkDemper]:

Which static PPS (out of 3 different) and SPS (out of 20 different) preset to use

So far the six sets present in libavformat/utils.c are sufficient for all samples we have (including the one that currently doesn't work), if you have a sample that needs another set, please upload;-)

depends on 50 or 100 Mbps

I don't think the bitrate information is present in the mxf file (or if it is I don't know where to search).

format 720p/1080i/p

Of course, the issue is that a wrong resolution is written in the file header / the same resolution as for a file that needs another SPS/PPS.

and framerate 24/25/29.97/50/59.94.

That information may be present (I didn't look) but I wonder if it would really be helpful?

For the 'No Frames SPS-PPS Header' file variants these need to be derived from the MXF Header Metadata (Picture Descriptor & Track EditRate)

Probably, but I don't know where to search...

[...]

I have a stash about that issue, that should builds sps/pps parameters instead of blindly applying arbitrary ones. Unfortunately, the ffmpeg ones make use of some unknown (reserved) fields according to my outdated spec.

I have requested to Panasonic Japan if we can use the static SPS/PPS tables that we have once received from them for VLC/ffmpeg open source. If the answer is positive, I think it is easier to discuss the issue. Stay tuned...

Replying to [comment:17 HenkDemper]:

I have requested to Panasonic Japan if we can use the static SPS/PPS tables that we have once received from them for VLC/ffmpeg open source.

If the answer is positive, I think it is easier to discuss the issue. Stay tuned...

How would that help? We have the necessary SPS/PPS but we don't know how to decide when to use it. If you see the difference between the AVCI50 and AVCI100 mxf "No frames" files, I can probably (quickly) fix the issue.

How would that help? I thought that if we have all 20 SPS variants listed (I saw less in ffmpeg), then it would be easier to see what would be needed in terms of metadata to get it right in all cases...

If you see the difference between the AVCI50 and AVCI100 mxf "No frames" files, I can probably (quickly) fix the issue. I see 2 differences:

The MPEG Video Descriptor has a field 'Video Coding Scheme ID' (a 16 byte Universal Label) that is different, in fact it covers most variants:

06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.20.01 H.264/MPEG-4 AVC High 10 Intra Unconstrained Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.01 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 1080/59.94i Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.02 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 1080/50i Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.03 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 1080/29.97p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.04 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 1080/25p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.08 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 720/59.94p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.21.09 H.264/MPEG-4 AVC High 10 Intra RP2027 Constrained Class 50 720/50p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.01 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 1080/59.94i Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.02 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 1080/50i Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.03 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 1080/29.97p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.04 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 1080/25p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.08 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 720/59.94p Coding
06.0E.2B.34.04.01.01.0A-04.01.02.02.01.32.31.09 H.264/MPEG-4 AVC High 422 Intra RP2027 Constrained Class 100 720/50p Coding

The (fixed) frame sizes are different for most of the 20 variants, like (sizes without 512 bytes SPS-PPS at the front):

AVC-Intra 100 Mbps, 1080i60: 472064 bytes
AVC-Intra 100 Mbps, 1080i50: 568320 bytes
AVC-Intra 50 Mbps, 1080i60: 232448 bytes
AVC-Intra 50 Mbps, 1080i50: 280576 bytes
...

I'm not sure if either or both of above information is available at the place you need it ?

Replying to [comment:19 HenkDemper]:

How would that help? I thought that if we have all 20 SPS variants listed (I saw less in ffmpeg), then it would be easier to see what would be needed in terms of metadata to get it right in all cases... As said, if you have a sample for which there is no working SPS/PPS set in FFmpeg, please provide it: For the failing AVCI50 Train sample, the issue was just how to choose the correct set without breaking the matching AVCI100 (Sport) sample.

If you see the difference between the AVCI50 and AVCI100 mxf "No frames" files, I can probably (quickly) fix the issue. I see 2 differences:

The MPEG Video Descriptor has a field 'Video Coding Scheme ID' (a 16 byte Universal Label) that is different, in fact it covers most variants:

Thank you for your help, patch sent!

patch sent!

Sounds good ! Can I see/track/test somewhere the progress of the patch ?

The patch will cover VLC-playback of all the MXF files... or also fix the VLC-playback QuickTime No-Frame SPS-PPS variants ?

Replying to [comment:21 HenkDemper]:

patch sent!

Sounds good ! Can I see/track/test somewhere the progress of the patch ?

http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/203944

Not much to test...

The patch will cover VLC-playback of all the MXF files...

Yes, when using --demux=ffmpeg

or also fix the VLC-playback QuickTime No-Frame SPS-PPS variants ?

I believe all samples work fine with --demux=ffmpeg, if one doesn't work, please tell me!

Replying to [comment:20 cehoyos]:

Thank you for your help, patch sent! I applied the patch, the file should work once vlc updates its internal copy of FFmpeg.

I applied the patch, the file should work once vlc updates its internal copy of FFmpeg Thanks for fixing, Carl Eugen !

I'll check all test files with resulting VLC when the FFmpeg-updated version becomes available...

Should/can I/you now update the status of this ticket ?

Replying to [comment:24 HenkDemper]:

Should/can I/you now update the status of this ticket ? I will not tamper with stati on this bug tracker;-)

This is not fixed for VLC/mp4.

It does seem that passing only AnnexB extradata or a derived avcC to avcodec is not sufficient as it only decodes one picture, then other are broken. Repeating the SPS/PPS before picture is no help either.

[h264 @ 0x7f8c94c5b560] top block unavailable for requested intra4x4 mode -1 at 96 6
[h264 @ 0x7f8c94c5b560] error while decoding MB 96 6
[h264 @ 0x7f8c94c5b560] out of range intra chroma pred mode at 24 10
[h264 @ 0x7f8c94c5b560] error while decoding MB 24 10
[h264 @ 0x7f8c94c5b560] out of range intra chroma pred mode at 74 13
[h264 @ 0x7f8c94c5b560] error while decoding MB 74 13
[h264 @ 0x7f8c94c5b560] top block unavailable for requested intra4x4 mode -1 at 0 17
[h264 @ 0x7f8c94c5b560] error while decoding MB 0 17
[h264 @ 0x7f8c94c5b560] negative number of zero coeffs at 49 20
[h264 @ 0x7f8c94c5b560] error while decoding MB 49 20
[h264 @ 0x7f8c94c5b560] top block unavailable for requested intra4x4 mode -1 at 96 23
[h264 @ 0x7f8c94c5b560] error while decoding MB 96 23
[h264 @ 0x7f8c94c5b560] negative number of zero coeffs at 25 27
[h264 @ 0x7f8c94c5b560] error while decoding MB 25 27
[h264 @ 0x7f8c94c5b560] top block unavailable for requested intra4x4 mode -1 at 72 30
[h264 @ 0x7f8c94c5b560] error while decoding MB 72 30

Ok, I found out, that was wong table, Sport (AVC-Intra, 100 Mbps, No Frames SPS-PPS Header).mov being 1080i Works sending it as avcC.

Btw, this is now fixed since 289691ce in 3.0

added Component::Demuxers Status::fixed labels

closed

QuickTime AVC-Intra 50/100 and MXF AVC-Intra 50 without SPS-PPS not playing

Child items 0

Activity