This was inefficient and caused attachments from the meta engine or from
the slave demuxers to get lost.

Note that those attachments will still get lost later if the master
demuxer sets the metadata update flag (which affects the Ogg demuxer
only so far), or when adding a slave input.

All regressions introduced in e4673369.