demux: abort module loading upon search string creation failure

if in "any" mode and falling back to an extension based match, and if the
`asprintf()` call fails to create the extension based shortcut search
string, the consensus is that it should give up rather than proceed to try
an "any" based demux module load.

note that the `modbuf = NULL` statement was just resetting the undefined
state of the variable from the failed `asprintf()` call to ensure that the
later `free(modbuf)` was valid. it is not needed now that we go down the
error path.

note, the init of `s->psz_filepath` in `vlc_stream_CustomNew()` is not
strictly necessary to avoid `free()` on an initialised attribute, since
`calloc()` is actually involved in the object creation - much of the
initialisation done in `vlc_stream_CustomNew()` is actually redundant.