tools: Improve demuxer probing

Increase the probing size, and change the logic to assume a stream is
valid even if no conclusive decision could be made within the probing
window as long as a sequence header was detected.