CI: don't rebuild contribs when a non-existing folder changes

There's no extras for Linux, so we don't need to rebuild when anything changes in extra/package.

It should produce the same result as now, but it's cleaner.