Decide whether to cache frames or not based on actual render time
Rather than deciding manually whether a given combination of settings is 'trivial' (not worth caching) or not, we could make this decision based on the previous frame's render time (assuming the settings haven't changed).
Might be either more robust, or less predictable, depending on your point of view...