memstream: make the error handling private

We can use a special value in length to signal an error internally. The length (and pointer) are only supposed to be valid on success.

We can use less memory for the memstream on platforms where open_memstream is not supported.

Extracted from !4590