Revert "Don't assume some default prefix is used"

This reverts commit ebf341dc.

We can't force a prefix at make install with DESTDIR
files are installed in $(DESTDIR)/$(prefix) not $(DESTDIR)

This also breaks OSX binaries, and thus test suite (is that libtool's fault?)