Software benchmarking is not at the level of testing. And that's a big problem for software quality in general.
At Stream HPC we continuously had to build our own tools for software benchmarking, as none of the available tools really worked. It started with some scripts, then various automation, then software, then integrations in more projects, etc. Since our weekly reports used graphs made by those same tools, we started to get questions and requests from customers. Since we only needed to add user-authorization, to make it available to customers, we started to do just that.
We're building the tool with a small team and are focused on requirements-discovery, based on projects we currently run.