Background: Large language models (LLMs) are solely judged by their scores on biomedical benchmarks, yet these scores are rarely reproducible since the prompts, inference settings and model builds are seldom reported. We developed BenchmarkHub, an open-source platform for auditable evaluation of language and vision-language models for biomedical research. Methods: BenchmarkHub is a Python web application (Django 5.2, SQLite) released under the MIT licence and available at https://github.com/juanjuanignacio/benchmarkhub. The application can be accessed through a browser interface, a set of command-line tools and a REST API, so interactive exploration and scripted pipelines produce identical, comparable records. The platform samples questions with fixed seeds, runs models directly or as tool-using agents in a code sandbox, grades answers with deterministic rules and stores every prompt, reply and tool call. The interface provides a leaderboard to rank performance of the models, allowing model comparison without any preexisting programming knowledge. Findings: We used BenchmarkHub to evaluate 12 open-weight models in five experiments on 15 biomedical benchmarks (11 text, two multimodal and two computational biology) and two general benchmarks. The pooled accuracy ranged from 46.1% (95% CI 43.0-49.3) to 80.9% (78.3-83.3). Re-grading the responses with simple answer-extraction rules lowered a model's accuracy by up to 33.1 percentage points, leading to a reorder of the leaderboard. Published scores were found for 23 out of the 143 benchmark/model combinations tested. The accuracy measured by BenchmarkHub was lower than in the published results in 19 out of the 23 available comparisons. The effect of four prompting protocols (1: default prompt; 2: asking the model to answer the letter only; 3: chain-of-thought prompting; 4: three-shot prompting) changed MedQA accuracy by up to 11.3 points but was not significant after multiple-testing correction with 150 questions. We also tested the impact of ablation on benchmarking using the MMMU benchmark dataset. Removing images lowered the accuracy by 18.0 and 19.3 points respectively for the two strongest models. Finally, the accuracy of agentic benchmarks using a sandboxed interpreter increased on computational biology tasks by 36-63 points. Interpretation: Benchmarking is essential to assess the impact and validity of LLMs for medical applications. BenchmarkHub allows a fair and rigorous comparison of models across established datasets, including the possibility to upload their own datasets. The suite includes visual tools to ease model comparison and ranking without any programming knowledge. Self-hosted infrastructure that records every choice and parameter makes benchmark claims checkable without data leaving the institution.