Native benchmark results
Human-facing benchmark reports in this directory use Markdown. Raw TSV files preserve every experiment and are the authoritative inputs for statistics.
| Workload | Report | Raw samples |
|---|---|---|
| Coordinated retry, BiteOpt retry, and DE→CMA retry on GTOP | benchmark_gtop.md | benchmark_gtop_100_raw.tsv, benchmark_gtop_tandem_100_raw.tsv, benchmark_biteopt_gtop_rust_100_raw.tsv, benchmark_de_cma_gtop_rust_100_raw.tsv |
| fcmaes versus independent Rust optimizer crates | optimizer-comparison/comparison.md | raw artifacts |
| Core boundary: DE versus Nelder–Mead refinement and EGO at equal wall deadlines | optimizer-boundary/results/decision-v2/comparison.md | refiner-raw.tsv, bo-trace.tsv |
Recreate the recorded native fcmaes workloads from the repository root:
cargo run --release -p fcmaes-examples --bin benchmark-gtop -- \
--runs 100 --workers 32 --seed 1 \
--raw-output benchmarks/benchmark_gtop_100_raw.tsv
python3 benchmarks/run_coordinated_tandem.py
cargo run --release -p fcmaes-examples --bin benchmark-biteopt-gtop -- \
--algo biteopt --runs 100 --workers 24 --retries 24 \
--evaluations 10000 --seed 1
cargo run --release -p fcmaes-examples --bin benchmark-biteopt-gtop -- \
--algo de_cma --runs 100 --workers 24 --retries 24 \
--evaluations 10000 --seed 1
The binaries print Markdown tables and accept --table-output PATH when a
separate generated .md file is useful.
The recorded slow Tandem run also includes
benchmark_gtop_tandem_100_metadata.json
with its exact configuration and total invocation time.
Run the dependency-isolated optimizer comparison with:
benchmarks/optimizer-comparison/run_all_external.sh
Reproduce the dependency-isolated optimizer-boundary experiment using the
commands in its protocol. It keeps the
experiment-only egobox-ego and Nelder–Mead implementations outside the root
workspace and fcmaes-core dependency graph.
Wall times depend strongly on CPU, operating system, compiler version, and background load. Treat recorded timings as reproducibility data for the stated machine, not as universal performance guarantees.