Native Rust GTOP benchmark results

This report collects the recorded GTOP tables in one Markdown document. All three tests use relaxed stopping values of approximately 1.005 * absolute_best for positive objectives and the corresponding 0.5% relaxation toward zero for negative objectives. The raw TSV files preserve every individual experiment.

The problem definitions and putative best solutions originate from ESA’s Global Trajectory Optimisation Problems database. The unusually difficult Tandem case is documented separately on ESA’s TandEM problem page, including its 18-variable box bounds and selectable fly-by sequences.

Coordinated retry

Messenger Full remains excluded because its full 100-run measurement is too expensive for this test. Tandem was measured separately with the same configuration and is included here.

Raw data: benchmark_gtop_100_raw.tsv and benchmark_gtop_tandem_100_raw.tsv. The separate benchmark_gtop_tandem_100_metadata.json records the slow-run configuration and total invocation time.

Configuration:

• 32 native retry threads and 100 independent experiments per problem
• 1,500 initial evaluations per retry
• advanced maximum evaluation factor 50
• diversity checkpoint interval 100 retries
• optimizer sequence: 40% Differential Evolution, then 60% active CMA-ES
• per-problem retry caps and value limits from examples/src/benchmark_gtop.rs

The coordinated test has an adaptive budget, not the 240,000-evaluation allowance used by the basic-retry and external-library comparison. A retry’s limit grows linearly from 1,500 to 75,000 evaluations as the run ID approaches the per-problem retry cap. The exact theoretical ceilings and actual recorded means are:

Every run consumed less than its theoretical ceiling. The 15 Tandem failures are included in its mean and sdev. Across all 100 Tandem runs, the mean final optimum was -1488.503885 (population sdev 21.850556); the best was -1500.468716.

The original Python/C++ fcmaes Performance report records 81% Tandem success, 166.92 s mean wall time, and 147.87 s wall-time sdev on the same AMD 9950X class of CPU with 32 parallel Python processes. The native Rust run records 85%, 40.21 s, and 39.11 s with 32 native retry threads. The close success rates are consistent with the implementations using the same adaptive search policy; the wall times also reflect their different process and thread execution models.

The substantially larger actual budgets explain why coordinated retry has much better success rates than the 240,000-evaluation comparison. This table is a quality reference, not an equal-budget speed comparison.

BiteOpt basic retry

This benchmark includes Tandem, excludes Messenger Full, and uses 24 independent BiteOpt retries of at most 10,000 evaluations on 24 native worker threads for every experiment.

Raw data: benchmark_biteopt_gtop_rust_100_raw.tsv.

DE→CMA basic retry

This test has the same 100 experiments, 24 workers, 24 retries, and 10,000 evaluations per retry as the BiteOpt test. Each retry assigns 4,000 evaluations to Differential Evolution and 6,000 to active CMA-ES.

Raw data: benchmark_de_cma_gtop_rust_100_raw.tsv.

Statistics and environment

Means and standard deviations include every experiment, including failures. Standard deviations use the population definition (ddof=0), matching NumPy’s default.

• CPU: AMD Ryzen 9 9950X, 16 physical cores / 32 logical CPUs
• Compiler: rustc 1.93.0 for the original six-problem run and rustc 1.97.1 for the later Tandem run; Cargo release profile
• OS: Linux 6.8.0-136-generic x86_64
• Date: 2026-07-22 for the original run; 2026-07-23 for Tandem

Rust workers are native threads. Each top-level experiment is run sequentially and has exclusive use of its configured retry workers.

Reproduce

From the repository root:

cargo run --release -p fcmaes-examples --bin benchmark-gtop -- \
  --runs 100 --workers 32 --seed 1 \
  --raw-output benchmarks/benchmark_gtop_100_raw.tsv

python3 benchmarks/run_coordinated_tandem.py

cargo run --release -p fcmaes-examples --bin benchmark-biteopt-gtop -- \
  --algo biteopt --runs 100 --workers 24 --retries 24 \
  --evaluations 10000 --seed 1 \
  --raw-output benchmarks/benchmark_biteopt_gtop_rust_100_raw.tsv

cargo run --release -p fcmaes-examples --bin benchmark-biteopt-gtop -- \
  --algo de_cma --runs 100 --workers 24 --retries 24 \
  --evaluations 10000 --seed 1 \
  --raw-output benchmarks/benchmark_de_cma_gtop_rust_100_raw.tsv

Each binary prints the corresponding Markdown table. --table-output PATH can also write that table to a chosen .md file. The Tandem driver records each experiment in a resumable shard before combining the final raw file. For coordinated retry, --include-slow adds Tandem and Messenger Full; --problem NAME runs one case explicitly.