Surrogate validation: delivered M0 versus reviewed build
Generated by scripts/surrogate/analyze.py from examples/benchmark.rs and
examples/overhead.rs outputs. M0 is the delivered code (4dd693b); now is
the reviewed build: bounded L-BFGS MAP fit with analytic gradients, packed
row-major Cholesky, a trust region for two of three proposals and nugget 1e-10
(M0: 1e-8). The comparison with other libraries is in
benchmarks/surrogate-comparison.
Host: x86_64 Linux, 2 usable CPUs, rustc 1.97.0 (2d8144b78 2026-07-07).
Timings come from a shared host and are observations, not portable guarantees.
Proposal overhead
examples/overhead.rs: one propose call on a Levy-like objective over an LHS
history of n points in d dimensions, default configuration, median of three seeds.
Fit is a round with a fresh MAP fit (two restarts) plus the acquisition for
q points; carried reuses the hyperparameters (no fit). NLP is the fitted negative
log posterior (lower is better). Times are the minimum over alternating M0/now
repetitions where several were recorded.
1 installed thread
| d | n | q | fit ms M0 | fit ms now | speedup | carried ms M0 | carried ms now | speedup | NLP M0 | NLP now |
|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 32 | 1 | 2.3 | 1.2 | 1.9x | 0.8 | 0.7 | 1.2x | 14.521 | 14.505 |
| 2 | 32 | 8 | 7.9 | 5.8 | 1.4x | 6.3 | 5.3 | 1.2x | 14.521 | 14.505 |
| 2 | 128 | 1 | 39.0 | 15.9 | 2.5x | 6.1 | 3.8 | 1.6x | -179.478 | -179.594 |
| 2 | 128 | 8 | 61.5 | 38.7 | 1.6x | 33.8 | 25.5 | 1.3x | -179.478 | -179.594 |
| 2 | 248 | 1 | 148.3 | 79.4 | 1.9x | 14.6 | 10.2 | 1.4x | -589.062 | -589.289 |
| 2 | 248 | 8 | 224.0 | 151.3 | 1.5x | 96.8 | 72.3 | 1.3x | -589.062 | -589.289 |
| 5 | 32 | 1 | 4.8 | 2.0 | 2.5x | 1.0 | 1.1 | 1.0x | 35.684 | 35.673 |
| 5 | 32 | 8 | 12.1 | 9.4 | 1.3x | 8.4 | 8.3 | 1.0x | 35.684 | 35.673 |
| 5 | 128 | 1 | 81.2 | 18.6 | 4.4x | 9.2 | 4.9 | 1.9x | 99.690 | 99.612 |
| 5 | 128 | 8 | 130.8 | 52.9 | 2.5x | 54.8 | 38.9 | 1.4x | 99.690 | 99.612 |
| 5 | 248 | 1 | 413.6 | 115.1 | 3.6x | 21.9 | 15.1 | 1.5x | 130.133 | 130.020 |
| 5 | 248 | 8 | 525.4 | 209.8 | 2.5x | 152.8 | 108.7 | 1.4x | 130.133 | 130.020 |
| 10 | 32 | 1 | 12.1 | 3.2 | 3.7x | 2.0 | 1.9 | 1.0x | 49.148 | 49.046 |
| 10 | 32 | 8 | 27.3 | 15.6 | 1.8x | 16.8 | 16.3 | 1.0x | 49.148 | 49.046 |
| 10 | 128 | 1 | 197.8 | 33.0 | 6.0x | 14.8 | 9.0 | 1.6x | 152.486 | 152.304 |
| 10 | 128 | 8 | 288.5 | 91.8 | 3.1x | 92.2 | 66.4 | 1.4x | 152.486 | 152.304 |
| 10 | 248 | 1 | 896.0 | 187.7 | 4.8x | 33.1 | 23.5 | 1.4x | 250.468 | 250.048 |
| 10 | 248 | 8 | 1164.4 | 345.4 | 3.4x | 265.1 | 196.9 | 1.3x | 250.468 | 250.048 |
| 20 | 32 | 1 | 31.5 | 5.1 | 6.2x | 4.2 | 3.7 | 1.1x | 70.054 | 69.995 |
| 20 | 32 | 8 | 69.3 | 32.4 | 2.1x | 40.5 | 30.5 | 1.3x | 70.054 | 69.995 |
| 20 | 128 | 1 | 545.2 | 62.1 | 8.8x | 30.6 | 17.2 | 1.8x | 168.387 | 167.740 |
| 20 | 128 | 8 | 703.5 | 186.5 | 3.8x | 198.2 | 139.0 | 1.4x | 168.387 | 167.740 |
| 20 | 248 | 1 | 2289.9 | 299.4 | 7.6x | 64.6 | 46.6 | 1.4x | 302.929 | 301.935 |
| 20 | 248 | 8 | 2636.8 | 631.4 | 4.2x | 524.3 | 357.0 | 1.5x | 302.929 | 301.935 |
| 32 | 32 | 1 | 67.9 | 9.4 | 7.2x | 8.9 | 7.9 | 1.1x | 90.640 | 90.312 |
| 32 | 32 | 8 | 137.5 | 65.8 | 2.1x | 77.7 | 64.3 | 1.2x | 90.640 | 90.312 |
| 32 | 128 | 1 | 1151.7 | 90.8 | 12.7x | 55.6 | 33.4 | 1.7x | 177.059 | 175.827 |
| 32 | 128 | 8 | 1530.2 | 347.4 | 4.4x | 384.5 | 281.5 | 1.4x | 177.059 | 175.827 |
| 32 | 248 | 1 | 4592.3 | 498.4 | 9.2x | 111.5 | 86.6 | 1.3x | 243.343 | 241.918 |
| 32 | 248 | 8 | 5572.4 | 1087.9 | 5.1x | 963.6 | 664.5 | 1.5x | 243.343 | 241.918 |
The current fit reaches an equal or lower NLP in 46 of 50 configurations (all sizes, including those not shown).
2 installed threads
| d | n | q | fit ms M0 | fit ms now | speedup | carried ms M0 | carried ms now | speedup | NLP M0 | NLP now |
|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 32 | 1 | 1.7 | 1.0 | 1.6x | 0.6 | 0.6 | 1.0x | 14.521 | 14.505 |
| 2 | 32 | 8 | 5.6 | 4.1 | 1.4x | 4.3 | 3.8 | 1.1x | 14.521 | 14.505 |
| 2 | 128 | 1 | 22.5 | 9.3 | 2.4x | 4.4 | 2.3 | 1.9x | -179.478 | -179.594 |
| 2 | 128 | 8 | 44.1 | 25.4 | 1.7x | 19.7 | 14.6 | 1.3x | -179.478 | -179.594 |
| 2 | 248 | 1 | 85.0 | 47.2 | 1.8x | 8.7 | 6.5 | 1.3x | -589.062 | -589.289 |
| 2 | 248 | 8 | 140.2 | 97.8 | 1.4x | 68.0 | 44.4 | 1.5x | -589.062 | -589.289 |
| 5 | 32 | 1 | 3.6 | 1.2 | 3.0x | 0.9 | 0.7 | 1.2x | 35.684 | 35.673 |
| 5 | 32 | 8 | 9.9 | 5.9 | 1.7x | 6.1 | 5.2 | 1.2x | 35.684 | 35.673 |
| 5 | 128 | 1 | 44.1 | 12.4 | 3.6x | 5.3 | 3.8 | 1.4x | 99.690 | 99.612 |
| 5 | 128 | 8 | 70.5 | 31.2 | 2.3x | 32.0 | 21.8 | 1.5x | 99.690 | 99.612 |
| 5 | 248 | 1 | 235.5 | 58.7 | 4.0x | 14.1 | 10.0 | 1.4x | 130.133 | 130.020 |
| 5 | 248 | 8 | 261.2 | 124.3 | 2.1x | 86.6 | 66.2 | 1.3x | 130.133 | 130.020 |
| 10 | 32 | 1 | 7.3 | 2.7 | 2.7x | 1.5 | 1.4 | 1.1x | 49.148 | 49.046 |
| 10 | 32 | 8 | 20.4 | 10.7 | 1.9x | 9.6 | 9.1 | 1.1x | 49.148 | 49.046 |
| 10 | 128 | 1 | 109.7 | 24.8 | 4.4x | 8.2 | 6.5 | 1.2x | 152.486 | 152.304 |
| 10 | 128 | 8 | 146.2 | 51.3 | 2.8x | 51.5 | 37.4 | 1.4x | 152.486 | 152.304 |
| 10 | 248 | 1 | 445.6 | 100.8 | 4.4x | 20.1 | 13.8 | 1.5x | 250.468 | 250.048 |
| 10 | 248 | 8 | 628.0 | 192.3 | 3.3x | 154.9 | 103.2 | 1.5x | 250.468 | 250.048 |
| 20 | 32 | 1 | 16.0 | 4.0 | 4.0x | 2.5 | 2.7 | 0.9x | 70.054 | 69.995 |
| 20 | 32 | 8 | 36.2 | 21.0 | 1.7x | 21.3 | 18.5 | 1.2x | 70.054 | 69.995 |
| 20 | 128 | 1 | 266.5 | 39.3 | 6.8x | 16.3 | 11.3 | 1.4x | 168.387 | 167.740 |
| 20 | 128 | 8 | 361.9 | 116.0 | 3.1x | 112.5 | 100.4 | 1.1x | 168.387 | 167.740 |
| 20 | 248 | 1 | 1213.2 | 201.7 | 6.0x | 42.6 | 29.2 | 1.5x | 302.929 | 301.935 |
| 20 | 248 | 8 | 1510.4 | 370.5 | 4.1x | 329.6 | 227.6 | 1.4x | 302.929 | 301.935 |
| 32 | 32 | 1 | 37.2 | 6.4 | 5.9x | 5.2 | 4.6 | 1.1x | 90.640 | 90.312 |
| 32 | 32 | 8 | 88.3 | 39.6 | 2.2x | 45.0 | 40.9 | 1.1x | 90.640 | 90.312 |
| 32 | 128 | 1 | 624.9 | 55.5 | 11.3x | 30.0 | 22.9 | 1.3x | 177.059 | 175.827 |
| 32 | 128 | 8 | 811.2 | 198.5 | 4.1x | 225.7 | 164.1 | 1.4x | 177.059 | 175.827 |
| 32 | 248 | 1 | 2645.2 | 270.8 | 9.8x | 84.0 | 50.6 | 1.7x | 243.343 | 241.918 |
| 32 | 248 | 8 | 3005.1 | 585.8 | 5.1x | 565.7 | 385.2 | 1.5x | 243.343 | 241.918 |
The current fit reaches an equal or lower NLP in 46 of 50 configurations (all sizes, including those not shown).
Quality, branin, hartmann3
5 seeds, batch size 4, 2 installed threads; branin 48 calls, hartmann3 48 calls. All methods share the eight-point centre/LHS start. Simple regret = best found minus known minimum (lower is better). Proposal seconds are the GP’s total time in propose per run.
| Problem | Method | Median regret | Mean regret | Median regret M0 | Mean regret M0 | Seeds where now < M0 | Proposal s/run | Proposal s/run M0 |
|---|---|---|---|---|---|---|---|---|
| branin | gp | 3.52e-07 | 1.239e-06 | 5.739e-06 | 8.957e-06 | 4 of 5 | 0.022 | 0.021 |
| branin | random | 0.6437 | 0.7301 | |||||
| branin | cmaes | 0.1962 | 0.3342 | |||||
| branin | biteopt | 0.1108 | 0.5061 | |||||
| hartmann3 | gp | 2.487e-07 | 3.628e-07 | 7.698e-06 | 1.466e-05 | 5 of 5 | 0.023 | 0.028 |
| hartmann3 | random | 0.4682 | 0.4605 | |||||
| hartmann3 | cmaes | 0.1381 | 0.1433 | |||||
| hartmann3 | biteopt | 0.01668 | 0.07834 |
Quality, hartmann6, styblinski10
20 seeds, batch size 4, 2 installed threads; hartmann6 96 calls, styblinski10 128 calls. All methods share the eight-point centre/LHS start. Simple regret = best found minus known minimum (lower is better). Proposal seconds are the GP’s total time in propose per run.
| Problem | Method | Median regret | Mean regret | Median regret M0 | Mean regret M0 | Seeds where now < M0 | Proposal s/run | Proposal s/run M0 |
|---|---|---|---|---|---|---|---|---|
| hartmann6 | gp | 1.056e-06 | 0.03577 | 9.68e-05 | 0.0454 | 20 of 20 | 0.137 | 0.179 |
| hartmann6 | random | 1.272 | 1.282 | |||||
| hartmann6 | cmaes | 0.4147 | 0.4761 | |||||
| hartmann6 | biteopt | 0.4347 | 0.5825 | |||||
| styblinski10 | gp | 57.14 | 62.55 | 87.85 | 86.95 | 15 of 20 | 0.365 | 0.726 |
| styblinski10 | random | 153.1 | 150.1 | |||||
| styblinski10 | cmaes | 128.3 | 123.8 | |||||
| styblinski10 | biteopt | 35.91 | 52.49 |
Quality, oscillator
20 seeds, batch size 4, 2 installed threads; oscillator 32 calls. All methods share the eight-point centre/LHS start. Simple regret = best found minus known minimum (lower is better). Proposal seconds are the GP’s total time in propose per run.
| Problem | Method | Median regret | Mean regret | Median regret M0 | Mean regret M0 | Seeds where now < M0 | Proposal s/run | Proposal s/run M0 |
|---|---|---|---|---|---|---|---|---|
| oscillator | gp | 5.518e-07 | 8.772e-06 | 1.981e-06 | 5.025e-06 | 13 of 20 | 0.009 | 0.008 |
| oscillator | random | 0.005749 | 0.009219 | |||||
| oscillator | cmaes | 0.001843 | 0.006996 | |||||
| oscillator | biteopt | 0.005304 | 0.006558 |
Baseline methods are unchanged code: all 210 paired random/CMA-ES/BiteOpt histories are identical between the two builds.
Thread-count determinism
newmedium: all 160 runs have identical observation histories on 1, 2 installed threads.newoscillator: all 80 runs have identical observation histories on 1, 2 installed threads.newsmall: all 40 runs have identical observation histories on 1, 2, 8 installed threads.
Scope
Small deterministic test functions with known minima (the oscillator is the
tutorial’s RK4 calibration), not the expensive Rust simulations the profile targets.
Per-seed counts, no significance tests. The equal-budget comparison with BoTorch,
egobox, scikit-optimize and Optuna is in benchmarks/surrogate-comparison.