Tutorial result schema

The application tutorials write a small, versioned set of machine-readable artifacts. Native Rust performs simulation and optimization; Python reads these artifacts to create documentation figures. For model-fitting tutorials, native Rust likewise owns fitting, prediction, and metric evaluation.

Most individual optimizer runs use the manifest schema below. A tutorial may instead publish a documented aggregate evidence bundle when one figure combines multiple runs or validation studies. For example:

Room ventilation combines per-seed Pareto/archive and convergence CSVs with held-out evaluations, CFD fields, and a resolution study.
Neural policy search combines fixed/rotating-scenario comparisons, parallel-scaling repeats, baselines, monitor histories, a selected policy, and a frozen final test.

Both use a byte-for-byte checked plot_results.py. Aggregate bundles retain the same minimization, feasibility, full-precision CSV, seed, budget, and artifact-link conventions even when no single run.json can describe the combined figure.

Schema version 1 uses one run.json manifest per optimization run:

{
  "schema_version": 1,
  "tutorial": "rapier-trebuchet",
  "formulation": "mo",
  "command": "cargo run --release -- --mode multi ...",
  "seed": 42,
  "workers": 24,
  "requested_evaluations": 20000,
  "actual_evaluations": 20096,
  "elapsed_seconds": 12.34,
  "objectives": [
    {"column": "objective_target_error", "label": "Target error", "unit": "m"}
  ],
  "descriptors": [],
  "artifacts": {
    "pareto": "pareto.csv",
    "convergence": "convergence.csv"
  }
}

All objective values are minimized. Constraints are feasible at values less than or equal to zero. A manifest may add simulator-specific metadata, but must not change those conventions.

An independently executed comparison arm that cannot produce a feasible candidate is not omitted. If it ran, it writes status: "completed", its actual evaluations, and a zero feasible-result count. status: "skipped" with a machine-readable reason, actual_evaluations: null, and an empty artifacts object is reserved for an arm that was not executed. A frozen study plan may list such arms under excluded_arms; excluded arms are never evaluated on final-test data.

When one objective call expands into several simulations or model fits, the manifest should record both logical candidate calls and physical work. The HPO tutorial, for example, separates tuning_model_fits from selection_model_fits and also reports their total as model_fits.

Pareto data

pareto.csv contains one row per retained point:

point_id,feasible,selected,objective_*,constraint_*,decision_*

selected identifies tutorial representatives. More than one representative may be selected when the text discusses extremes and a compromise.

Quality-diversity data

qd_archive.csv contains one row per occupied niche:

niche_id,grid_x,grid_y,quality_train,quality_validation,
descriptor_*_train,descriptor_*_validation,visit_count,decision_*,constraint_*

For a regular two-dimensional archive, run.json includes qd.grid_shape = [columns, rows]. CVT archives instead export their center coordinates. Invalid or infeasible simulations are represented by non-finite fitness while optimizing and never become archive elites. Validation-aware tutorials may use selection_feasible and retained_niche to distinguish failure of a held-out constraint from movement to a different behavior niche.

When a QD archive is a machine-consumed control catalogue, publish decoded integer controls alongside normalized decisions. The phased-array codebook, for example, adds semicolon-separated phase_codes and attenuator_codes to codebook.csv; replay tests reconstruct the physical state from those codes. The image is a visualization, while the decoded table is the product.

Convergence data

The first columns are:

evaluations,elapsed_seconds

MODE runs report best_quality after those axes. It is a documented scalar summary used for convergence visualization, not a replacement for the Pareto front. Constrained MODE may add feasible_population and pareto_population; other metric columns must be named in run.json.convergence_metrics. MAP-Elites runs report coverage, qd_score, best_quality and invalid_fraction. Evaluation count, rather than generation number, is the common comparison axis.

Agent-guided route-search data

The GTOC1 route-search tutorial uses one schema-v1 manifest per agent, random, or evolutionary arm. Its full configuration object is the replayable invocation contract and includes grammar, optimizer, promotion, fidelity, transport, and result-path settings. budget separates proposal attempts, accepted L0 candidates, cache hits, requested/actual L0 and L1 evaluations, allocated worker-seconds, promotions, and token usage.

Its additional artifacts are:

archive.jsonl       checksummed append-only candidate and promotion revisions
archive.json        atomic latest-state snapshot
archive.csv         one row per logical candidate
proposal_log.jsonl  invalid, duplicate, repaired, diversity, and transport events
agent_log.jsonl     redacted requests/responses usable by replay transport
promotions.csv      paired L0/L1 scores, surrogate gap, threshold and failure
convergence.csv     accepted candidates, best L0/L1, niches and worker-seconds

L0 is a proxy and never establishes feasibility. An executed L1 arm with no threshold-passing refinement remains status: "completed" and records l1_threshold_passed: 0; budget exhaustion is not physical infeasibility. Allocated worker-seconds are wall time multiplied by resolved workers, not measured CPU time.

Reproducibility

Manifests record the exact command, optimizer seed, simulation or model training and validation seeds, effective workers, requested and actual budgets, elapsed wall time, descriptor bounds and relevant optimizer parameters. Values are written at full f64 precision; Markdown tables and plot labels perform display rounding.

Native experiment runner profile (source release 0.1)

The source-only fcmaes-native-worker uses this same schema, with tutorial: "fcmaes-native-worker" as the stable non-tutorial producer ID and formulation: "single_objective". All stored objectives remain minimized. best is either null or {x, y, retry_id}; feasible_results counts valid completed retries, not distinct mathematical feasible points. These initial objectives have no separately enforced constraints; finite objective values below 1e99 (the core’s replacement for NaN) qualify as valid.

Namespaced runner metadata records the resolved problem/bounds, optimizer, retry budget, execution settings, binary SHA-256, spec digest, attempt ID, build provenance, termination reason, completed retries and accounting status. The spec digest covers the resolved job specification except request_key, so identical experiments submitted under different keys share a digest. retries.jsonl is linked through artifacts.retries. The checksummed payload contains full-precision decisions/objective and the actual evaluation count. The job input schema (fcmaes.job/0.1) is an execution request, not another scientific result format.

Additional terminal status values are cancelled, timed_out, failed, and interrupted. Readers must not assume every manifest is completed. job.json contains transient submitted/starting/waiting/running lifecycle state; progress.json is a rebuildable completed-retry snapshot. Neither replaces run.json. A normally completed zero-valid-result arm remains completed. skipped retains the existing meaning of never executed.

If the worker returns cooperatively, actual_evaluations is exact for the attempt, including all retries it performed. A graceful cancel stops at the next retry-wave boundary and stays exact. After a force kill (jobs cancel --now, the wall-time deadline plus budget.grace_seconds), crash, or unresolved accounting, actual_evaluations is null; runner.known_evaluations_lower_bound includes only verified journal records and runner.accounting_complete is false. Do not replace null with the lower bound. Repair after launcher loss keeps the worker’s exact result when the orphaned worker finished on its own; otherwise it records interrupted with null elapsed time, because the execution duration is unknown. Queue duration is reported separately in job.json; it is excluded from the execution deadline.

BiteOpt advertises a hard per-retry cap. Serial DE uses a generation-granular soft cap with at most 2 * population - 1 extra evaluations per retry at the reviewed source revision. Requested and actual work must remain separate. Comparison of legacy manifests missing objective/data definitions is explicitly descriptive and warns that scientific compatibility has not been established. No objective transform, Pareto scaling or hypervolume reference is inferred.

Keyboard shortcuts