Index

The navix leaderboard's Benchmark protocol (issue #130): scores an AlgorithmEntry against a preset, e.g. Navix1M().run(entry).

This package is split into benchmark.py (TrainingCurve, what AlgorithmEntry.train returns - the only requirement; __post_init__ checks this, so a malformed train fails at construction time, not partway through a real benchmark run; CostAnalysis; BenchmarkResult - a TrainingCurve plus the wall-clock timing and cost only an external, un-jitted wrapper can measure; and the abstract Benchmark base every protocol implements), hardware.py (the hardware-detection functions AlgorithmEntry auto-populates itself from), scratch.py (FromScratchBenchmark and its Navix1M/Navix100K presets - the one concrete protocol so far), search.py (search_hparams - an optional, Benchmark-independent Evolution-Strategies search an entry's own run.py can use to tune its hyperparameters before scoring; not part of Benchmark/AlgorithmEntry itself, since what's searchable is inherently entry-specific - benchmark.py also has Benchmark. plot_summary/plot_details/plot_diagnostics for locally inspecting a scored run's summary/details/diagnostics.npz without the online leaderboard, implemented directly on Benchmark itself rather than delegating to standalone functions, since that data is specific to Benchmark's own shapes), and plotting.py (a local no-wandb dashboard for logs - see Agent's docstring and issue #60, unrelated to Benchmark scoring - plus the small generic formatting/detection helpers Benchmark.plot_summary/plot_details/plot_diagnostics build their figures from). A future protocol (curriculum learning, continual learning, open-ended learning) gets its own file alongside scratch.py, subclassing Benchmark directly.

Typical usage:

entry = MyEntry(name=..., author=..., paper_url=...,
                 navix_commit_url=..., algorithm_commit_url=...)
# entry's construction already checked MyEntry.train returns a
# TrainingCurve
benchmark = Navix1M()
results = benchmark.run(entry)
summary = benchmark.summary(results)
benchmark.submit_entry(entry, results)

See benchmarks/README.md for the full submission workflow.

`MANDATORY_METRICS = {'agent/episode/returns': 'Episodic Return', 'agent/episode/success_rate': 'Success Rate', 'agent/episode/length': 'Episode Length', 'experiment/costs/fps': 'Training Throughput (steps/s)', 'experiment/costs/wall_time': 'Wall-clock Training Time (s)'}` `module-attribute`

The plots every navix agent's logs should support, so results are directly comparable across algorithms. Kept intentionally small: only metrics that (a) exist regardless of which algorithm produced logs, and (b) are actually necessary to tell whether training worked at all.

`AlgorithmEntry` `dataclass`

One algorithm to score against a Benchmark.

To submit an algorithm: subclass AlgorithmEntry, override train to build whatever model env_id needs and train it and return a TrainingCurve - the only requirement - then construct an instance with the provenance fields below (see benchmarks/README.md). The hardware fields (gpu_type through jaxlib_version) are auto-detected in __post_init__, not constructor arguments. __post_init__ also checks that train returns a TrainingCurve with the right shape, so a malformed train fails at construction time, not partway through a real benchmark run.

Attributes:

Name	Type	Description
`name`	`str`	Algorithm name, e.g. "PPO".
`author`	`str`	This implementation's author (GitHub handle), not the paper's. Validated in `__post_init__`.
`paper_url`	`str`	Link to the paper the algorithm is from.
`navix_commit_url`	`str`	Link to the navix commit this result was produced against (issue #130's `navix.sha`, as a URL). Validated in `__post_init__`.
`algorithm_commit_url`	`str`	Link to the algorithm implementation's own commit (issue #130's `agent.sha`, as a URL) - same commit as `navix_commit_url` for a navix-shipped agent, a different repo's commit for an external one. Validated in `__post_init__`.
`gpu_type`	`Optional[str]`	The GPU model JAX runs on, or `None` if JAX isn't running on a GPU.
`cpu_type`	`str`	The CPU's model name.
`ram_bytes`	`int`	Total system RAM, in bytes.
`cuda_version`	`Optional[str]`	The CUDA version jaxlib runs on, or `None` if JAX isn't running on a GPU.
`cudnn_version`	`Optional[str]`	The cuDNN version jaxlib runs on, or `None` if JAX isn't running on a GPU.
`jax_version`	`str`	`jax.__version__`.
`jaxlib_version`	`str`	`jaxlib.__version__`.

`cost_analysis(env_id, budget)`

Compiles self.train(env_id, budget, ...) and reads its FLOPs/memory/compile-time. Always seed 0, since cost is shape-driven, not value-driven.

Necessarily includes whatever env interaction train does internally. In practice this lands close to "one update's cost", not the whole run's: every navix-shipped agent's train is init + jax.lax.scan(self.update, ..., length=num_updates), and XLA's cost_analysis() on a compiled scan reports one iteration's cost, not length copies. An agent whose train isn't scan-shaped will report a different figure here.

Parameters:

Name	Type	Description	Default
`env_id`	`str`	The environment to build the model for.	required
`budget`	`int`	Training budget, passed to `self.train`.	required

Returns:

Name	Type	Description
`CostAnalysis`	`CostAnalysis`	FLOPs, peak memory, and compile time for one
	`CostAnalysis`	`self.train(env_id, budget, ...)` call.

`train(env_id, budget, rng)`

Builds a fresh, env-shaped model for env_id and trains it at budget. The one method every submission must override - building the model is inherently algorithm-specific, so there's no protocol-agnostic default.

Runs inside a jax.jit/jax.vmap trace (see Benchmark. run_env), so only genuinely jittable work belongs here - wall-clock timing and cost are measured separately, from outside any trace (see BenchmarkResult, cost_analysis).

Parameters:

Name	Type	Description	Default
`env_id`	`str`	The environment to train on.	required
`budget`	`int`	Training budget - whatever the running `Benchmark` protocol passed to `Benchmark.run_env`. Use it to build your hparams (e.g. `PPOHparams(budget=budget)`) if your algorithm's training length should respect it.	required
`rng`	`Array`	The PRNG key to train with.	required

Returns:

Name	Type	Description
`TrainingCurve`	`TrainingCurve`	`episodic_returns`/`lengths` filled in;
	`TrainingCurve`	`diagnostics` optionally, with whatever per-update values
	`TrainingCurve`	help debug this algorithm.

Raises:

Type	Description
`NotImplementedError`	Always, on `AlgorithmEntry` itself - must be overridden.

`validate_train_contract()`

Checks self.train returns a TrainingCurve with the right shape, without running any real training - jax.eval_shape traces self.train for its output structure only, against one representative registered environment. A single (un-vmapped) call, so episodic_returns/lengths/diagnostics' values must all be rank 1 (one point per update).

Raises:

Type	Description
`TypeError`	If `self.train`'s output isn't a `TrainingCurve`.
`AssertionError`	If `episodic_returns`/`lengths`/ `diagnostics`' values aren't rank 1 (`chex.assert_rank`).

`Benchmark`

Bases: PyTreeNode

An experimental protocol - the fixed set of choices that make two algorithms' numbers comparable: which environments, what frame budget, how many seeds, and how a run is scored and summarised. It says nothing about how an algorithm is implemented, only how it is measured, so any AlgorithmEntry can be run against it and against the literature.

Use an instance, e.g. Navix1M().run(entry); the protocol's name is type(self).__name__. A concrete protocol (e.g. FromScratchBenchmark, behind Navix1M / Navix100K) supplies run / summary / details; submit_entry is shared - it writes out whatever those produced, identically for every protocol.

`details(results)`

Reduces a BenchmarkResult from run into per-row diagnostics about this benchmark run - the same kind of columns summary aggregates, but one row per whatever this protocol's leading axis represents (e.g. one row per environment for FromScratchBenchmark), instead of a single further-aggregated row. Diagnostics about the benchmark run itself, not a leaderboard's per-algorithm click-through page - what that page shows depends on the algorithm, not the benchmark protocol.

A concrete override should include whatever labels row identity (e.g. env_ids) as one of its own returned entries, if this protocol has a meaningful one - submit_entry writes out exactly what this returns and nothing more. Values aren't required to be Array (e.g. env_ids is a Tuple[str, ...], not every metric need be jax-typed) - unlike summary, this isn't reduced to a uniform type.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
`Dict[str, Any]`	Dict[str, Any]: One row per unit of this protocol's
`Dict[str, Any]`	leading axis - which columns exist, and what that axis is,
`Dict[str, Any]`	is protocol-specific.

Raises:

Type	Description
`NotImplementedError`	Always, on `Benchmark` itself - must be overridden by a concrete protocol.

`plot_details(results)`

self.details(results) as a local, offline bar-chart figure, one panel per numeric metric, one bar per row (e.g. one bar per environment for FromScratchBenchmark) - mean plus a std-dev error bar over whatever trailing axis details keeps raw (e.g. FromScratchBenchmark keeps every seed's own value, unlike summary's already-averaged numbers - see FromScratchBenchmark.details's docstring).

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
	matplotlib.figure.Figure: One panel per numeric metric.

`plot_diagnostics(results)`

results.curve's raw training curves as a local, offline figure, one panel per curve (episodic_returns, length, any curve.diagnostics entries) - mean line plus a min-max band over self.seeds, same convention as navix.benchmarks. plotting.plot_metric. Same curves submit_entry writes (resampled) into diagnostics.npz, but at full resolution and without needing a file round-trip.

A TrainingCurve doesn't carry absolute frame counts (unlike the raw logs pytree plot_metric plots), so the x-axis is training progress as a 0-100% fraction of however many points the curve has, not a frame count.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
	matplotlib.figure.Figure: One panel per curve.

`plot_summary(results)`

self.summary(results) as a local, offline metric/value table figure. Independent of whatever charts the online leaderboard renders from the same summary.json.

A bar chart would be misleading here: summary's metrics live on wildly different scales in the same dict (episodic returns in [0, 1] next to flops in the hundreds of millions), so a table keeps every value legible without implying they're comparable.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
	matplotlib.figure.Figure: The table figure.

`run(entry)`

Trains entry under this protocol.

Parameters:

Name	Type	Description	Default
`entry`	`AlgorithmEntry`	The algorithm to score.	required

Returns:

Name	Type	Description
`BenchmarkResult`	`BenchmarkResult`	However many things this protocol
	`BenchmarkResult`	measures per run, stacked along a leading axis - which
	`BenchmarkResult`	axis, and what it represents (e.g. one row per environment
	`BenchmarkResult`	for `FromScratchBenchmark`), is protocol-specific.

Raises:

Type	Description
`NotImplementedError`	Always, on `Benchmark` itself - must be overridden by a concrete protocol.

`run_env(entry, env_id, budget)`

Trains entry on env_id at budget, vmapped over self.seeds, timing the run (compile time excluded) and reading its cost.

Parameters:

Name	Type	Description	Default
`entry`	`AlgorithmEntry`	The algorithm to train.	required
`env_id`	`str`	The environment to train on.	required
`budget`	`int`	Training budget, passed to `entry.train`/ `entry.cost_analysis`.	required

Returns:

Name	Type	Description
`BenchmarkResult`	`BenchmarkResult`	`curve` is `entry.train`'s output, one
	`BenchmarkResult`	per seed stacked along a new leading axis. `wall_time`/
	`BenchmarkResult`	`fps`/`cost` are each a single scalar for the whole
	`BenchmarkResult`	vmapped call - seeds train together in one fused
	`BenchmarkResult`	computation, so there's no meaningful per-seed timing
	`BenchmarkResult`	breakdown.

`submit_entry(entry, results, max_points=50, subdir='')`

Writes one self.run(entry)'s output into the directory of whichever script called this - the same convention every submission's run.py already follows for config.yml/requirements.txt, so a submission's results end up right alongside them.

Writes three files:

summary.json: entry's provenance/hardware fields plus self.summary(results) - the leaderboard's table row.
details.json: self.details(results) - per-row diagnostics about this run.
diagnostics.npz: results itself. curve.episodic_returns/ curve.lengths/curve.diagnostics' values (written as benchmark/episode/returns/benchmark/episode/length, plus curve.diagnostics' own keys unchanged - see curve_diagnostics) are resampled to exactly max_points evenly-spaced points along their trailing axis, so every submission's curve fields end up the same fixed shape regardless of how many updates actually ran; benchmark/costs/* (wall_time/fps/cost.*) are already scalars and are written as-is.

Parameters:

Name	Type	Description	Default
`entry`	`AlgorithmEntry`	The algorithm that produced `results`.	required
`results`	`BenchmarkResult`	This protocol's `run` output.	required
`max_points`	`int`	Number of points `curve.episodic_returns`/ `curve.lengths`/`curve.diagnostics`' values are resampled to in `diagnostics.npz`.	`50`
`subdir`	`str`	If non-empty, writes into a subdirectory of the caller's own directory instead of the directory itself (created if missing). For a `run.py` that scores the same algorithm under more than one configuration it doesn't expose as a structured `AlgorithmEntry` field (e.g. observation type - see `navix.benchmarks.search`'s module docstring for the same reasoning applied to hyperparameters: what's configurable is inherently entry-specific, so `Benchmark` stays agnostic to it) - one `submit_entry` call per configuration, each into its own `subdir`, instead of colliding on one shared `summary.json`.	`''`

`summary(results)`

Reduces a BenchmarkResult from run into a leaderboard's table row for one algorithm entry.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
`Dict[str, Array]`	Dict[str, Array]: The named columns a leaderboard's table
`Dict[str, Array]`	row shows for this entry - which columns exist is
`Dict[str, Array]`	protocol-specific.

Raises:

Type	Description
`NotImplementedError`	Always, on `Benchmark` itself - must be overridden by a concrete protocol.

`BenchmarkResult`

Bases: PyTreeNode

One AlgorithmEntry's scored run under a Benchmark protocol - a TrainingCurve plus everything only an external, un-jitted wrapper can measure. Built by Benchmark.run_env in one shot, from three independently-measured pieces.

Attributes:

Name	Type	Description
`curve`	`TrainingCurve`	`AlgorithmEntry.train`'s output.
`wall_time`	`Array`	Real wall-clock time to execute the already-compiled `AlgorithmEntry.train` (all of `Benchmark.seeds`, vmapped together) - timed and `jax.block_until_ready`'d from outside any `jax.jit` trace, unlike anything `train` could measure about itself. Excludes compile time (see `cost.compile_time_seconds`). Scalar.
`fps`	`Array`	Training throughput: `budget / wall_time`. Scalar. Comparable only across results measured on the same hardware.
`cost`	`CostAnalysis`	From `AlgorithmEntry.cost_analysis`.

`CostAnalysis`

Bases: PyTreeNode

The cost of one AlgorithmEntry.train call, as measured by AlgorithmEntry.cost_analysis.

Attributes:

Name	Type	Description
`flops`	`float`	FLOPs, from `compiled.cost_analysis()`.
`memory_bytes`	`float`	Peak memory proxy (argument + temp + output size), from `compiled.memory_analysis()`.
`compile_time_seconds`	`float`	Wall-clock time to compile. Hardware/XLA-version-sensitive.

`FromScratchBenchmark`

Bases: Benchmark

Trains entry from scratch, independently, across a flat list of environments - no ordering, no transfer assumed between them.

env_ids/seeds/budget are all fixed per preset - overridden by subclassing, not by a run argument, so every run of a given class scores the same environments with the same seeds; Navix1M/Navix100K (below) fix budget on top of this.

Attributes:

Name	Type	Description
`budget`	`int`	Passed to `entry.train`/`entry.cost_analysis` as their `budget` argument.
`env_ids`	`Tuple[str, ...]`	Environments to train on. Defaults to `DEFAULT_ENV_IDS` - a small, curated set spanning several environment families, not every registered environment. Falsy (e.g. explicitly set to `None`) resolves lazily, at `run`/`details` time, to every registered environment instead.

`NON_NUMERIC_DETAILS = ('env_ids',)` `class-attribute`

self.details(...)'s keys that aren't jnp.mean-able - row labels, not metrics.

`details(results)`

Per-environment breakdown of this run's metrics.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
`Dict[str, Any]`	Dict[str, Any]: The same last-percent reduction `summary`
`Dict[str, Any]`	aggregates further, but stopped one step earlier - every
`Dict[str, Any]`	column keeps its leading env axis. Includes `env_ids`
`Dict[str, Any]`	(which row is which - a `Tuple[str, ...]`, not an `Array`),
`Dict[str, Any]`	`benchmark/episode/length` (not in `summary`, but a useful
`Dict[str, Any]`	per-env diagnostic), and `benchmark/episode/finite_fraction`
`Dict[str, Any]`	(fraction of `self.seeds` whose
`Dict[str, Any]`	`benchmark/episode/convergence_rate` was finite - i.e. made
`Dict[str, Any]`	some real progress in the final 20% of training; a
`Dict[str, Any]`	reliability signal `benchmark/episode/returns`' bias alone
`Dict[str, Any]`	can't distinguish "consistently mediocre" from "mostly
`Dict[str, Any]`	zero, one seed got lucky").

`run(entry)`

Runs entry against self.env_ids independently, at self.budget, using self.seeds.

Parameters:

Name	Type	Description	Default
`entry`	`AlgorithmEntry`	The algorithm to score.	required

Returns:

Name	Type	Description
`BenchmarkResult`	`BenchmarkResult`	One result per env_id, stacked along a
	`BenchmarkResult`	new leading axis - row `i` is `self.env_ids[i]` (or the
	`BenchmarkResult`	`i`-th registered env, if `self.env_ids` is unset); see
	`BenchmarkResult`	`details`.

`summary(results)`

The leaderboard table row for this protocol.

Parameters:

Name	Type	Description	Default
`results`	`BenchmarkResult`	This protocol's `run` output.	required

Returns:

Type	Description
`Dict[str, Array]`	Dict[str, jax.Array]: Each numeric column of
`Dict[str, Array]`	`self.details(results)` meaned across its env axis -
`Dict[str, Array]`	`benchmark/episode/returns`' last-percent-mean (bias),
`Dict[str, Array]`	variance, convergence rate, and finite fraction (see
`Dict[str, Array]`	`self.details`), plus `benchmark/costs/*`'s bias.
`Dict[str, Array]`	`benchmark/episode/length`/`env_ids` (see
`Dict[str, Array]`	`NON_NUMERIC_DETAILS`) aren't included - still on
`Dict[str, Array]`	`self.details(results)`. Non-finite values (e.g.
`Dict[str, Array]`	`benchmark/episode/convergence_rate`'s `overall / target`
`Dict[str, Array]`	is `0/0` or `x/0` when an environment's
`Dict[str, Array]`	`benchmark/episode/returns` never leaves zero - a real
`Dict[str, Array]`	algorithm never solving that environment, not a bug) are
`Dict[str, Array]`	excluded from the mean rather than propagated - one
`Dict[str, Array]`	degenerate environment/seed shouldn't blank out every
`Dict[str, Array]`	other one's otherwise-valid signal. `self.details(results)`
`Dict[str, Array]`	keeps the raw, un-filtered per-environment values (a NaN
`Dict[str, Array]`	there is itself informative), only this aggregate step
`Dict[str, Array]`	filters them.

`Navix100K`

Bases: FromScratchBenchmark

Same as Navix1M, at 100K frames - a cheaper preset for quick checks.

`Navix1M`

Bases: FromScratchBenchmark

1M-frame budget per environment - PPOHparams/PQNHparams' own default.

`TrainingCurve`

Bases: PyTreeNode

One AlgorithmEntry.train call's measurements - purely what's computable from inside a jax.jit trace, from the real (state, action, reward, done) interaction stream. No wall-clock timing, no cost - time.time() inside a jitted function only ever fires at trace time, not per call, so neither can be measured here; see BenchmarkResult for those.

Every field is a per-update curve - shape (num_updates,) for a single training curve (checked by AlgorithmEntry. validate_train_contract), so last_percent_mean/ last_percent_variance/convergence_rate reduce all of them the same way, uniformly, no exceptions.

Attributes:

Name	Type	Description
`episodic_returns`	`Array`	Episodic return, masked-mean over completed episodes only.
`lengths`	`Array`	Episode length, masked-mean over completed episodes only.
`diagnostics`	`Dict[str, Array]`	Free-form per-update diagnostic curves (e.g. `{"loss": ..., "lr": ...}`) - whatever helps debug the algorithm. Each value shape `(num_updates,)`. Empty by default.

`convergence_rate(percent=20)`

Reduces every field to a normalized area under its curve: the ratio of the curve's overall mean to its last_percent_mean. Near 1 means the curve was close to its own asymptote for most of training (fast convergence); near 0 means most of training was spent far from it (slow).

Not bounded to [0, 1] - and shouldn't be clipped to look like it is. That range only holds when the curve improves monotonically toward its own tail. Whenever the tail is worse than the training-long average - policy collapse, instability, catastrophic forgetting, all real and fairly common RL failure modes - overall / target correctly exceeds 1, in principle arbitrarily far (confirmed in practice: a seed that learned real signal early, then collapsed to near-zero returns by the final 20% of training, produced 77 here - correctly flagging that specific seed as "learned, then collapsed" rather than "steady, healthy convergence", which a clipped value couldn't distinguish). A negative value is similarly meaningful for a negative-reward task, not an error.

Parameters:

Name	Type	Description	Default
`percent`	`float`	Percentage of the trailing axis that defines the asymptote (see `last_percent_mean`).	`20`

Returns:

Name	Type	Description
`TrainingCurve`	`TrainingCurve`	A copy with every field reduced along its
	`TrainingCurve`	trailing axis.

`last_percent_mean(percent=20)`

Reduces every field to its mean over the last percent% of its trailing axis.

Parameters:

Name	Type	Description	Default
`percent`	`float`	Percentage of the trailing axis to average over.	`20`

Returns:

Name	Type	Description
`TrainingCurve`	`TrainingCurve`	A copy with every field reduced along its
	`TrainingCurve`	trailing axis.

`last_percent_variance(percent=20)`

Reduces every field to its variance over the last percent% of its trailing axis - how much an already-converged curve still fluctuates update-to-update, not variance across seeds. The training-curve analogue of the per-update variance in Rowland, Dabney & Munos, "Adaptive Trade-Offs in Off-Policy Learning" (https://arxiv.org/abs/1910.07478, Definition 1.4).

Parameters:

Name	Type	Description	Default
`percent`	`float`	Percentage of the trailing axis to compute the variance over.	`20`

Returns:

Name	Type	Description
`TrainingCurve`	`TrainingCurve`	A copy with every field reduced along its
	`TrainingCurve`	trailing axis.

`cpu_type()`

Reads the CPU's model name.

Returns:

Name	Type	Description
`str`	`str`	`/proc/cpuinfo`'s `model name` on Linux,
	`str`	`sysctl -n machdep.cpu.brand_string` on macOS, or
	`str`	`platform.processor()`/`platform.machine()` as a fallback on
	`str`	any other platform, or if the platform-specific lookup fails.

`cuda_version()`

Reads the CUDA version jaxlib is running on.

Returns:

Type	Description
`Optional[str]`	Optional[str]: The installed `nvidia-cuda-runtime-cuXX`
`Optional[str]`	package version (the same one a `pip install jaxlib[cudaXX]`
`Optional[str]`	pins), falling back to `nvidia-smi`'s reported CUDA version if
`Optional[str]`	that package isn't found. `None` if JAX isn't running on a
`Optional[str]`	GPU.

`cudnn_version()`

Reads the cuDNN version jaxlib is running on.

Returns:

Type	Description
`Optional[str]`	Optional[str]: The installed `nvidia-cudnn-cuXX` package
`Optional[str]`	version (the same one a `pip install jaxlib[cudaXX]` pins).
`Optional[str]`	`None` if JAX isn't running on a GPU.

`gpu_type()`

Reads the specific GPU model JAX is running on.

Returns:

Type	Description
`Optional[str]`	Optional[str]: The GPU model (e.g. distinguishes an SXM from a
`Optional[str]`	PCIe variant of the same chip), or `None` if JAX isn't running
`Optional[str]`	on a GPU.

`is_commit_url(url)`

Checks whether url is a full URL ending in a commit SHA.

Parameters:

Name	Type	Description	Default
`url`	`str`	The URL to validate.	required

Returns:

Name	Type	Description
`bool`	`bool`	True if `url` has an http(s) scheme and its last path
	`bool`	segment is a 7-40 character lowercase hex SHA.

`plot_dashboard(logs, metrics=None, x_key='agent/train/frames', xlabel='Frames')`

Plots metrics (MANDATORY_METRICS by default) as a single combined figure, one panel per metric.

This is intentionally agnostic about "mandatory vs. diagnostic" - navix doesn't know, and shouldn't need to know, what a given algorithm considers diagnostic (an algorithm submitted to a leaderboard won't necessarily have any navix-specific code to declare that in). That categorisation belongs to whatever consumes this module - e.g. a leaderboard's own file mapping algorithm -> diagnostic keys - which can merge its own metrics dict with MANDATORY_METRICS and pass the result here.

Parameters:

Name	Type	Description	Default
`logs`	`Dict[str, Array]`	The `logs` pytree (see `plot_metric`).	required
`metrics`	`Dict[str, str]`	A mapping of `logs` key to plot title. Defaults to `MANDATORY_METRICS`. Keys missing from `logs` are silently skipped.	`None`
`x_key`	`str`	The key in `logs` to use as the x-axis.	`'agent/train/frames'`
`xlabel`	`str`	The x-axis label.	`'Frames'`

Returns:

Type	Description
	matplotlib.figure.Figure: The combined dashboard figure.

`plot_metric(logs, key, title=None, x_key='agent/train/frames', xlabel='Frames', ax=None)`

Plots a single metric against x_key, aggregated with a mean line and a min-max shaded band over any leading batch dimensions (e.g. seeds).

Parameters:

Name	Type	Description	Default
`logs`	`Dict[str, Array]`	The `logs` pytree, as returned by `navix.agents.agent.derive_episodic_metrics` (for `agent/episode/` keys) or directly from `Experiment.run()` (for raw keys like `agent/train/`, `agent/diagnostics/`, `experiment/costs/`).	required
`key`	`str`	The key in `logs` to plot.	required
`title`	`str`	The plot title. Defaults to `key`.	`None`
`x_key`	`str`	The key in `logs` to use as the x-axis.	`'agent/train/frames'`
`xlabel`	`str`	The x-axis label.	`'Frames'`
`ax`	`Axes`	An existing axes to draw into. If `None`, a new standalone figure and axes are created.	`None`

Returns:

Type	Description
	matplotlib.figure.Figure: The figure `ax` belongs to.

`plot_metrics(logs, metrics, x_key='agent/train/frames', xlabel='Frames')`

Plots each metric in metrics as its own standalone figure.

Parameters:

Name	Type	Description	Default
`logs`	`Dict[str, Array]`	The `logs` pytree (see `plot_metric`).	required
`metrics`	`Dict[str, str]`	A mapping of `logs` key to plot title, e.g. `MANDATORY_METRICS`, or a leaderboard-side mapping of algorithm -> diagnostic keys.	required
`x_key`	`str`	The key in `logs` to use as the x-axis.	`'agent/train/frames'`
`xlabel`	`str`	The x-axis label.	`'Frames'`

Returns:

Type	Description
`Dict[str, Figure]`	Dict[str, Figure]: One figure per metric, keyed by the same key
`Dict[str, Figure]`	as `metrics`. Keys missing from `logs` are silently skipped.

`ram_bytes()`

Reads total system RAM.

Returns:

Name	Type	Description
`int`	`int`	Total system RAM, in bytes. POSIX-only (Linux/macOS).

`search_hparams(trainable, hparams_distr, seeds, pop_size=8, num_generations=10, sigma=1.0, solver=None, n_probe=256)`

Evolution-strategies hyperparameter search (see navix.es for the shared antithetic-sampling/probe-statistics math, and Experiment.run_hparam_search's docstring for the full algorithm description - this is the same algorithm, generalized past navix's own Agent/HParams).

Each generation: sample an antithetic population of pop_size hyperparameter sets around the current mean, call trainable on every one of them (vmapped over both the population and seeds), score each by its last-20%-mean episodic_returns (TrainingCurve. last_percent_mean, averaged over seeds), then take an ES step. The best-scoring hyperparameter set actually evaluated across every generation - not the (never directly trained) mean trajectory itself - is what's returned.

Parameters:

Name	Type	Description	Default
`trainable`	`Trainable`	`(hparams, rng) -> TrainingCurve` - see `Trainable`'s own docstring for the closure convention.	required
`hparams_distr`	`Dict[str, Distribution]`	One distribution per searched field - seeds that field's starting value/scale/valid range (via `navix.es. probe_hparam_field_stats`), not resampled every generation. Every candidate and every ES update is clipped to that field's empirical `[min, max]` - without it, nothing stops the search drifting a field outside the range this distribution was ever meant to describe (e.g. a `gae_lambda` distribution meant to express "search within `[0.8, 0.99]`" wouldn't stop the search drifting past `1.0`, an invalid value).	required
`seeds`	`Tuple[int, ...]`	PRNG seeds `trainable` is vmapped over per candidate - must have more than one, so fitness isn't just RNG luck for a single rollout.	required
`pop_size`	`int`	Population size per generation. Must be even - antithetic sampling pairs (+/-).	`8`
`num_generations`	`int`	Number of ES update steps.	`10`
`sigma`	`float`	Noise scale, in units of each field's own empirical probe std.	`1.0`
`solver`	`GradientTransformation`	The ES mean update rule. Defaults to `optax.sgd(0.1)`.	`None`
`n_probe`	`int`	Samples drawn from each field's distribution to estimate that field's starting value and scale.	`256`

Returns:

Type	Description
`Dict[str, float]`	Tuple[Dict[str, float], Array]: The best-scoring hyperparameter
`Array`	set actually evaluated across every generation (plain Python
`Tuple[Dict[str, float], Array]`	floats, ready to splice into whatever config `trainable`'s
`Tuple[Dict[str, float], Array]`	closure builds from), and its fitness (last-20%-mean
`Tuple[Dict[str, float], Array]`	`episodic_returns`, averaged over `seeds`).

Raises:

Type	Description
`ValueError`	If `seeds` has one or fewer entries, or `pop_size` is odd.

Index

MANDATORY_METRICS = {'agent/episode/returns': 'Episodic Return', 'agent/episode/success_rate': 'Success Rate', 'agent/episode/length': 'Episode Length', 'experiment/costs/fps': 'Training Throughput (steps/s)', 'experiment/costs/wall_time': 'Wall-clock Training Time (s)'} module-attribute

AlgorithmEntry dataclass

cost_analysis(env_id, budget)

train(env_id, budget, rng)

validate_train_contract()

Benchmark

details(results)

plot_details(results)

plot_diagnostics(results)

plot_summary(results)

run(entry)

run_env(entry, env_id, budget)

submit_entry(entry, results, max_points=50, subdir='')

summary(results)

BenchmarkResult

CostAnalysis

FromScratchBenchmark

NON_NUMERIC_DETAILS = ('env_ids',) class-attribute

details(results)

run(entry)

summary(results)

Navix100K

Navix1M

TrainingCurve

convergence_rate(percent=20)

last_percent_mean(percent=20)

last_percent_variance(percent=20)

cpu_type()

cuda_version()

cudnn_version()

gpu_type()

is_commit_url(url)

plot_dashboard(logs, metrics=None, x_key='agent/train/frames', xlabel='Frames')

plot_metric(logs, key, title=None, x_key='agent/train/frames', xlabel='Frames', ax=None)

plot_metrics(logs, metrics, x_key='agent/train/frames', xlabel='Frames')

ram_bytes()

search_hparams(trainable, hparams_distr, seeds, pop_size=8, num_generations=10, sigma=1.0, solver=None, n_probe=256)

`MANDATORY_METRICS = {'agent/episode/returns': 'Episodic Return', 'agent/episode/success_rate': 'Success Rate', 'agent/episode/length': 'Episode Length', 'experiment/costs/fps': 'Training Throughput (steps/s)', 'experiment/costs/wall_time': 'Wall-clock Training Time (s)'}` `module-attribute`

`AlgorithmEntry` `dataclass`

`cost_analysis(env_id, budget)`

`train(env_id, budget, rng)`

`validate_train_contract()`

`Benchmark`

`details(results)`

`plot_details(results)`

`plot_diagnostics(results)`

`plot_summary(results)`

`run(entry)`

`run_env(entry, env_id, budget)`

`submit_entry(entry, results, max_points=50, subdir='')`

`summary(results)`

`BenchmarkResult`

`CostAnalysis`

`FromScratchBenchmark`

`NON_NUMERIC_DETAILS = ('env_ids',)` `class-attribute`

`details(results)`

`run(entry)`

`summary(results)`

`Navix100K`

`Navix1M`

`TrainingCurve`

`convergence_rate(percent=20)`

`last_percent_mean(percent=20)`

`last_percent_variance(percent=20)`

`cpu_type()`

`cuda_version()`

`cudnn_version()`

`gpu_type()`

`is_commit_url(url)`

`plot_dashboard(logs, metrics=None, x_key='agent/train/frames', xlabel='Frames')`

`plot_metric(logs, key, title=None, x_key='agent/train/frames', xlabel='Frames', ax=None)`

`plot_metrics(logs, metrics, x_key='agent/train/frames', xlabel='Frames')`

`ram_bytes()`

`search_hparams(trainable, hparams_distr, seeds, pop_size=8, num_generations=10, sigma=1.0, solver=None, n_probe=256)`