Scratch

FromScratchBenchmark and its Navix1M/Navix100K presets - trains an entry from scratch, independently, across a flat list of environments. See navix.benchmarks (this package's __init__.py) for the full design.

DEFAULT_ENV_IDS = ('Navix-Empty-8x8-v0', 'Navix-Dynamic-Obstacles-5x5-v0', 'Navix-FourRooms-v0', 'Navix-KeyCorridorS4R3-v0', 'Navix-DoorKey-8x8-v0', 'Navix-SimpleCrossingS9N2-v0') module-attribute

FromScratchBenchmark.env_ids's default - chosen for the RL capability each env isolates, not just family coverage: Empty-8x8 (convergence rate - solvable with no confounding structure, so speed differences are legible), Dynamic-Obstacles-5x5 (variance - the stochasticity is in the environment itself, moving obstacles differ every episode), FourRooms (exploration - Sutton & Precup's original domain, no reward signal until the goal room is found), KeyCorridorS4R3 (credit assignment, deep chain - key -> door -> goal with no partial credit), DoorKey-8x8 (credit assignment, shallow chain - the same idea at a smaller difficulty gradient), SimpleCrossingS9N2 (static- obstacle pathing). Not every registered environment - falsy (e.g. explicitly set to None) resolves lazily, at run/details time, to every registered environment instead.

FromScratchBenchmark

Bases: Benchmark

Trains entry from scratch, independently, across a flat list of environments - no ordering, no transfer assumed between them.

env_ids/seeds/budget are all fixed per preset - overridden by subclassing, not by a run argument, so every run of a given class scores the same environments with the same seeds; Navix1M/Navix100K (below) fix budget on top of this.

Attributes:

Name	Type	Description
budget	int	Passed to entry.train/entry.cost_analysis as their budget argument.
env_ids	Tuple[str, ...]	Environments to train on. Defaults to DEFAULT_ENV_IDS - a small, curated set spanning several environment families, not every registered environment. Falsy (e.g. explicitly set to None) resolves lazily, at run/details time, to every registered environment instead.

NON_NUMERIC_DETAILS = ('env_ids',) class-attribute

self.details(...)'s keys that aren't jnp.mean-able - row labels, not metrics.

details(results)

Per-environment breakdown of this run's metrics.

Parameters:

Name	Type	Description	Default
results	BenchmarkResult	This protocol's run output.	required

Returns:

Type	Description
Dict[str, Any]	Dict[str, Any]: The same last-percent reduction summary
Dict[str, Any]	aggregates further, but stopped one step earlier - every
Dict[str, Any]	column keeps its leading env axis. Includes env_ids
Dict[str, Any]	(which row is which - a Tuple[str, ...], not an Array),
Dict[str, Any]	benchmark/episode/length (not in summary, but a useful
Dict[str, Any]	per-env diagnostic), and benchmark/episode/finite_fraction
Dict[str, Any]	(fraction of self.seeds whose
Dict[str, Any]	benchmark/episode/convergence_rate was finite - i.e. made
Dict[str, Any]	some real progress in the final 20% of training; a
Dict[str, Any]	reliability signal benchmark/episode/returns' bias alone
Dict[str, Any]	can't distinguish "consistently mediocre" from "mostly
Dict[str, Any]	zero, one seed got lucky").

run(entry)

Runs entry against self.env_ids independently, at self.budget, using self.seeds.

Parameters:

Name	Type	Description	Default
entry	AlgorithmEntry	The algorithm to score.	required

Returns:

Name	Type	Description
BenchmarkResult	BenchmarkResult	One result per env_id, stacked along a
	BenchmarkResult	new leading axis - row i is self.env_ids[i] (or the
	BenchmarkResult	i-th registered env, if self.env_ids is unset); see
	BenchmarkResult	details.

summary(results)

The leaderboard table row for this protocol.

Parameters:

Name	Type	Description	Default
results	BenchmarkResult	This protocol's run output.	required

Returns:

Type	Description
Dict[str, Array]	Dict[str, jax.Array]: Each numeric column of
Dict[str, Array]	self.details(results) meaned across its env axis -
Dict[str, Array]	benchmark/episode/returns' last-percent-mean (bias),
Dict[str, Array]	variance, convergence rate, and finite fraction (see
Dict[str, Array]	self.details), plus benchmark/costs/*'s bias.
Dict[str, Array]	benchmark/episode/length/env_ids (see
Dict[str, Array]	NON_NUMERIC_DETAILS) aren't included - still on
Dict[str, Array]	self.details(results). Non-finite values (e.g.
Dict[str, Array]	benchmark/episode/convergence_rate's overall / target
Dict[str, Array]	is 0/0 or x/0 when an environment's
Dict[str, Array]	benchmark/episode/returns never leaves zero - a real
Dict[str, Array]	algorithm never solving that environment, not a bug) are
Dict[str, Array]	excluded from the mean rather than propagated - one
Dict[str, Array]	degenerate environment/seed shouldn't blank out every
Dict[str, Array]	other one's otherwise-valid signal. self.details(results)
Dict[str, Array]	keeps the raw, un-filtered per-environment values (a NaN
Dict[str, Array]	there is itself informative), only this aggregate step
Dict[str, Array]	filters them.

Navix100K

Bases: FromScratchBenchmark

Same as Navix1M, at 100K frames - a cheaper preset for quick checks.

Navix1M

Bases: FromScratchBenchmark

1M-frame budget per environment - PPOHparams/PQNHparams' own default.