vesselnav-bench v2 — Leaderboard

→ VesselNav-Bench v1 (archived)

Seeded navigation suite for comparing autonomous-vessel navigation policies under identical physics, scenarios, and scoring: canonical COLREGs encounters, coastal planning, and randomized worlds, in calm, disturbed (current + wind gust + sea state), degraded-perception, and shoaling (depth / under-keel clearance) conditions.

generated 2026-06-23T08:22:10 11 scenarios × 20 seeded episodes × 6 conditions config hash per condition (physics fingerprint, not a seed): calm: 16e87b9cbe01a104disturbed: 8eb3fc5d0e399f8bdegraded: 73438db0aa38ecacshoaling: 9f29bc10527d6257hull_randomized: 284c51f7d6bc18f4restricted_visibility: a9551fd639e0949d

Benchmark score = 100 × (0.6 × success rate + 0.25 × COLREGs compliance + 0.15 × path efficiency). Rates carry Wilson 95% CIs, means carry seeded bootstrap 95% CIs; all metrics are computed from ground-truth episode logs. Scores are comparable only between runs with identical config hashes.

The agents

classicalrule-based (layered)

by V. Ravendranathan (VesselNav-Bench baseline)

A* plans the route, ILOS guidance tracks it, and a predictive COLREGs-inspired avoider overrides the helm when rollouts with the real vessel dynamics predict a conflict on the route.

mpcoptimization (model predictive control)

by V. Ravendranathan (VesselNav-Bench baseline)

Solves a local optimal-control problem each second: candidate maneuver plans are rolled out with the real dynamics and scored by one cost combining progress, separation, effort, and a COLREGs prior.

rl_ppolearning (deep reinforcement learning)

by V. Ravendranathan (VesselNav-Bench baseline)

PPO policy over a labeled 35-feature observation (goal vector, own dynamics, land lidar, nearest traffic); each decision logs the full action probability distribution and value estimate.

rl_ppo_shieldedhybrid (learning + runtime safety filter)

by V. Ravendranathan (VesselNav-Bench baseline)

The PPO policy proposes; a classical predictive filter vets every proposal with real-dynamics rollouts and substitutes the nearest safe course when needed. Every intervention is logged.

What is being tested

All agents steer the same own-ship model: a 3-DOF surge–sway–yaw maneuvering model (first-order Nomoto yaw response, sideslip toward −k·u·r, turn-induced speed loss, first-order surge), an IMO-standard rudder actuator (35° limit, 70°-in-11-s slew rate), and a course-over-ground PD autopilot. Disturbed episodes add a scenario-seeded water current (drifting the traffic too) and wind gusts. Agents command intent (desired course and speed); the physics decides what the hull does.

Scenario	COLREGs rule	Situation
`open_water`	No obstacles	Pure route following, no rule applies
`head_on`	Rule 14 — head-on	Reciprocal courses; both vessels must alter to starboard and pass port-to-port
`crossing_starboard`	Rule 15/16 — give-way	Traffic crosses from starboard; own ship must take early, substantial action to keep clear
`crossing_port`	Rule 17 — stand-on	Traffic crosses from port; own ship should hold course while it remains safe
`overtaking`	Rule 13 — overtaking	Slow vessel ahead on the same course; overtaking vessel keeps clear, either side
`coastal`	Mixed	Landmasses forming a channel plus two traffic vessels; planning and avoidance together
`multi_vessel`	Rules 14+15 combined	Two simultaneous conflicts timed to overlap, plus a third vessel passing by
`narrow_channel`	Rule 14 in a fairway (Rule 9 flavor)	Oncoming vessel inside a narrow channel; limited sea room on both sides
`random`	Mixed	Seeded random islands and wandering traffic

Each scenario runs 20 seeded episodes per condition; calm has no disturbances, disturbed adds a scenario-seeded random current (up to 0.3 cells/s) and wind gusts. Every agent faces the identical episodes.

Podium — Overall

Ranked by the unweighted mean benchmark score across all 6 conditions (each carries the same episode count), so all-round competence wins.

classical

93.5

overall score

94% mean success · 6 conditions

mpc

93.9

overall score

97% mean success · 6 conditions

rl_ppo_shielded

87.7

overall score

93% mean success · 6 conditions

Condition: calm

Baseline physics — no current, wind, sea state or perception noise. Isolates pure navigation and COLREGs skill and anchors every robustness comparison.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	classical V. Ravendranathan (VesselNav-Bench baseline)	96.8	99.6% [97.5%, 99.9%]	0.0% [0.0%, 1.7%]	0.4% [0.1%, 2.5%]	0.92 (n=212)	210.2 [204.7, 215.7]	10.84 [9.97, 11.85]
2	mpc V. Ravendranathan (VesselNav-Bench baseline)	95.7	100.0% [98.3%, 100.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.87 (n=230)	209.9 [204.2, 216.0]	11.17 [10.30, 12.19]
3	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	85.4	90.9% [86.4%, 94.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.70 (n=223)	256.6 [246.4, 266.9]	9.92 [8.75, 11.21]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	60.5	61.4% [54.8%, 67.5%]	27.3% [21.8%, 33.5%]	11.4% [7.8%, 16.2%]	0.35 (n=243)	203.7 [198.7, 209.0]	7.15 [5.85, 8.54]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
classical	0.76 (n=68)	1.00 (n=84)	1.00 (n=60)
mpc	0.72 (n=106)	0.98 (n=64)	1.00 (n=60)
rl_ppo_shielded	0.65 (n=81)	0.58 (n=82)	0.93 (n=60)
rl_ppo	0.48 (n=81)	0.40 (n=102)	0.07 (n=60)

Condition: disturbed

Environmental robustness — scenario-seeded water current (up to 0.3 cells/s) and wind gusts, plus a moderate sea state (Hs ≈ 1.25 m, sea state 3–4) that adds speed loss from added resistance and a first-order yaw oscillation at the wave encounter frequency. Wave direction is drawn per episode.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	88.1	91.8% [87.4%, 94.8%]	0.9% [0.2%, 3.2%]	2.7% [1.3%, 5.8%]	0.76 (n=202)	269.7 [251.6, 290.1]	12.01 [10.74, 13.45]
2	mpc V. Ravendranathan (VesselNav-Bench baseline)	86.3	86.4% [81.2%, 90.3%]	1.8% [0.7%, 4.6%]	0.0% [0.0%, 1.7%]	0.81 (n=225)	208.8 [200.4, 218.6]	10.73 [9.67, 11.91]
3	classical V. Ravendranathan (VesselNav-Bench baseline)	84.0	80.5% [74.7%, 85.2%]	1.8% [0.7%, 4.6%]	0.9% [0.2%, 3.2%]	0.88 (n=227)	247.5 [234.3, 262.0]	12.01 [10.96, 13.18]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	71.3	74.1% [67.9%, 79.4%]	13.6% [9.7%, 18.8%]	10.4% [7.1%, 15.2%]	0.47 (n=211)	216.3 [204.6, 231.3]	10.94 [9.37, 12.54]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
rl_ppo_shielded	0.68 (n=75)	0.73 (n=66)	0.88 (n=61)
mpc	0.73 (n=80)	0.85 (n=85)	0.87 (n=60)
classical	0.74 (n=79)	0.96 (n=88)	0.95 (n=60)
rl_ppo	0.54 (n=76)	0.56 (n=74)	0.28 (n=61)

Condition: degraded

Perception robustness at AIS/GPS-grade sensing — noisy target position (~12 m), course (6°) and speed, fixes only every 4 s (staleness on moving targets) and occasional target dropout. Physics stays calm to isolate sensing.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	mpc V. Ravendranathan (VesselNav-Bench baseline)	95.6	100.0% [98.3%, 100.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.85 (n=237)	203.6 [198.8, 208.6]	9.83 [8.98, 10.88]
2	classical V. Ravendranathan (VesselNav-Bench baseline)	92.7	93.6% [89.6%, 96.2%]	2.3% [1.0%, 5.2%]	0.9% [0.2%, 3.2%]	0.91 (n=230)	263.7 [255.5, 272.6]	11.35 [10.48, 12.37]
3	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	89.5	95.5% [91.8%, 97.5%]	2.3% [1.0%, 5.2%]	0.4% [0.1%, 2.5%]	0.75 (n=224)	278.8 [261.3, 299.0]	9.71 [8.52, 11.04]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	59.6	60.0% [53.4%, 66.2%]	28.6% [23.1%, 34.9%]	11.4% [7.8%, 16.2%]	0.34 (n=242)	201.5 [197.0, 206.0]	6.59 [5.30, 7.98]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
mpc	0.76 (n=92)	0.90 (n=85)	0.92 (n=60)
classical	0.73 (n=72)	0.99 (n=98)	1.00 (n=60)
rl_ppo_shielded	0.64 (n=85)	0.72 (n=79)	0.96 (n=60)
rl_ppo	0.44 (n=86)	0.42 (n=96)	0.07 (n=60)

Condition: shoaling

Depth instead of binary land — a charted depth field that shoals toward shore, so grounding becomes depth < draft + under-keel clearance. Coastal scenarios gain a realistic shoal apron while open water stays deep.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	classical V. Ravendranathan (VesselNav-Bench baseline)	96.8	99.6% [97.5%, 99.9%]	0.0% [0.0%, 1.7%]	0.4% [0.1%, 2.5%]	0.92 (n=212)	210.2 [204.7, 215.7]	10.84 [9.97, 11.85]
2	mpc V. Ravendranathan (VesselNav-Bench baseline)	95.7	100.0% [98.3%, 100.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.87 (n=230)	209.9 [204.2, 216.0]	11.17 [10.30, 12.19]
3	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	85.4	90.9% [86.4%, 94.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.70 (n=223)	256.6 [246.4, 266.9]	9.92 [8.75, 11.21]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	60.5	61.4% [54.8%, 67.5%]	27.3% [21.8%, 33.5%]	11.4% [7.8%, 16.2%]	0.35 (n=243)	203.7 [198.7, 209.0]	7.15 [5.85, 8.54]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
classical	0.76 (n=68)	1.00 (n=84)	1.00 (n=60)
mpc	0.72 (n=106)	0.98 (n=64)	1.00 (n=60)
rl_ppo_shielded	0.65 (n=81)	0.58 (n=82)	0.93 (n=60)
rl_ppo	0.48 (n=81)	0.40 (n=102)	0.07 (n=60)

Condition: hull_randomized

Own-ship model mismatch — the hull's maneuvering parameters (Nomoto K/T and the 3-DOF constants) are jittered ±25% per episode while the controller still plans with the nominal model. Tests robustness to an imperfectly known ship.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	classical V. Ravendranathan (VesselNav-Bench baseline)	96.5	99.1% [96.8%, 99.8%]	0.0% [0.0%, 1.7%]	0.9% [0.2%, 3.2%]	0.92 (n=212)	220.5 [214.4, 227.0]	10.83 [9.97, 11.83]
2	mpc V. Ravendranathan (VesselNav-Bench baseline)	95.7	100.0% [98.3%, 100.0%]	0.0% [0.0%, 1.7%]	0.0% [0.0%, 1.7%]	0.87 (n=230)	210.6 [204.7, 216.8]	11.03 [10.18, 12.04]
3	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	89.4	97.7% [94.8%, 99.0%]	0.0% [0.0%, 1.7%]	0.4% [0.1%, 2.5%]	0.70 (n=223)	282.2 [263.9, 301.6]	9.86 [8.67, 11.16]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	60.5	61.4% [54.8%, 67.5%]	27.3% [21.8%, 33.5%]	11.4% [7.8%, 16.2%]	0.35 (n=243)	203.8 [198.7, 209.0]	7.17 [5.88, 8.56]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
classical	0.76 (n=68)	1.00 (n=84)	1.00 (n=60)
mpc	0.73 (n=106)	0.97 (n=64)	1.00 (n=60)
rl_ppo_shielded	0.66 (n=81)	0.58 (n=82)	0.90 (n=60)
rl_ppo	0.49 (n=81)	0.40 (n=102)	0.07 (n=60)

Condition: restricted_visibility

Fog / Rule 19 — radar-only perception (coarser noise, slower 6 s fixes, heavier dropout than degraded) and a scorer that drops the give-way / stand-on roles of Rules 14–17 for Rule 19: act in ample time, no turn to port for a vessel forward of the beam.

#	Agent	Score	Success	Collision	Grounding	COLREGs	Time to goal (s)	Min separation
1	mpc V. Ravendranathan (VesselNav-Bench baseline)	94.1	97.7% [94.8%, 99.0%]	2.3% [1.0%, 5.2%]	0.0% [0.0%, 1.7%]	0.84 (n=241)	202.7 [197.7, 207.9]	8.96 [8.03, 10.02]
2	classical V. Ravendranathan (VesselNav-Bench baseline)	94.0	94.5% [90.7%, 96.9%]	2.7% [1.3%, 5.8%]	0.4% [0.1%, 2.5%]	0.94 (n=232)	285.5 [275.5, 296.1]	11.58 [10.62, 12.63]
3	rl_ppo_shielded V. Ravendranathan (VesselNav-Bench baseline)	88.5	94.1% [90.1%, 96.5%]	5.0% [2.8%, 8.7%]	0.0% [0.0%, 1.7%]	0.74 (n=225)	275.3 [256.1, 297.9]	9.29 [8.06, 10.67]
4	rl_ppo V. Ravendranathan (VesselNav-Bench baseline)	57.4	55.5% [48.9%, 61.9%]	33.2% [27.3%, 39.6%]	11.4% [7.8%, 16.2%]	0.37 (n=228)	197.0 [192.8, 201.4]	7.48 [6.02, 8.99]

Episode outcomes

Safety vs speed trade-off

Success rate by scenario

COLREGs compliance by encounter type

Agent	crossing-port	crossing-starboard	head-on
mpc	0.79 (n=93)	0.86 (n=88)	0.88 (n=60)
classical	0.89 (n=77)	0.96 (n=95)	0.98 (n=60)
rl_ppo_shielded	0.72 (n=83)	0.75 (n=82)	0.74 (n=60)
rl_ppo	0.38 (n=84)	0.54 (n=84)	0.10 (n=60)

Robustness: calm vs disturbed, degraded, shoaling, hull_randomized, restricted_visibility

How much does each model lose (or gain) when conditions get harder? Solid = calm (baseline), hollow = disturbed, degraded, shoaling, hull_randomized, restricted_visibility; the printed delta is the success-rate change vs baseline.

Success rate by condition

Submit your model

Implement the Agent contract (reset(obs) / decide(obs) → Decision) and run:

python main.py benchmark --suite benchmarks/v2.yaml \
    --agent your_package.your_module:YourAgent

See the repository and docs/SUBMITTING.md. Every leaderboard number is backed by a replayable per-step episode log.