A transparent, reproducible benchmark where rule-based, optimization and learning agents navigate the same seas under the same collision-avoidance rules — every number backed by a replayable episode log.
Seeded navigation suite for comparing autonomous-vessel navigation policies under identical physics, scenarios, and scoring: canonical COLREGs encounters, coastal planning, and randomized worlds, in calm and disturbed (current + wind gust) conditions.
generated 2026-06-14T12:51:099 scenarios ×
20 seeded episodes × 2
conditionsconfig hash per condition (physics fingerprint, not a seed):calm: 1baf1e352220ed61disturbed: 9571b64e338c5753
Benchmark score = 100 × (0.6 × success rate +
0.25 × COLREGs compliance + 0.15 × path efficiency). Rates carry Wilson
95% CIs, means carry seeded bootstrap 95% CIs; all metrics are computed
from ground-truth episode logs. Scores are comparable only between runs
with identical config hashes.
The agents
classicalrule-based (layered)
by V. Ravendranathan (VesselNav-Bench baseline)
A* plans the route, ILOS guidance tracks it, and a predictive COLREGs-inspired avoider overrides the helm when rollouts with the real vessel dynamics predict a conflict on the route.
classical-legacyrule-based (layered, original avoider)
by V. Ravendranathan (VesselNav-Bench baseline)
Ablation of the classical pipeline: same A* route and ILOS guidance, but collision avoidance uses the project's original commitment-based avoider, which lacks route-aware rollouts and reduced-speed escape maneuvers. Kept to quantify how much the avoidance layer contributes.
mpcoptimization (model predictive control)
by V. Ravendranathan (VesselNav-Bench baseline)
Solves a local optimal-control problem each second: candidate maneuver plans are rolled out with the real dynamics and scored by one cost combining progress, separation, effort, and a COLREGs prior.
rl_ppolearning (deep reinforcement learning)
by V. Ravendranathan (VesselNav-Bench baseline)
PPO policy over a labeled 35-feature observation (goal vector, own dynamics, land lidar, nearest traffic); each decision logs the full action probability distribution and value estimate.
The PPO policy proposes; a classical predictive filter vets every proposal with real-dynamics rollouts and substitutes the nearest safe course when needed. Every intervention is logged.
velocity-obstaclesreactive (velocity space)
by Fiorini & Shiller (1998); reference implementation
Picks the velocity closest to the preferred course that lies outside every obstacle's collision cone (starboard-preferring, A* route reference).
potential-fieldsreactive (force field)
by Khatib (1986); reference implementation
Steers along the resultant of an attractive force toward the route and repulsive forces from traffic and land. Historically foundational; known to suffer local minima and oscillation.
What is being tested
All agents steer the same own-ship model: a 3-DOF surge–sway–yaw maneuvering model (first-order Nomoto yaw response, sideslip toward −k·u·r, turn-induced speed loss, first-order surge), an IMO-standard rudder actuator (35° limit, 70°-in-11-s slew rate), and a course-over-ground PD autopilot. Disturbed episodes add a scenario-seeded water current (drifting the traffic too) and wind gusts. Agents command intent (desired course and speed); the physics decides what the hull does.
Scenario
COLREGs rule
Situation
open_water
No obstacles
Pure route following, no rule applies
head_on
Rule 14 — head-on
Reciprocal courses; both vessels must alter to starboard and pass port-to-port
crossing_starboard
Rule 15/16 — give-way
Traffic crosses from starboard; own ship must take early, substantial action to keep clear
crossing_port
Rule 17 — stand-on
Traffic crosses from port; own ship should hold course while it remains safe
overtaking
Rule 13 — overtaking
Slow vessel ahead on the same course; overtaking vessel keeps clear, either side
coastal
Mixed
Landmasses forming a channel plus two traffic vessels; planning and avoidance together
multi_vessel
Rules 14+15 combined
Two simultaneous conflicts timed to overlap, plus a third vessel passing by
narrow_channel
Rule 14 in a fairway (Rule 9 flavor)
Oncoming vessel inside a narrow channel; limited sea room on both sides
random
Mixed
Seeded random islands and wandering traffic
Each scenario runs
20 seeded episodes per condition;
calm has no disturbances, disturbed adds a scenario-seeded
random current (up to 0.3 cells/s) and wind gusts. Every agent faces the
identical episodes.
Podium — Overall
Ranked by the unweighted mean benchmark score across all 2 conditions (each carries the same episode count), so all-round competence wins.
2
classical
96.3
overall score
99% mean success · 2 conditions
1
mpc
96.7
overall score
99% mean success · 2 conditions
3
velocity-obstacles
93.8
overall score
99% mean success · 2 conditions
Condition: calm
Baseline physics — no current, wind, sea state or perception noise. Isolates pure navigation and COLREGs skill and anchors every robustness comparison.
#
Agent
Score
Success
Collision
Grounding
COLREGs
Time to goal (s)
Min separation
1
classical
V. Ravendranathan (VesselNav-Bench baseline)
Episode outcomesSafety vs speed trade-offSuccess rate by scenario
COLREGs compliance by encounter type
Agent
crossing-port
crossing-starboard
head-on
classical
0.76 (n=25)
0.99 (n=63)
1.00 (n=60)
mpc
0.76 (n=50)
0.99 (n=83)
1.00 (n=60)
velocity-obstacles
0.76 (n=45)
0.84 (n=83)
0.83 (n=60)
classical-legacy
0.76 (n=45)
0.75 (n=62)
1.00 (n=60)
rl_ppo_shielded
0.76 (n=66)
0.95 (n=66)
0.83 (n=60)
rl_ppo
0.76 (n=64)
0.71 (n=66)
0.41 (n=60)
potential-fields
0.36 (n=65)
0.30 (n=63)
0.31 (n=60)
Condition: disturbed
Environmental robustness — scenario-seeded water current (up to 0.3 cells/s) and wind gusts, plus a moderate sea state (Hs ≈ 1.25 m, sea state 3–4) that adds speed loss from added resistance and a first-order yaw oscillation at the wave encounter frequency. Wave direction is drawn per episode.
#
Agent
Score
Success
Collision
Grounding
COLREGs
Time to goal (s)
Min separation
1
mpc
V. Ravendranathan (VesselNav-Bench baseline)
96.3
98.3% [95.2%, 99.4%]
0.0% [0.0%, 2.1%]
0.0% [0.0%, 2.1%]
0.92 (n=191)
45.4 [43.3, 47.7]
13.22 [11.96, 14.60]
2
classical
V. Ravendranathan (VesselNav-Bench baseline)
Episode outcomesSafety vs speed trade-offSuccess rate by scenario
COLREGs compliance by encounter type
Agent
crossing-port
crossing-starboard
head-on
mpc
0.75 (n=50)
0.98 (n=81)
0.99 (n=60)
classical
0.76 (n=25)
0.99 (n=61)
0.87 (n=60)
velocity-obstacles
0.74 (n=43)
0.87 (n=83)
0.85 (n=60)
rl_ppo_shielded
0.76 (n=66)
0.91 (n=66)
0.76 (n=60)
classical-legacy
0.66 (n=45)
0.76 (n=64)
1.00 (n=60)
rl_ppo
0.76 (n=65)
0.71 (n=64)
0.42 (n=60)
potential-fields
0.44 (n=55)
0.27 (n=74)
0.24 (n=60)
Robustness: calm vs disturbed
How much does each model lose (or gain) when conditions get
harder? Solid = calm (baseline), hollow = disturbed; the printed
delta is the success-rate change vs baseline.
Success rate by condition
Submit your model
Implement the Agent contract
(reset(obs) / decide(obs) → Decision) and run: