Tasks, scores and standings

Use the result file to see what your robot completed and what to improve. A single run earns up to 100 points; a complete task ranking uses five runs.

What is a task, and why five evaluations?

A task is the challenge, such as station keeping or obstacle avoidance. One upload runs the same task five times under different conditions. It does not select five different tasks.

The map stays the same. Target positions and route layout vary slightly. Wind and current range from gentle conditions to rough sea with changing current. All entrants receive the same five conditions. Each run is scored from its own measured path, so scores may match or differ. For example, completing no targets in all five runs gives five zero scores.

View the recorded path

Choose an Evaluation run on your result page. Use Play record and the time slider to inspect the measured vehicle positions. Blue shows the path, green shows targets and orange shows obstacles. The page also shows targets reached, distance, elapsed time, position/heading/depth errors and estimated obstacle contacts.

The leaderboard shows best/lowest scores, mean time and a View paths and details link. Completed submission paths and scores are public; program code, output logs and account email stay private. Email contains per-run scores, time, targets reached and path thumbnails. Open its result link for full playback.

Example · station keeping stays pinned as a reference. Open its five measured paths and download the tested code. Its average is 12.81 / 100 and its competition score is 10.24 / 100. Compare your improvements against this example.

Choose a task

The figures show target order from above. They use evaluation 1 as an example. Numbered circles are targets; filled circles are buoys or posts. Choose a safe route between targets. Other evaluations move the targets slightly.

Hold a position

Hold a position · surface_station · WAM-V · 120 simulation seconds

1. Move to target 1. Turn to the requested heading.
2. Keep both position and heading inside the allowed range for 45 consecutive seconds.
3. Leaving the range resets the hold timer. Approach slowly so you can stop at the target.

Pass gates and return

Pass gates and return · surface_gates · WAM-V · 240 simulation seconds

1. Visit the centres of gates 1, 2 and 3 in order. Red and green buoys mark each gate.
2. Return to the start, target 4.
3. Face the return heading and hold for 5 seconds. Avoid touching the buoys on both journeys.

Avoid buoys and berth

Avoid buoys and berth · surface_slalom · WAM-V · 300 simulation seconds

1. Follow targets 1–4, alternating around the four obstacle buoys.
2. Plan a safe path back to target 5 at the start.
3. Face the berth heading and hold for 10 seconds. Reaching a target does not cancel collision penalties.

Dive through a gate and hold

Dive through a gate and hold · underwater_gate · BlueROV2 · 180 simulation seconds

1. Dive to 2 m and reach target 1.
2. Pass between the gate posts at target 2 while keeping that depth.
3. Reach target 3, face its requested heading and hold position and depth for 15 seconds.

Follow an inspection path

Follow an inspection path · underwater_path · BlueROV2 · 240 simulation seconds

1. Dive to 3 m. Visit targets 1, 2 and 3 above the orange seabed markers.
2. Follow the ordered route at that depth; the markers are on the seabed below you.
3. Return to target 4 at 3 m, face its heading and hold for 5 seconds.

Avoid posts, return and surface

Avoid posts, return and surface · underwater_return · BlueROV2 · 300 simulation seconds

1. Dive to 2 m and visit the entry gate at target 1.
2. Follow targets 2, 3 and 4 around the submerged posts.
3. Return through the same gate at target 5. Surface at target 6, the start, and hold for 5 seconds.

A holding target needs the correct position, heading and, underwater, depth. Leaving its range resets the consecutive hold timer.

Read your result in VS Code

Open your emailed result link or Your submissions → View result, then download the JSON record. Each entry under reports contains its course report.

• score: points from 0 to 100.
• completion: fraction of the ordered targets completed, from 0 to 1.
• position_rms_m: position or route error in metres; smaller is better.
• heading_rms_rad: heading error in radians; smaller is better.
• depth_rms_m: depth error for underwater tasks.
• gate_centre_rms_m: distance from the gate centre at crossings; smaller is better.
• best_consecutive_hold_seconds: longest successful station hold.
• collision_proxies: obstacles struck according to the scoring envelope.
• invalid: an error reason for an invalid run; valid runs have no reason.

Expand Program output for Python messages and errors. Keep downloaded results when comparing code changes.

Improve one thing at a time

First complete the route. Then reduce position, heading and depth errors. On gate tasks, pass close to the midpoint between the two posts. Finally improve speed while keeping a safe distance from obstacles.

Finishing a task earns its 20 progress points. Most of the remaining points require accurate control. Position, heading and depth errors reduce accuracy, speed and safety points together. A loose, completed route can therefore receive a low score.

On a gate task, the scorer finds where your measured path crosses the line between the two posts. Passing near the centre earns more accuracy points than passing close to a post. The result shows the centre error in metres for each crossing. The vehicle must fit between the posts, and underwater it must also have the right depth.

Station keeping gives partial progress for your longest consecutive successful hold, so a short hold can earn a few points before you manage all 45 seconds. Each struck obstacle costs 15 points, and a collision caps the total at 50. A valid timeout keeps the earned partial score. Invalid or cancelled runs do not enter official standings.

Finish five runs and check the Leaderboard

1. Submit your code for one task. The server evaluates the same task under five conditions.
2. The server chooses the common course sequence. You do not need to type a seed.
3. Open Leaderboard on the website and find your public alias.
4. Repeat for the other two tasks in your chosen surface or underwater track.

A task score is 80% of the five-run average + 20% of the lowest score. A track score is the average of its three task scores. An incomplete set is marked provisional. The latest completed submission replaces your task ranking. A pending or failed submission keeps the completed ranking intact.

The ArduPilot comparison row uses the shipped default vehicle configuration. Not measured means there is no measurement for that row. It does not mean a score of zero.

Advanced: exact rules, scoring and fairness

The rule set is maritime-class-3, an educational adaptation of RobotX, VRX and RoboSub tasks. These are WWW-POSIM rules. The simulator computes scores from native trajectory observations; a client cannot submit a claimed numeric score.

One run

Let c be ordered targets completed divided by total targets. Progress p=c for route tasks. For station keeping, p is the longest consecutive valid hold divided by 45 seconds, capped at 1. e_p, e_h, e_d are time-weighted RMS horizontal/route, yaw and depth errors. g is RMS gate centre error divided by the available half-width after subtracting the buoy and hull radii. Tasks without a passed gate use g=0. t is elapsed simulation time, T the timeout and I the ideal completion time.

quality    = exp(-max(0,e_p-0.05)/0.6
                 -max(0,e_h-2π/180)/0.25
                 -max(0,e_d-0.02)/0.2
                 -(g/0.25)^2)
progress   = 20 p
accuracy   = 60 p quality
speed      = 10 c clamp((T-t)/(T-I), 0, 1) quality
safety     = 10 p clamp(minimum_hull_clearance / 2, 0, 1) quality
raw        = progress + accuracy + speed + safety - 15 × struck_obstacles
score      = clamp(raw, 0, 100)

Without accuracy samples, quality=0. A good score requires both progress and precision. Crossing halfway from the gate centre toward the edge of the usable channel makes the gate quality factor exp(-4), about 0.018. It does not earn high accuracy simply by completing the crossing.

With no obstacle geometry safety is 10 p quality. Hull envelopes are conservative 3.5m radius for WAM-V and 0.5m for BlueROV2, with vertical overlap against each cylinder. This is a documented geometry proxy, not measured contact force, and can penalize a near pass before exact mesh contact. The same proxy applies to all entrants. A collision caps the score at 50. Each obstacle is counted once per run. Land/seafloor collision force integration is not an additional invented scoring input; native terrain safety/motion checks can stop or invalidate the run.

Surface gate width is 12.4 m and underwater gate width is 6.4 m before the seeded geometry scaling. Every official evaluation uses the authored 2 km × 2 km competition arena with a flat seabed at 20 m. Its geographic origin and surface start are 35.074899 N / 129.084759 E. Physics uses target 1× and a 10 ms timestep. This arena is synthetic training geometry.

Arrival envelopes: horizontal error <0.8 m surface / <0.5 m underwater; underwater depth error <0.2 m. A holding target also needs yaw error <0.12 rad (about 6.9°). Leaving the envelope resets consecutive hold time. A gate is completed by crossing its plane in the requested direction with the hull inside the usable opening. The scorer interpolates between consecutive measured poses to compute centre and depth errors at the crossing. Station precision measurement starts on first entry into its target envelope. Route accuracy projects onto the current bounded segment. Final holding error is measured inside the arrival envelope.

Examples: a collision-free complete run with all accuracy thresholds met and centred gate crossings, at or before I, and ≥2m hull clearance gets 100. With c=0.5, only half progress and bonus weights are available. A complete run with one struck obstacle is capped at 50 even if its raw score is higher. No meaningful trajectory, nonfinite values, backward/discontinuous simulation time, a pose jump, expired ownership or session change invalidates the run; invalid runs do not enter the official leaderboard. Timeouts produce the earned partial score if native data stayed valid.

RMS is integrated over native simulation time, not the number of browser polls. Physics timestep and scoring tolerances are versioned; official runs cannot change waves/physics during scoring. Full-score thresholds are strict but nonzero. For surface holding in current, the requested bow heading points against the current. In condition 5 it follows the current bearing variation, allowing the two-thruster boat to counter the flow. yaw_current_follow marks these goals; the profile includes current_epoch_sim_time, period, phase and swing to calculate the requested heading at each simulation time. If measured model/sensor limits make them unattainable, the instructor must change the rules version and remeasure the baseline instead of relabeling old scores.

Five evaluations and standings

Installed practice offers the same five evaluation conditions. The rule version, task and condition number determine the common seed. Official submissions always use that seed; custom practice layouts are available under Advanced.

Current is horizontal water velocity used by the vehicle hydrodynamics. Evaluation 5 changes speed and direction smoothly on a 45-second cycle (direction ±25°). The current starts when scoring starts. Seeded layout changes are scale ±10%, lateral shift ±3 m and holding heading ±0.25 rad. Wind bearing is 110–160° and current bearing 75–105° before its variation.

For a task, suite_score = 0.8 × mean(five scores) + 0.2 × min(five scores). A track result is the arithmetic mean of its three task scores. Fewer than five distinct seeds is provisional and displayed with a run count. For uploaded code, standings use the latest completed five-run submission; the advanced local client uses the most recent valid result for each seed. Disclose retries and compute budgets to the organizer. A completed five-seed suite starts its next retry at index zero in the current implementation; the operator can schedule a new series/rules version for a finals round.

Server stores timestamp, owner ID, public alias, task, rules version, course seed, native trajectory, metrics, controller type and baseline parameter hash. Email addresses are excluded from public responses. Installed local trials stay practice-only: an open-source client can change its own code/data. Server observation protects score provenance; it is not proof the student wrote the code or used a real navigation estimator. Ground-truth simulated odometry is currently allowed for navigation, explicitly. Obstacle planning exercises should use measured lidar/camera rather than world obstacle poses; stronger perception-only enforcement requires a separate sensor/navigation profile and should be versioned.

ArduPilot baseline

Pinned rows use a freshly started Autopilot world with the bundled vehicle-default parameter profile and simulator navigation integration. The operator invokes the identical ordered targets, measures native state and records the parameter digest. No parameter-edit session can be used as a baseline; parameter changes during a run invalidate it. Default avoidance/perception is not presumed. Parameters required for Gazebo/SITL/external navigation are part of the shipped profile, so this is not a claim about pure factory hardware settings.

Unmeasured rows say Not measured, never a made-up zero or 100. One real seed is a provisional baseline, not a completed five-seed ranking. Course or parameter/model changes require a new baseline/rules version. The baseline remains visible even when students exceed its score.

Baseline integrity

The shipped ArduPilot baseline is measured in a fresh server world. The full FCU parameter list is forced and checked before and after each run. Only SIM_SPEEDUP (checked separately at 1×), STAT_RUNTIME/STAT_FLTTIME (read-only elapsed counters), and BARO1_GND_PRESS/BARO2_GND_PRESS/BARO3_GND_PRESS (read-only, volatile automatic ground-pressure calibration) are excluded from the configuration hash. A changed control parameter invalidates the score. A run stays in finalizing until controller shutdown, the end check and the database commit finish; releasing its slot beforehand invalidates it. Invalid development attempts are retained privately and excluded from public standings. A pinned zero is a valid unsuccessful run, not a fabricated success.

Result export and local evaluation

Website submissions run uploaded code in an isolated server runner and connect it to an exclusive Gazebo world. The advanced installed Evaluation client can also run an algorithm on the participant PC. The server starts scoring before that command starts and continuously checks official physics/wave conditions. Completed or interrupted evidence is exported by its owner; after releasing the slot, /competition/run/export?run_id=... still requires that same owner. The JSON hash detects changes to the export and is not a cheating-proof certificate. There is no endpoint awarding an official score from a local file. An administrator also cannot download another owner's private export through this endpoint.

Back to competition · Set up Evaluation