CostNav: A Navigation Benchmark for Real-World Economic-Cost Evaluation of Physical AI Agents

AI summary

Overview

Research area: Embodied AI and robot navigation benchmarking, sitting at the intersection of high-fidelity physics simulation (NVIDIA Isaac Sim), autonomous last-mile delivery, and cost–revenue modeling with break-even analysis.

Technical level: Intermediate. The benchmark design and economic framework are accessible without deep robotics background, but the paper assumes familiarity with navigation stacks (Nav2, AMCL, GPS localization), imitation-learning policies (GNM, ViNT, NoMaD, NavDP, CANVAS), and standard cost-accounting concepts (CAPEX, OPEX, contribution margin).

Scope: CostNav evaluates seven navigation baselines on a simulated Segway E1 delivery robot over 100 food-delivery episodes, converting physics-level signals (collision impulse, jerk, energy draw) into real-world-referenced monetary costs to determine whether any method is economically viable.

What This Paper Is About

Existing navigation benchmarks score robots on task-oriented metrics such as success rate, collision rate, path length, and navigation time, but these metrics say nothing about whether a delivery robot deployment actually makes money. CostNav introduces an economic navigation benchmark that connects simulator-measured behavior to real-world financial data drawn from SEC filings, Abbreviated Injury Scale (AIS) injury reports, and commercial delivery pricing, then scores each navigation method by profit per run and break-even point. The central question the paper asks is: which navigation approach minimizes cost and maximizes revenue on the path to profitability?

Key Contributions

1. High-fidelity physics simulation with sim-to-real evidence. CostNav grounds cost metrics in Isaac Sim physics rather than binary collision outcomes, mapping collision impulse and its variation (Δv) to the Abbreviated Injury Scale and crash reports, and capturing jerk-induced food spoilage and pedestrian injury liability. A sim-trained CANVAS policy deployed on a physical Segway E1 along an outdoor urban sidewalk route achieved 62.5% SLA compliance across 8 delivery scenarios, close to its 70% simulation rate.

2. A real-world-referenced cost–revenue model with break-even point analysis. The framework spans pre-run capital expenditure (CAPEX) and per-run operating expenditure (OPEX), and computes the number of deliveries required to recover capital expenditure. Revenue is modeled from actual delivery service pricing, modulated by success rate and Service-Level Agreement (SLA) compliance, with every parameter drawn from public sources such as SEC filings, AIS injury reports, and commercial delivery pricing.

3. The "profit per run" metric. CostNav proposes contribution margin per delivery (revenue minus OPEX) as the evaluation target, and defines BEP = CAPEX / (R − OPEX), where a negative contribution margin means the BEP is undefined and every delivery increases cumulative losses.

4. A fully open-source benchmark. The release includes simulation scenarios, real-world-referenced cost models, baseline methods, training code and dataset, and evaluation code, hosted at https://github.com/worv-ai/CostNav.

Main Findings

• No method is economically viable. All seven evaluated methods yield negative contribution margins and none reach break-even. The specific contribution-margin dollar figures are rendered incompletely in the available text, but the paper states the sign is negative for every method.

• Pedestrian safety cost dominates OPEX. Under the current cost model, the largest modeled OPEX component is pedestrian safety cost, ranging from $9.3/run (NavDP) to $29.89/run (ViNT) across baselines, with CANVAS at $14.38/run.

• CANVAS leads on task success. CANVAS, using only an RGB camera and GPS, attains the highest task success and the least-negative margin among methods with non-zero SLA compliance (70% SLA), outperforming LiDAR-equipped Nav2 w/ GPS (46% SLA) and Nav2 w/ AMCL (43% SLA).

• NavDP's "least negative" margin is a degenerate result. NavDP has the least negative margin, but the policy barely moves, producing 91% timeout and 0% SLA compliance, so both revenue and OPEX stay near zero.

• Smoothness matters for spoilage. Nav2 with GPS-only reaches a 29% food-spoilage rate because its DWB local planner is tuned to hold forward speed, causing abrupt lateral evasive maneuvers. CANVAS, trained on human teleoperation collected under the explicit instruction to drive smoothly so food does not spill, reaches 0% timeout (lowest of all baselines) and 10% spoiled.

• Several learned policies fail to reach the goal. GNM, NoMaD, and NavDP achieve near-zero SLA compliance (0%, 1%, 0%) with timeout rates of 84%, 78%, and 91%, getting stuck on or colliding with obstacles. ViNT reaches 10% SLA compliance but has a 36% physical-assistance rate, indicating frequent collisions requiring operator intervention.

• Failure causes are diagnosis-specific. GNM, ViNT, and NoMaD condition on topological sub-goal images sampled along the demonstrated path; when the robot deviates to avoid a crossing pedestrian or make a sharp turn, image-based localization fails out of distribution and the policy cannot recover. NavDP conditions on a single image goal plus a point goal, which provides no intermediate guidance and does not scale to long-horizon routes.

• Sim-to-real transfer holds at the episode-outcome level. In the real-world CANVAS deployment, SLA compliance was 62.5% versus 70% in simulation, spoiled 12.5% versus 10%, timeout 0% in both, and physical assistance 25% versus 20%. Average velocity was 0.9358 m/s real versus 1.1003 m/s simulated. Real-world hardware cost was $11,000 (total CAPEX $13,165.79) versus $10,000 simulated (total CAPEX $12,165.79), with a real contribution margin of −$19.65319277072429 per run.

• Cost ranking is robust to the weakest proxy. Even setting the pedestrian-safety term to zero leaves every evaluated method with a negative margin, so the viability conclusion does not depend on that estimate.

• Property damage was negligible in simulation. Property damage cost is $0/run for all seven baselines in the main simulation results, though real-world CANVAS deployment incurred $8.75/run from plant and fence contact.

Methodology in Plain English

The researchers built a simulated city: a 200 m × 200 m sidewalk network in NVIDIA Isaac Sim, containing distinct zones such as construction sites, work areas, and standard sidewalks, populated with pedestrians at configurable densities and with urban infrastructure obstacles. A single Segway E1 delivery robot — chosen because it is commercially deployed and its specifications and component prices are publicly documented — carries out popcorn deliveries across this map.

Rather than counting successes and failures, the simulator logs physical quantities: average velocity, average mechanical power, collision impulse, collision Δv, and runtime. These raw measurements are then run through a cost model.

The cost model has two halves. CAPEX covers upfront hardware (robot plus optional LiDAR and GPS) and, for learning-based methods, the cost of paying data collectors at $33/hour to gather training data — totaling $2,165.79 for the learning baselines. OPEX covers per-run electricity (computed from simulated power draw and an electro-mechanical efficiency factor), repairs (driven by "physical-assistance events," meaning collisions that tip the robot or exceed a recoverable impulse threshold), customer compensation for spoiled food and timeouts, pedestrian injury costs (derived by mapping Δv to AIS severity probabilities and then to dollar costs per severity level, adjusted for vehicle weight differences), and property damage to bollards, mailboxes, trash bins, and building glass.

Revenue per delivery is the market robot delivery price ($3.49/run) multiplied by SLA compliance, and is zero on a timeout. If Service-Level Agreement compliance fails, the run earns nothing. Subtracting OPEX from revenue gives contribution margin per delivery; dividing CAPEX by that margin gives the break-even point.

The seven baselines span two families. Rule-based: Nav2 with AMCL localization and Nav2 with GPS-only localization. Learning-based: GNM, ViNT, NoMaD, NavDP, and CANVAS, all finetuned on a 40.71-hour CostNav teleoperation dataset. Each method ran 100 popcorn delivery episodes. CANVAS was modified to predict command velocities end-to-end through the FAST tokenizer, replacing its original waypoint tokens and the separate PD controller.

Why This Matters

Impact on research. CostNav reframes navigation evaluation around a question that conventional benchmarks cannot answer: does this policy make money? It shows empirically that high task success does not imply economic viability — the method with the highest task success still yields a negative contribution margin. The benchmark also introduces physics-grounded cost signals (collision impulse mapped to injury scale, jerk mapped to food spoilage) as first-class evaluation quantities rather than reporting them separately from task metrics.

Real-world applications:

• Last-mile food delivery robots on urban sidewalks and university campuses, where spoilage from sharp turning and liability from pedestrian contact are the dominant cost drivers.
• Warehouse and logistics mobile robots, where the paper notes autonomous vehicles transport goods and cost is a principal adoption factor.
• Hospital and construction-site service robots, domains the related-work survey identifies as having documented deployment economics.
• Fleet-level capital planning, since the BEP calculation tells operators how many deliveries are needed before an upfront hardware investment pays back.

Industry relevance. Every cost and revenue parameter is sourced from public filings and market data — robot retail prices, LiDAR and GPS module prices, SEC-filed useful-life and fleet-ratio figures, the U.S. average retail electricity price ($0.2704/kWh), the $3.49 campus robot delivery fee, and the $35.42 average refund price — rather than simulator-invented values. That makes the benchmark legible to operators and investors reasoning about real deployments, and it exposes the gap between research metrics and commercial deployment.

Future Directions

1. Broaden sim-to-real validation. The current real-world check covers one policy (CANVAS), one platform, one route, and 8 delivery scenarios. The authors call for collecting more real-world deployment data to strengthen the transfer evidence and to refine the pedestrian-injury cost proxy, which currently rescales passenger-vehicle injury costs by mass ratio.

2. Expand the economic model. CAPEX does not yet include training compute or rule-based mapping and environment-survey costs; revenue does not account for demand or dynamic pricing; and episode outcomes do not cover failure modes such as battery depletion. Rule-based mapping costs are explicitly left for future extensions.

3. Improve the baselines. A subset of learning baselines reach 10% SLA or below, and the authors emphasize that results reflect performance under a common CostNav setup rather than each method's absolute ceiling. Method-specific optimization and stronger baselines are open work.

4. Achieve economic viability. The paper explicitly challenges the community to reach viability on CostNav, which scores methods by cost–revenue outcomes rather than task metrics. Raising SLA compliance, reducing spoilage, and cutting the pedestrian-injury term are the levers the results suggest.

Target Audience

Researchers working on embodied navigation, robot learning, and physical AI evaluation; engineers building or benchmarking autonomous delivery stacks; and technical readers in robotics operations or investment analysis who need to connect navigation performance to deployment economics. Readers interested in benchmark design will find the cost–revenue modeling approach transferable beyond navigation, since the paper positions CostNav as extending value-oriented evaluation from digital tasks to embodied systems.

Authors’ abstract

Current navigation benchmarks focus on task success but do not capture the economic constraints essential for commercializing autonomous delivery systems. We introduce CostNav, an Economic Navigation Benchmark that evaluates physical AI agents on a cost-revenue and break-even analysis, pairing Isaac Sim's collision and cargo dynamics with industry-standard data such as Securities and Exchange Commission (SEC) filings and Abbreviated Injury Scale (AIS) injury reports. To our knowledge, CostNav is the first physics-grounded economic benchmark to use regulatory and financial data to quantify the gap between navigation metrics and commercial deployment, revealing that high task-success rates alone do not ensure economic viability. Evaluating seven baselines (two rule-based and five imitation-learning methods), we find no method economically viable: all yield negative contribution margins. CANVAS, using only an RGB camera and GPS, attains the highest task success and the least-negative margin among methods with non-zero Service-Level Agreement (SLA) compliance (-\$28.40/run), outperforming LiDAR-equipped Nav2 w/ GPS (-\$37.34/run). A sim-trained policy evaluated on a real delivery robot yields SLA compliance close to its simulation result, indicating that policy performance in CostNav's simulation transfers to real-world deployment. We challenge the community to achieve economic viability on CostNav, which scores methods by cost-revenue outcomes. All resources are available at https://github.com/worv-ai/CostNav.