Same runtime evaluated in isolated JavaScript realm without fetch, XMLHttpRequest, WebSocket, require or imports; 100 answers equal ordinary execution. Not an OS-level network-disabled device test.
Failure and policy checks
Injected faults and stricter policies exercise specified software behavior. The unknown-acquisition-date case uses the actual source limitation. These checks do not estimate field reliability.
Corrupted bytesblocked
Injected bit flip fails expected SHA-256 before sampling.
Wrong horizontal frameblocked
Injected incompatible frame is refused; runtime does not guess a projection.
Wrong vertical datumblocked
Injected unresolved height reference is refused.
Stale receiptblocked
Injected stale retrieval receipt fails the declared 30-day test policy; this is not a source-acquisition freshness claim.
Unknown acquisition dateblocked
Actual mixed-mosaic acquisition date is unknown; strict known-capture-date policy refuses it.
Incorrect payload identityblocked
Injected lineage mismatch is refused even with a recomputed outer digest.
Outside coverageblocked
Out-of-extent query returns unknown; no extrapolation.
Invalid coordinateblocked
Nonfinite coordinate returns unknown.
Missing terrain cellblocked
Injected missing cell returns unknown; validity is preserved.
Scope and remaining evidence
No independent surveyed control or field navigation validation. Absolute terrain accuracy and positioning integrity are not established.
The export samples a mixed-resolution USGS mosaic at 15.447 m spacing; spacing is not native source resolution or accuracy.
Acquisition date is not established per pixel. Retrieval time is not capture time; terrain is not current-conditions evidence.
EPQS comparisons use the same USGS source family and compare sampling behavior, not independent accuracy.
Injected faults test declared software behavior; they do not establish real-world detection rates or an integrity guarantee.
Timing measures warm local numeric lookups and in-memory encoding on the recorded host; it excludes download, reprojection, source decoding and real-world data admission.
Offline evidence is capability-isolated local replay, not a disconnected physical-device or navigation test.
Inspect position error, missing output and sustained recovery.
Synthetic scoring controls · No flight measurements · Research only
ScreenPASS
Peak position error2.0 m
Unavailable output3.0 s
Recovery confirmed12.0 s
nominal · Position available · Target visible · Link available
Checks and evidence
independent truth declaration — PASS
unavailable — PASS
position limit — PASS
false accepts — PASS
recovery — PASS
clearance — PASS
perception latency — PASS
Recovery requires 2 seconds of observed in-limit output without stale intervals. Thresholds are provisional engineering screens.
Clearance and latency are supplied trace values. No estimator, vehicle or sensor pipeline runs here. A passing screen does not establish operational qualification.
Customer maps / Paired evaluation
Your maps. Same test.
Planned protocol · Navigation results not measured
Same recordings. Different maps. Measure what changes.
Establish a reproducible baseline with available public data for the operating area.
02 / Customer data
Same model. Your reference.
Hold the model fixed to isolate the effect of substituting the customer’s maps.
03 / Combined data
Same model. Combined reference.
Test whether complementary sources close coverage gaps or improve reliable fixes.
04 / Adaptation
Adapted model. Your reference.
Compare with configuration 02 to measure the added value of customer-specific training.
Control the comparison
Change the reference. Keep the test.
Use identical held-out sensor logs, routes, outage scenarios and estimator settings for the reference comparisons. Freeze versions and evaluate on the same edge hardware. Keep evaluation regions, flights and dates outside adaptation training.
Report paired differences by route segment, terrain, sensor, altitude, conditions and map age, with sample counts and uncertainty. A fleet-wide average can hide a difficult corridor.
Proposed navigation scorecard
Error distribution
Median and tail position error against independent reference truth.
False fixes
Incorrect locations accepted as valid, with the rejection rate alongside them.
Recovery and availability
Time to first fix, recovery after tracking loss, and time without a trustworthy position.
Confidence
Whether reported uncertainty matches observed error and the mission’s limits.
Edge cost
Latency, memory, power and offline package size on the target computer.
Resolution
How much spatial detail can the reference represent?
Accuracy
How closely do its coordinates agree with independent control?
Freshness
Does the reference still describe the environment the vehicle observes?
Finer pixels do not guarantee better geographic accuracy. Test resolution, alignment and acquisition date separately to find the least costly reference that meets the mission’s limits.
Public references / Explicit limits
NASA, NOAA, USGS and Copernicus datasets would provide versioned comparison sources where their coverage and modality fit. They are not universal ground truth. Reconcile coordinate systems, elevation datums, dates and effective resolution; retain original offsets. Differences may reflect newer customer data or an error in either source. Absolute accuracy requires independent control.
Prediction
Simulated conditions
Expected behavior under stated sensor and environment assumptions.
Replay
Recorded observations
Measured performance on held-out logs; closed-loop flight behavior remains unproven.
Field
Controlled operation
Observed performance within the tested aircraft, area and operating conditions.
This section defines planned evaluation work. No customer-map navigation improvement is measured here. The retained Ojai terrain measurements above remain a separate software benchmark.
The benchmark program
Test the conditions that matter.
01 / Compile
What does the environment cost to produce?
Measure retained bytes, compilation work, data fidelity and provenance.
Local terrain baseline published above
02 / Query
How does the artifact answer?
State the exact lookup, runtime and warm-up. Separate local computation from network and application latency.
Local lookup baseline published above
03 / Degrade
What happens when evidence fails?
Inject known faults and record whether the software returns a result, blocks or recovers.
Specified injected cases published above
04 / Operate
Does it improve the mission?
Evaluate localization, uncertainty and decisions against representative operating conditions and independent truth.
