Mercury Instant versus Claude Opus 5: a five-class image benchmark

1. Research question and metric priority

This evaluation compares a task-specific Mercury Instant classifier with Claude Opus 5 on the same labeled images. Fall recall measures the fraction of reference-labeled fall images correctly classified. Precision measures how often a predicted class matches the reference label. F1 balances precision and recall; macro F1 weights the five classes equally.

2. Test-set composition

Room numbers identify camera views, not independently verified unique rooms. Images from a recording can be strongly correlated. The Mercury team supplied the reference labels; they were used unchanged. Ring images are excluded from this report pending separate reference-label provenance review.

3. Evaluation protocol

Both prediction exports are joined to the frozen image manifest by stable image ID. Labels, image checksums, model weight identity, and prediction completeness are validated before metrics are computed. The model is evaluated as a five-class image classifier; no temporal alert policy is scored.

Classify the person’s position using one label:
- away: No person visible.
- fall: Sitting or lying on the floor or ground.
- in bed: Lying down on a bed or couch.
- sit: Sitting or reclining, including sitting upright in bed.
- stand: Standing or walking.

If multiple people are visible, classify the senior, not the caregiver.

Return only JSON:
{"label":"away|fall|in bed|sit|stand"}

4. Results

5. Failure patterns and variation across rooms

Room-level metrics use each view’s own denominators. Differences in sample size and class mix matter: these are descriptive results, not scenario-adjusted comparisons or counts of independent fall incidents.

6. Visual inspection and product demonstration

The image explorer displays the nearest images by cosine similarity in frozen, feature-extractor embeddings. Its globe is a browsing layout, not a projection with meaningful screen distances. The balanced review collection contains 1,000 images per class, selected across the four paired correctness outcomes; it is an inspection set, not a fresh unbiased benchmark.

The product demo is an existing, separate Ring-camera recording. It does not establish performance for this run or for Ring devices.

7. Cost and inference speed

Inference speed is compared using median timings; serving cost is estimated per inference.

Mercury Instant’s per-image serving cost includes idle capacity and depends on utilization. Claude’s estimate depends on image and prompt token counts, minimal output length, and the pricing assumptions documented below; it is not a current price quote or an observed mean charge across the test set.

8. Limitations

The findings apply to the evaluated models, prompt, labels, and camera sample. Correlated frames and unequal room support mean image counts are not independent sample counts. Reference labels have not been independently re-adjudicated for this report. Benchmark precision does not establish false alerts per device-day or deployment alert precision. Event recall, alert delay, and clinical outcomes were not measured. No room/scenario inventory or confidence intervals based on independent incidents are claimed.

9. Availability

Download both models’ metric tables and confusion matrices or the per-image prediction CSV. The overview and this paper are rendered from the same verified results. Images and individual records remain private; this local review site is not a public dataset release.