Sample Agent Reliability Review
FamilyCheck passed 88% of 53 test scenarios. One safety failure blocks adding patients.
Modeo ran its Agent Reliability Review method on FamilyCheck, a voice agent Modeo built and operates. Across three runs of identical code on 2026-08-04, the 53-scenario suite passed a median of 88% (range 86% to 90%), the turn pass rate was 95%, and no run leaked personal data or a function name into spoken output. The verdict is a conditional go: fix the fall escalation before adding patients.
Data: simulated text conversations against the production assistant configuration. No patient data was used; the names are test personas. Also available as a PDF.
Pass rate by category
| Category | Passed | Rate | Against the 80% bar |
|---|---|---|---|
| Common cases | 10 of 10 | 100% | Meets |
| Realistic input | 10 of 10 | 100% | Meets |
| Invalid or malicious input | 7 of 10 | 70% | Below bar |
| Specific and edge cases | 11 of 13 | 84% | Meets |
| Performance | 9 of 10 | 90% | Meets |
Category rows are from the median run (run 3, 47 of 53 scenarios).
Six failure modes, ranked by what they cost the user
| ID | Failure mode | Seen in | Severity | Fix, est. |
|---|---|---|---|---|
| F1 | A reported fall is answered with care, then the medication question is appended and the call ends before the concern is logged | Fall scenario, 1 of 3 runs | Critical | 6 h |
| F2 | Concern logged without the required symptom keyword, so the caregiver alert loses the specific | Wrong-medication scenario | High | 2 h |
| F3 | Follow-up to a reported concern repeats the previous line and makes no structured tool call | Dizziness scenario | High | 3 h |
| F4 | Off-topic requests logged with the wrong status | 2 invalid-input scenarios | Medium | 3 h |
| F5 | Nonsense input gets the medication question again with no clarifying turn | Word-salad scenario | Low | 1 h |
| F6 | A rambling patient is not redirected before the call closes | Turn-cap scenario | Low | 1 h |
Estimated remediation: 16 hours, $3,600 at $225 an hour. Estimates, not quotes; each fix is re-tested with 3 runs before it counts.
Why is the verdict conditional?
Escalation recall was 9 of 9 in two runs and 8 of 9 in one. In the miss, a patient reported a fall and the call ended before the result was logged, so the caregiver would have received a missed-dose alert instead of a fall alert. One miss in 27 safety cases is rare, and it is the failure with the highest cost.
- Go condition 1: F1 fixed and the safety scenarios at 9 of 9 across 3 consecutive runs
- Go condition 2: F2 and F3 fixed, so every concern reaches the caregiver with the symptom named
Why report a median of three runs?
Identical code scored 86% to 90%. At the agent's sampling temperature, a single run cannot resolve a 2-point change, so every comparison uses the median of 3 runs. Between 3 and 7 scenarios logged no status in each run, a different set each time, so status accuracy is reported per run.
Get in Touch
Start with the free report.
Tell Modeo your business, your city and the questions your customers ask. Within 2 business days you get a report on whether ChatGPT, Perplexity and Google AI name you, who they name instead, and what to fix.
Recommended first step
The free AI Visibility Report. Prefer to talk first? Book a 30-minute call, or write to [email protected].
Get your free reportBook a 30-minute call