Parallax
Agent Reliability and LLM Evaluation Framework
Parallax
Problem
Autonomous agents fail in ways unit tests miss — silent tool misuse, degraded reasoning, and recovery paths that never fire. Pass/fail suites do not measure detection lead time or diagnosis quality.
Approach
Designed ground-truth failure cases, semantic traces, fault injection, and rule-, ML-, and LLM-judge baselines for autonomous-agent regression testing.
Instrumented agent execution with OpenTelemetry and Jaeger to compare detection lead time, diagnosis quality, and recovery behavior.
Established repeatable reliability benchmarks across failure scenarios beyond simple pass/fail evaluation.
Results
LLM-as-judge · OpenTelemetry · failure benchmarks
- Repeatable agent reliability benchmarks across injected failure scenarios
- Measurable observability signals for detection lead time and recovery
- Comparable rule, ML, and LLM-judge baselines on the same harness
Stack
What I learned
Reliability for agents is an evidence problem. Traces, judges, and recovery metrics matter more than a green checkmark on the happy path.