https://elbanic.github.io/tags/observability/Recent content in Observability on Euijun's Personal BlogHugo -- gohugo.ioen-usMon, 15 Jun 2026 20:00:00 +0900https://elbanic.github.io/posts/observing-agent-quality-with-evaluation/Mon, 15 Jun 2026 20:00:00 +0900https://elbanic.github.io/posts/observing-agent-quality-with-evaluation/<p>Across <a href="https://elbanic.github.io/posts/develop-and-operating-agents-in-a-distributed-environment/">Part 1</a> and <a href="https://elbanic.github.io/posts/managing-agent-memory-in-a-distributed-systems/">Part 2</a> of this 「Agentic AI Dev Note」 series, we made our agent operable in the cloud. Now it&rsquo;s time to check whether that agent actually behaves the way we expect. There are three ways to measure an agent&rsquo;s quality.</p> <ul> <li><strong>Code quality</strong>: Most of it can be verified with unit/integration/e2e tests. The same input produces exactly the same output, so it can be verified precisely.</li> <li><strong>System quality</strong>: Measured through runtime error rates, latency, dependency failures, and the like, via alarms and monitoring. Canary tests can monitor system quality too.</li> <li><strong>LLM response quality</strong>: Measures whether the LLM produced the expected answer to the user&rsquo;s input.</li> </ul> <p>But how do you guarantee the response quality of a non-deterministic LLM, where the same input can yield a different output every time? For the first two, the same input gives the same output, so you can <em>prove</em> it is &ldquo;correct.&rdquo; LLM responses, however, cannot be proven. And if you can&rsquo;t prove it, the only option left is to <em>observe</em> it: watch the trend of behavior instead of any single correct answer. This post focuses on that.</p>