We fail every day, but that failure becomes the success of our next attempt (feat. Claude Code)

Dev Sentinel: Recording Failure So We Don’t Repeat It

“We fail every day, but that failure becomes the success of our next attempt.”

Failure Repeats

Not long ago, a production incident occurred in a service I operate.

An SQS-triggered Lambda handler processed a single event that spawned an excessive number of DynamoDB transactions in parallel. Eventually, the Lambda hit its timeout limit.

It was a reminder that Promise-based parallelism is not always optimal. IO-bound workloads sometimes require concurrency control or even sequential execution. The code appeared logically sound, yet it failed under real-world resource constraints.

I had a similar experience in a toy project.

A memory-based persistence layer worked perfectly in a local environment, but after deploying to a cloud-based Docker setup, consistency tests failed. The root cause was a stateful design in an environment where container instances could change per request.

Both incidents taught more than simple bug fixes.

Code may be logically correct, but if it ignores the constraints of its execution environment, the system will eventually break.

These lessons clearly help in future projects.

Does Every Failure Lead to Success?

We fail every day. We encounter bugs, make incorrect assumptions, and struggle with performance issues.

Becoming a senior engineer often means accumulating these experiences until they turn into intuition. Operational incidents, scaling failures, and flawed architectural decisions gradually build a sense that “something feels risky.”

However, in reality, not every failure translates into future success. Many lessons are never recorded, never revisited, and similar mistakes reappear in different contexts.

The problem is not failure itself. The problem is that failure disappears.

Dev Sentinel: Detecting and Preserving Failure

Dev Sentinel started from this observation. The core idea is simple: Detect failure cycles, record them, and surface them when a similar situation arises again.

I define failure as a cycle: Attempt → Frustration → (Abandonment | Resolution)

Within this cycle, I designed two concepts.

1. Experience Capture

Detect frustration, then track whether the attempt ended in abandonment or resolution, and store it as experience. This is not just logging. It structures the context: what the problem was, where the difficulty emerged, and how it concluded.

Experience Capture process showing how Dev Sentinel detects and records failure cycles

2. Active Recall

When similar frustration is detected again, surface relevant past experiences. The goal is not to solve the problem automatically, but to remind the developer: “You have seen something like this before.”

To implement this, Dev Sentinel leverages Claude Code’s UserPromptSubmit and Stop hooks.

  • During UserPromptSubmit, it detects signals of frustration (phrasing, repeated attempts, context shifts, etc.).
  • During Stop, it determines whether the failure cycle concluded and stores the structured experience.

Active Recall mechanism showing how Dev Sentinel surfaces relevant past experiences when similar patterns are detected

What If Dev Sentinel Had Existed Then?

Looking back at the earlier incidents:

  • Lambda timeout caused by excessive parallelism
  • Stateful design failure in a containerized environment

Both were not merely implementation mistakes. They shared a pattern: underestimating environmental constraints.

If Dev Sentinel had existed, then in a later project:

  • When designing high-concurrency processing logic
  • When introducing in-memory caching casually

It could have surfaced past experiences as a reminder.

Not to provide the answer, but to ask: “Have you seen this pattern fail before?”

Between Organizational Runbooks and Personal Memory

Many organizations reduce failure through runbooks, SOPs, wikis, and knowledge bases. These are highly effective. However, organizational knowledge does not always align perfectly with individual context.

Dev Sentinel takes the opposite direction:

  • Capture individual failure experiences first
  • Accumulate them over time
  • Return them to the individual when relevant

The expectation is that accumulated personal experience leads to better design judgment, and that judgment ultimately influences projects and organizations.

Closing Thoughts

We fail every day. But failure does not automatically become an asset. Failures that are not remembered are repeated.

Dev Sentinel is not a grand AI system. It is a small mechanism to prevent failure from quietly fading away. Tonight feels like a good time to reflect on what today’s lesson was.

GitHub repo: https://github.com/elbanic/dev-sentinel

← Back to Blog