On-call work is a race to turn noisy signals into shared judgment. The hardest minutes of that race aren't the ones spent fixing the bug — they're the ones spent figuring out what the bug even is, while everyone in the thread is asking the same question from a slightly different angle.

Blueberry is the system we built at Instacart for those several minutes. It lives in the Slack thread where the team is already coordinating, picks up each alert as it fires, and lands a grounded explanation back in the thread in about three minutes — thousands of times a month.

On-call is mostly a clarity and speed problem

Most on-call pain does not start with a major incident. It starts in the noisy window right after a page fires, when the thread fills with links, dashboards, guesses, and partial context. Someone asks whether a deployment caused it. Someone else asks how broad it is. A third engineer joins and asks what is going on.

In that moment, the hardest problem is often not deep root-cause analysis. It is getting to a shared understanding quickly enough that the team can make a good next move. For us, that made two metrics matter most: time to first insight (TTFI) and time to test theories (TTTT). TTFI comes from auto-triaging every qualifying alert the moment it lands. TTTT comes from engineer-initiated on-demand investigations and follow-ups as the investigation deepens.

Blueberry is a Slack-native on-call reasoning harness

Blueberry was built for that first window of confusion. Blueberry lives in Slack because that is where on-call engineers already coordinate, ask clarifying questions, and make decisions under pressure. We wanted the system to work inside the shared operational conversation so the full team stays in sync in real-time and moves faster.

The right mental model for Blueberry is a reasoning harness: one shared runtime that can load context, expose the right evidence surfaces, apply team-specific overlays, and produce a grounded response in the same thread where the work is happening.

The Slack thread is the runtime: thread state, team profile, skills, and memory compose into the harness, and the answer returns to the same shared thread.

Architecture: a durable, tool-aware reasoning harness

The harness is a small number of components arranged so that any single diagnostic pass is durable, tool-aware, team-aware, and inspectable. The shape matters more than the parts, because most of what makes Blueberry sharp comes from how these pieces interact during a diagnostic pass rather than from any one of them in isolation.

Durable, tool-aware harness: production path plus side-mounted MCP catalog, persisted state, and reviewed-improvement loop.

Ingress is selective. Blueberry watches Slack thread events and only promotes the ones that deserve a diagnostic pass: auto-triggered alerts, @mentions, DMs, follow-ups in an active thread. The balance is between responding to too much or too little.

The job management layer is durable, not best-effort. Diagnostic passes are enqueued onto a Postgres-backed durable queue so that they survive worker bounces, restarts, and takeovers. Multiple worker threads can drain the same queue in parallel. If a worker fails mid-pass, another picks the work up cleanly from the persisted state. For an on-call system that has to keep responding at 3am during a rolling deployment, this is a load-bearing decision.

The processor is the composition layer. Before the agent loop starts, the processor resolves which team owns the channel, loads the team profile, layers in team-specific skills and MCP endpoints on top of the shared baseline, and prepares the tool surface that the agent will be given. Specialization happens here, not inside the agent. From the agent's perspective, the right tools are simply present.

There are three Model Context Protocol surfaces, not one. An in-process MCP runs alongside the agent for fast local helpers that share session state. A standalone shared MCP server hosts the heavier shared investigators, such as deploy and flag analysis, service-dependency lookup, error analysis, and anomaly sweeps, so they can be scaled and updated independently from the agent runtime. Team-hosted MCPs are mounted remotely so that data sources unique to a team show up as first-class tools without being baked into the core. The three surfaces have different latency, sharing, and ownership properties, and the processor presents them to the agent as one tool catalog.

A single diagnostic pass can fan out into parallel sub-agents. The agent loop can spawn focused sub-agents. The alert-investigate skill uses this to launch internal knowledge searches over postmortems, runbooks, and prior incidents in parallel with live signal collection, change collection, and operational context collection. Wall-clock time is set by the slowest collector, not the sum.

Persistence captures reasoning, not just outputs. Every diagnostic pass writes both a session-context snapshot, so the next pass or a different worker can take over without losing thread state, and a reasoning trace, so developers can later inspect how the agent decided what to do. These artifacts are what make the rest of the learning loop possible.

Reasoning ladder: where the agent gets freedom, where we hold the line

For an on-call agent, the meaningful design question is not just which model to use. It is where the system should let the agent reason dynamically, and where the harness should hold a hard line for reliability.

An earlier implementation used a more workflow-shaped agent hierarchy, which was predictable but made it hard for the system to choose what to do next when an alert did not match the expected path. Moving to a more dynamic agent loop let Blueberry decide when to call tools, fan out to sub-agents, deepen into evidence, or stop and respond, while the harness kept reliability constraints around persistence, output structure, and evidence discipline.

The interesting part is the balance of freedom and structure. The model is best at choosing what to look at, weighing evidence, and deciding when an investigation is done. The harness is best at the parts that have to stay stable across thousands of diagnostic passes like capturing reasoning, keeping output predictable, and enforcing the discipline that makes the synthesis trustworthy.

From alert to first insight

Alerts do not arrive in one clean format. They show up as monitor links, exception tracker issues, paging alerts, or raw text copied into Slack. Blueberry starts by normalizing those inputs into a shared context, optionally enriching them with team information, and then gathering evidence in parallel instead of one source at a time.

That parallelism is important. Historical context matters, so Blueberry looks for postmortems, runbooks, and prior incidents. Live operational evidence matters too, so it checks signals, changes, and surrounding context at the same time. The goal is to move from "something fired" to a first grounded explanation as quickly as possible. Across roughly 17k auto-triggered triages in April 2026, TTFI averaged about 3 minutes.

Evidence fan-out: AlertContext to historical, live, team, and conditional collectors, joined into one grounded synthesis.

Example: SEV2 before declaration

Blueberry was automatically triaging a wave of E2E test failures in a staging environment. Before the team had opened an incident channel, it identified the exact root cause: a code change had been deployed ahead of its corresponding database migration. The ORM threw a runtime error when it tried to access an undeclared enum attribute in the orders domain. Blueberry surfaced the full call stack, the deployment sequence, the dependency graph, and a clear explanation of what happened.

Minutes later, the team declared the incident. The timeline below shows the important point: the first grounded explanation landed before the incident room fully formed.

Slack thread screenshot showing Blueberry's auto-triage of the staging E2E failures landing in the channel before the incident channel was opened. 

This is the scenario Blueberry was built for: the window between the first signal and the moment the team assembles. Getting a grounded answer into that window compresses the entire response timeline, from the first alert through remediation.

From first insight to tested theories

Getting to a first explanation is only half the job. The next question is how fast the thread can move from speculation to evidence. Blueberry helps test whether the likely culprit is a deploy, a feature flag, a dependency regression, or a recurring failure pattern that already exists in historical context.

The design here is deliberate. Historical priors should inform the investigation, but live evidence should decide it. Blueberry compares the two, then narrows the next move: inspect a suspicious PR, review a flag change, look at a dependency edge, or go deeper into a trace or stack pattern if the first pass points in that direction. Across roughly 7k engineer-initiated theory checks and follow-ups in April, TTTT averaged about 3 minutes — about the same shape as TTFI on the original alert.

This is where the loop matters most. The agent can inspect the proposed culprit, test adjacent evidence, revise when the engineer adds context, and keep narrowing until the remaining explanation is backed by logs or code. 

Example: 8-turn PR exoneration ending at a smoking-gun WARN log

An engineer walked into a thread worried that his PR had broken staging notifications. Over eight turns, Blueberry worked through the evidence with him. It first confirmed the PR did what reviewers had asked: it converted an IllegalArgumentException to a typed NotFoundException and removed a catch-all Status.INTERNAL handler. When the pipeline failed, Blueberry traced the failure to a byte-diff between two markdown files the PR does not touch and pointed out that master was already broken pre-merge.

The engineer asked whether a separate Slack report from a parallel team was related. Blueberry reframed the investigation around what could plausibly produce the reported behavior, based on what the PR actually changed. It then pulled UpdateConfig calls from mesh logs and found that the only staging error occurred three minutes after the original report, which made it timing-incompatible with the trigger.

The smoking gun came from backend logs: a continuous WARN, "The event doesn't exist or is not configured for notifications: updateDeviceConfig," which had been firing for roughly 34 hours before the merge. Blueberry explained the design: the staging service config had updateDeviceConfig: [], an empty subscriber list by design, so the notification framework correctly logged WARN and returned before posting to Slack. It then drew the distinction in code: updateStoreConfig had subscribers and posted, while updateDeviceConfig had an empty list and stayed silent, both in ConfigManagementService.kt.

The engineer had code-level proof that the gap was a pre-existing configuration, not his change. This is the long-form, multi-turn investigative arc Blueberry is designed to support: not a one-shot answer, but a careful examination backed by code and logs.

Slack thread excerpt showing the multi-turn arc: opening worry from the engineer, Blueberry confirming the PR is likely not the cause, then finding the smoking gun and recommending next steps.

Example: a recurring external outage caught by historical pattern

When error rates appeared to be elevated from an external integration, the instinct was to look for a recent code change. Blueberry checked recent deployments, found nothing suspicious, and then cross-referenced against historical retailer context. It surfaced two prior incidents with the same error signature and the same root cause: a partner-side outage, not Instacart's code. The investigation pointed directly to the retailer's escalation path. Time from alert to actionable conclusion: under 8 minutes. 

Slack screenshot showing Blueberry calling out the matching January and February incidents, the partner-side root cause, the retailer escalation path, and the engineer's "long-standing Sentry patterns" reaction.

The two-pass synthesis was designed for this. The historical match did not boost confidence on its own; it was confirmed by the absence of contradicting live evidence on the deploy side, which let the agent flip from "probable internal regression" to "external dependency" without overshooting.

Why specialization matters more than generality

A fully general agent can sound flexible while still being too broad to feel sharp in production. Blueberry became more useful as we made specialization a first-class design choice. The shared harness still works even without a team-specific setup, but it gets significantly better when teams contribute their own profile notes, skills, and MCP endpoints.

By April 2026 in the early rollout, roughly 55% of teams had onboarded team profiles. The important part is not just that more teams were present; it is that their local habits, runbooks, skills, and MCPs were available to the same base harness, so the system could stay general in architecture while becoming local in practice.

That is where tribal knowledge starts to become reusable infrastructure. The little patterns that usually live in experienced engineers' heads, the "if you see X, check Y" habits, the team-specific data sources, and the known failure modes all become things the harness can reason across in a repeatable way.

Specialization without forking: ~60 onboarded team profiles compose with the shared Blueberry harness into one agent-facing tool catalog — general in architecture, local in practice.

Slack-native collaboration is part of the product

Blueberry is useful partly because it improves the shared conversation, encourages collaboration and extracts tribal knowledge into the shared space. When the system clarifies likely scope, summarizes what changed, or explains why a theory looks weak, that understanding becomes visible to everyone in the thread at once.

That changes who can participate. Late joiners can become useful faster. Secondaries do not need to reconstruct the whole thread from scratch. And more of the team's reasoning becomes widely useful, legible and reusable.

The most useful feedback signals were in the moments when engineers reacted to Blueberry’s sharp insights, speed of root-causing and ability to validate their theories rapidly

Because of the clear benefits of Blueberry, engineers are also investing in the tool, rapidly improving its effectiveness. Teams build skills, write runbooks in a format Blueberry can use, and push back when an investigation is wrong. They can engage with it in the way they engage with a capable colleague.

Slack-native collaboration: reasoning trapped in private browser tabs on the left, vs. legible to everyone in the thread on the right via a shared Blueberry synthesis surface (likely scope, what changed, why a theory looks weak, late-joiner context).

Better on-call hygiene, not just faster triage

 On-call is partly about root-causing alerts and partly about proactively maintaining system health. Blueberry helps by keeping thread context legible across handoffs, reducing repeated reconstruction loops, and giving the team a shared memory of what has already been checked. That becomes most visible during alert storms, when the real problem is lack of convergence.

That same shared memory matters during primary handoffs too: a new on-call can see current state, tested ideas, likely downstream impact, and open questions without restarting from scratch.

The tool is now  part of better on-call hygiene: fewer repeated reconstruction loops, cleaner shift changes, and a better shared memory of what the team already checked.

Example: 90 alerts, one root cause, 3 minutes

When a cluster-wide disruption hit the data engineering Kubernetes cluster, 90 P1 and P2 alerts fired across multiple namespaces in under 3 minutes, a cascade triggered by JobManagers going unhealthy simultaneously. Blueberry handled more than 50 investigations in parallel. Every single one arrived at the same conclusion: a cluster-level infrastructure failure, not application code, not a feature flag, not a bad deployment. 

No engineer had to triage all the alerts. The shared thread became a single point of ground truth before anyone had assembled the incident team. This is also the kind of scenario that pays back the architecture choices directly: a durable job queue, parallel workers, three composable MCP surfaces, and a processor that can produce consistent synthesis at fan-out scale.

Slack channel screenshot showing the volume of P1/P2 alerts firing across namespaces with Blueberry's parallel investigations converging on the same cluster-level infrastructure conclusion. 

What we are testing next

The next frontier is closing the loop between diagnosis, safe action, and learning. Early work is already exercising deploy-failure mitigation: when an internal service deploy fails, Blueberry can gather canary signals, inspect logs and error context, correlate the failure with recent PRs, and decide whether the right move is to redeploy, escalate, or stop because the evidence is unclear.

The key is that action has to move through a safety path, not a shortcut. Before execution, Blueberry needs explicit preconditions, evidence analysis, an action plan, action-plan reflection, an execution policy check, and result reflection. In parallel, reasoning traces and self-assessments are feeding reviewed prompt, skill, and routing improvements. That loop is being tested and scaled as part of the same harness: diagnostic passes produce evidence, safe actions stay policy-gated, and outcomes become learning material.

Policy-gated action and learning: mitigation as a governed path from diagnosis through bounded execution and reviewed improvement.

What changes at the organizational level

As Blueberry became useful in real threads, it started changing behavior. Engineers saw the system helping in the flow of actual work, which made AI adoption feel practical instead of abstract. At the same time, teams had a reason to externalize their own operational knowledge — runbooks, skills, MCPs — in ways the harness could actually use, so the best debugging instincts could be applied consistently, by more of the team.

That is Blueberry’s force-multiplication story. It helps the primary on-call engineer in the moment and gives teams a way to package their best debugging instincts so they can be reused.

Conclusion: Useful AI systems in operations need the right harness

The biggest lesson from building Blueberry is that model quality matters, but harness quality decides whether an agent is actually useful in operations. Grounding matters. Specialization matters. Continuity matters. Collaboration matters. A learning loop matters. When those pieces come together, an agent stops feeling like a demo and starts feeling like real operational infrastructure.

What that looked like in production in a single month: 

• Blueberry ran ~25k diagnostic passes across 270+ Slack channels, ~17k+ auto-triggered alert triages and ~7k+ engineer-initiated theory checks and follow-ups. 

• TTFI averaged ~3 minutes across the auto-triggered triages

• TTTT averaged ~3 minutes across the engineer-initiated follow-ups. 

• The system completed ~23k of those passes successfully with  a 99.9% success rate. 

• The same harness handled about 58k+ Model Context Protocol tool dispatches and supported ~60 onboarded team profiles, with sustained usage in dozens of channels.

Every on-call shift produces knowledge that can disappear. Blueberry is our attempt to make that knowledge durable. The diagnostic harness captures some of it. The team profiles, skills, and learning loop capture the rest. The best operational instincts now compound, instead of expire.

*Examples are illustrative and fictionalized based on real production patterns the agent handled