TL;DR Good developers’ caution about agentic development comes from knowing that built isn’t the same as working in production.
I’ve been surprised by how many excellent developers have been hesitant to go all in on coding agents. These are people who usually try new ways of working before anyone else and in many cases invent new tools and practices that support building the application well. So why have we had so much reluctance?
The easy (lazy?) answer is that they love writing code, or they’re worried about their jobs. I’m sure that’s true for some people, some of the time. And I know it is true for some of the developers I talk to but I always felt there was something else, another fear hiding behind those obvious ones. When we put together the Technology Radar every six months I try to block out a chunk of time for us to do an open space. In September, a discussion we had on agentic development, which pushed it to its limits, finally helped me see clearly what that fear is.
Building it right
Thoughtworkers care deeply about building the right thing and building it right. It’s pretty much our default way of working and thinking and it’s where many of the practices and technologies we invented and championed came from. We want software that is useful and helpful, can be changed next year, and doesn’t fall over in production. Our developers adopted, championed or invented some of the first versions of continuous integration, pairing, automated testing, continuous delivery, microservices and many others early because they helped us build the software right.
Early coding agents made it very easy to produce code, including code we wouldn’t have wanted anywhere near production, written by people who have not learned about the chasm between building something and making it run in production safely. The tools have improved enormously, but experienced developers still have questions. Are we confident this is the right behaviour? Does it work as expected? Are these the right boundaries? Will we be able to evolve it? Does it abide by the principles and practices we learned the hard way to give us confidence it will meet the needs of the customer and stay up in production? What happens when something fails? Will another team be able to live with it? And so on.
Underneath all of those sits a bigger question. How much can we trust a system when nobody has experience of building it this way? Let me state that another way. Nearly every system we have in production now was built the old way, by hand, by people we could trust (hopefully). Everything we build now is the new way and we have not lived with that yet.
After more than 25 years in technology, I’ve come to value the judgment people develop through experience. I regularly put similar questions to models and to senior technologists at Thoughtworks. For the difficult questions, I still get more useful answers from the humans or maybe more insightful with very specific experience that helps me better understand the point. They challenge the premise, recognise a pattern from something that went wrong ten years ago, or spot the organisational problem hiding inside the technical one. I can now access an extraordinary amount of information. I still need people who can help me think through what matters, I still need wisdom and insight. That type judgment is exactly what hesitant experienced developers are trying to protect.
And I want to be clear: they are not hesitant to use a coding assistant, they are hesitant to go all in and let the agents do all the execution. They are hesitant to take their hand off the wheel because they don’t have full confidence the agents have the right guardrails in place to run off on their own.
A week at the whiteboard
Recently, Giles Alexander, our Regional CTO for Europe, brought senior engineers, infrastructure consultants and product people together for a week of working in exactly this way. No code written by anyone. Everything built by agents. The point wasn’t to prove that an airline application could be built in a week. (Funnily enough they got 80% done and we all know what that means… you can’t make stuff up!) The purpose of the experiment was to learn what happens to the team’s work when agents do all of the coding. You can read some of what they learned here.
What struck me in our conversations was how much time they spent at the whiteboard. Working out the domain. Drawing boundaries. Deciding what should happen next and making plans the agents could act on. Meanwhile, the agents kept producing code beyond the team’s working day.
The humans were also designing the feedback around that work: tests, contracts, architecture checks and a delivery pipeline. They were working out how to give agents enough freedom to make progress while keeping confidence in the result. Most of the work at the keyboard was directing agents to put this learning into the harness, so the building and testing agents got it right on the next run and they could have more confidence in what they built. Interestingly in the end the product person committed more feature code than anyone else on the team.
When I asked them to estimate they said they spend about ⅔ of their time at the whiteboard and ⅓ on the harness, which could give us some clue as to how developers will work in the future, but before you get ahead of yourself redesigning your team, that’s not the end of the story…
The application ran. The build was red.
The application ran, but the build was frequently red. It worked on their machine - again you couldn’t make this up if you tried! A constant red build gives me the ick. A red build should mean something: stop, investigate, fix it. If it’s red so often that people stop paying attention, how do we spot the next real failure? And if we can’t trust the checks, a green build stops being reassuring too. That’s exactly the uncertainty continuous integration was supposed to remove, and we’ve been advocating it for over 25 years.
We’re still working through how to address it. Remember this was an experiment to push the very edge of agentic development. Were too many agents working in parallel and stepping on each other’s changes? Was the system not yet modular enough for agents to work on separate parts? Or were changes simply arriving faster than the checks could finish? Or was it a combination of all of these and more? Either way we definitely need to rethink how we will solve the last mile challenges. I can see continuous delivery/DevOps as a movement having a resurgence and a rethink in the agentic era in the same way the rest of the development life cycle is being upended. And it isn’t just our experiment. DORA’s 2025 research found AI adoption is now linked to higher delivery throughput, but to lower delivery stability. Exactly what continuous delivery and DevOps were trying to solve for!
All of these challenges prompted some good arguments internally. What role should test-driven development play when you aren’t reading every line and the agent created the tests alongside the code? Where would acceptance tests, contract tests and checks at system boundaries give us more confidence without being too brittle? How much design should we do before setting an agent loose, given how quickly we can revise an approach? Is there an appropriate point we will know we have gone too far? One of the biggest puzzles in all this is the size of the batch. I have heard many teams are moving from Stories to Epics. Great for speed of development but can your CI pipeline handle that size unit of work at speed and in parallel?
The experiment team didn’t settle all of this in a week or even two (they ran it twice with different teams). I wouldn’t trust anyone who claimed they had. But this is exactly the kind of work I want experienced developers involved in: experimenting, questioning and helping us discover which practices hold up and where we need some new ones.
Off the groomed slopes
After coming away from this conversation with some of the senior developers involved I kept thinking about learning to ski. Kids will often head straight down the hill without fear, kind of like a new builder who has never put software into production and has never fixed an on-call issue at 3am (something I will never forget, right up there with every production issue I personally caused and the resultant learnings from those, which are burned in my brain forever). When I first tried to ski as an adult, I was thinking about what I might break. I wanted to ski, it looked so fun and freeing, but I was old enough to see the consequences and I didn’t fancy hobbling around London with a broken limb or worse. My biggest issue was I hadn’t yet learned how to turn or stop effectively and without that I was a runaway train on a ski slope.
When I shared a draft of this post with Martin, who also learned to ski late, he pointed out where the comparison breaks down (like all analogies I like to push them to the edge, pun intended). On a ski slope you’ve watched countless people reach the bottom with no broken bones. The instructors have taught thousands of people before you on the same slope. They know which skills matter and how to teach them. Agentic development right now is more like a backcountry trip across unknown territory. We don’t yet know which skills will matter most. Not many people, if any, have finished the trip safely and the feedback on how to navigate it is slow or non-existent. As Martin put it, we won’t know whether a heavily agent-built system works until it has run in production for a while, survived a few rounds of change, and not sent all our planes to Aberystwyth. Built is not the same as working in production. Or as Charity Majors puts it, “staging is just a glorified laptop. Only prod is prod”. Which is why when someone says “it works on my machine” it’s usually intended as a joke. Or at least I always laugh.
That solidified a nagging discomfort I have had with leaders, technical or non-technical, claiming devs aren’t curious, or are laggards slow to adopt, and putting that down to easy-to-point-at fears like missing writing code (I do too) and fearing for our jobs (who isn’t right now?). Our best developers aren’t nervous beginners on a nursery slope that everyone else has already mastered. They’re looking at unmarked terrain and asking sensible questions about the route. Telling them to be less cautious misses the point. They need to be on the team that maps it.
Of course, experience can also make us too attached to familiar practices. Some of what we learned on the groomed slopes won’t hold out there, and we need to be willing to test our assumptions.
A challenge for both sides
So there’s a challenge here for both leaders and developers. Leaders need to listen carefully enough to tell useful concerns from reluctance to change, and to create real room for experimentation. That means time, permission to get things wrong, and not judging success by how much code the agents produced. Developers need to bring their experience into those experiments and be prepared to find that some of their assumptions no longer hold.
Some familiar practices will become more important. Others will need a different shape. Nobody is going to hand us a map first.
If you care deeply about building the right thing and building it right, your judgment is exactly what this new way of working needs. Bring the caution. Then let’s set off into the snow.
The webinar hasn’t been updated in years, so the vendor-specific information is mostly obsolete. However, I’m sure you’ll still find the requirements, architectures, designs, and technology overviews sections relevant.
When Jake Paul and Mike Tyson stepped into the ring in November 2024, we delivered the fight to roughly 65 million concurrent streams. On an ordinary day our infrastructure takes in more than 2 billion requests and handles upwards of 38 million real-time logging events every second. At that scale, the whole picture doesn’t fit into a single engineering team’s head or onto a single dashboard. With more Live content, Podcasts and real-time streams, our need to understand and update our systems as quickly as possible becomes imperative to provide a good user experience. Understanding the system at this scale and within the time constraints we have, often becomes a daunting exercise in finding a needle in a haystack, in a dark jungle, blindfolded.
This post is about that search for the needle.
The Four hours before a fix
Consider a regression of the kind we see in a given week. The home screen where you see the list of lists of movies, starts taking about 1.5 seconds longer to render. An alert fires. And then we go looking for what caused this issue and how do we fix it?
As it so happens, a regression on the application could be related to anywhere in the system — a client app regression from a new version deployment or A/B test rollout, a load shedding in the infrastructure, a GraphQL query responding with bad data because of a service rollout 6 hops downstream, are all suspects at this point. So, answering “what caused the issue?” needs insights from across the system, and this involves combing through layers of data in a meticulous manner.
The MELT Layer
“What caused the issue?”
Metrics, events, logs and traces, or MELT, in the industry’s shorthand, are the right raw materials for such an investigation, and each of them answers roughly a quarter of the question. We collect observability data across these four pillars, unified into a single observability data layer. This lets us observe our systems in real time, across what our users are experiencing, how our applications are doing and what our backend infrastructure and services are processing and responding to.
MELT — Metrics, Events, Logs and Traces, and how they are captured across the system.
The investigation that follows may look like this in each layer:
What each layer of MELT tells us and what it leaves out
The timeline of an investigation
So how long does it take to investigate an issue, figure out the root cause and fix it, so our users stop running into this issue again?
A timeline of an investigation: ~4 hours
In our chosen example, four hours. Very little of it went into computing anything difficult. Most of it went into reconstructing relationships that our observability systems already knew but we had never written down anywhere a human or machine could simply read and discover.
The disagreement underneath
The instinct here could have been, around 2023, to reach for more data or better dashboards, and while both help at the margins, neither addresses what makes these investigations slow.
It’s 2026 now, so we might say, “hey, agents”, but without the right context delivered to the agents, AI systems hallucinate, and especially when teams are on-call and need a fix “now”, we cannot let inaccurate agentic analysis lead us down the wrong path. We need the right needle, not any needle.
Each of our systems is internally coherent and externally isolated. Atlas, our telemetry platform, knows applications and their time series. Radar, our alerting system, knows thresholds and firing history. incident.io knows severities and responders. PagerDuty knows rotations. Spinnaker knows what shipped and when. And our Slack history knows, in a form no query engine can reach, which engineer solved something very similar last quarter and what they found.
The difficulty is that they do not agree on what they are talking about. A thing that is an application in one system is a service in a second and a cluster in a third, with names differing by a suffix, a casing convention, or a rename only a couple of people remember (SMEs, ring a bell?). Joining across them requires a human carrying the mapping in their head, and that person becomes the bottleneck in such cross-domain investigations. The hard part of end-to-end observability is probably better understood as a data modelling problem.
What we lacked was an explicit, machine-readable agreement about the entities in our operational world.
The Ontology
This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object. Consider the following example triples.
In the example here, we have already encoded six facts, and can answer in one query, a question that used to span three tools.
“Which incidents affected applications owned by Team Bedrock in us-east-1?” is a single query in place of looking up incidents, Spinnaker and Radar.
We call our ontology layer, The Observability Ontology. It acts as the schema, rule set and shared agreement that standardizes how different systems understand entities like applications, services, incidents, deployments, etc.
We built it on the W3C stack for this: RDF for the data model, OWL for the ontology, SHACL for validation, and SPARQL for querying. These standards are well specified and widely implemented, which keeps the model portable across graph stores.
The data from the ontology itself, is stored as a knowledge graph, persisted to QuipuDB, our homegrown RDF graph database, a distributed quad store based on redb, a Rust-based embedded key value db. Data here is generated by parsing raw telemetry, logs, and incident discussions into triples (subject-predicate-object facts).
Every harvest also produces a full Turtle serialization for archival, making it human-readable and diff-friendly, which has proven more useful for debugging than we expected.
Our ontology is organized into twelve namespaces, each owning one operational domain:
There are two decisions about this ontology that mattered while we built it, so they’re worth sharing.
The first is that we refused a generic Service type. “Service” means four different things across our tooling, so applications resolve to exactly one type, ops:Application, and every one of them is validated against our software catalog of roughly 49,000 entries before the entity is created. Unvalidated strings like usernames, team names, dependency names scraped from network topology, never become applications. We also cap dependency edges in both directions, because one popular service can carry thousands of reverse dependencies and would otherwise dominate the graph.
The second decision concerns what we chose to model. Incident chat carries the reasoning behind an incident. The hypotheses tried and discarded, the mitigation that worked, who knew why, is all context in the incident’s Slack channel and that knowledge evaporates when the channel goes quiet. So a 200-message, 50-person, 15-service incident channel becomes a structure, converting the operational chaos of an incident into ordered triples:
Creating determinism from non-determinism
Building this graph means using tools that are inherently stochastic. LLMs summarize discussions. Fuzzy matching resolves service names. A probabilistic engine proposes causal chains. None of these produce the same output twice with certainty, but what comes out at the end of our system is a deterministic graph that answers a SPARQL query the same way every time.
The ontology is the mechanism that creates this determinism through its pipelined approach. We think of the pipeline as a ladder that drives entropy down, where entropy means the proportion of information still trapped in unstructured form. For example, at the start of this pipeline, an LLM could say anything, but it cannot emit a triple that violates the type system. We constrain the output space tightly enough and a stochastic process yields a deterministic contract:
# The model may produce any prose. Its output MUST fit these predicates: category IN ["incident", "question", "discussion", "decision"] sentiment IN ["positive", "neutral", "negative", "urgent"]
# Result — a deterministic triple: thread:123 slack:category "incident" . thread:123 slack:isIncidentRelated "true"^^xsd:boolean .
There are five layers that get us down this ladder. Any single step of this ladder is non-deterministic. But in aggregate, the chain becomes deterministic. That inversion is the whole trick. We describe them below:
The five layers to create determinism from non-determinism
Enrichers: Agents as entropy reducers
We use what we call, enrichers, to reduce entropy across the knowledge graph. Dozens of enrichers run over the graph. Each one of them takes noisy input and emits typed triples, and each reads what earlier enrichers wrote. The contract for an Enricher is deliberately narrow:
class Enricher: async def _enrich_async(self, ...): raw = await self.fetch(...) # 1. read noisy input result = self.analyze(raw) # 2. process (may be stochastic) graph.add((uri, RDF.type, OPS.X)) # 3. emit TYPED triples graph.add((uri, OPS.pred, Literal(result.value))) return EnrichmentResult(success=True, triples_added=count)
Three invariants that are standard across the enrichers:
every entity carries an rdf:type,
every URI follows a single naming scheme, and
the catalog gates every ops:Application.
Two operational choices are worth naming here, as well. The enrichment pipeline runs sequentially — load the graph, add triples, and save.
This sequencing removes an entire category of race conditions at a cost in wall-clock time we are happy to pay. And calls out to external systems like metrics, logs, events and the rest could be best-effort, so a slow downstream API doesn’t block the graph harvest, albeit some degradation. To that end, we are also working to translate these separate systems into the ontology, such as the Metrics Ontology and the Alerts Ontology, so everything could be queried from the Ontology itself.
The Correlation Engine: finding the correlation chains
The correlation engine assimilates the vast number of signals from diverse input sources and condenses them into causal or correlatory chains of events. Today, the engine reads thirteen signal types, including log errors, metric anomalies, slow and failed traces, alerts, incidents, deployments, config changes, experiments, autoscaling, commits, end-to-end bottlenecks, and error patterns. It then searches for paths among them. Some ordered pairs are known to be causal and carry a high prior weight. For example, deployment before log error, config change before metric anomaly, failed trace before incident, and so on. Others, which show high correlation are considered correlation chains. Such chains are scored on temporal proximity and severity, inside a ten-minute window, with anything under 0.6 confidence getting discarded.
The Correlation Engine
The role of the engine here is to propose with evidence, not to make conclusions. Hence, the engine stores its chains along with the graph records that were used to create them.
A 0.87 chain in the above example, is a strong hypothesis for a human or an agent to check, and the graph records the confidence alongside the claim so that consumers can understand why a chain is what it is and weigh their options to make decisions.
The knowledge flywheel
Each harvest cycle turns the same four steps, and each rotation leaves the graph richer for the next. This is the Knowledge Flywheel, leading to continuous enrichment and improvement of the context about our system, stored in the knowledge graph.
The Knowledge Flywheel
Observe collects the signals. Enrich adds typed triples. Infer finds patterns invisible in the raw data. Adapt feeds knowledge back, so that established patterns become detection rules. The compounding result of the four phases leads to a self-improving system. A causal chain discovered in one incident becomes a known pattern for the next, and the ontology gradually stops being a schema and starts being a memory.
Consider the example above. When we first see an incident in api-gateway caused due to a timeout and resolve it, we convert this knowledge into entries to the ontology. This becomes encoded knowledge. As we go through multiple cycles of this encoding, the graph starts learning patterns. After N cycles, patterns become wisdom — “api-gateway deploys cause 40% of SEV-2s” and “30% of deployment issues in api-gateway come from Connection Timeouts”. Surface this as an insight and we proactively start reducing incidents, not just react quickly to new ones.
Putting the graph to work
On top of the graph sits AutoRCA, our internal agentic AI solution, built with a set of specialist agents coordinated around one question. Walking the home page regression from the top of this post, the agentic system’s reasoning chain might look something like this:
Each specialization to the inputs is built by people who understand their respective domain deeply. That division of labour only works if the specialists can pass findings to one another without having to agree on the terms of communication or even have knowledge of each other. The ontology acts as the interface. Each specialist learns one vocabulary — the graph’s. A team writing specialized use cases for how they use traces record their findings as triples about an ops:Application, the logs specialization reads triples about that same ops:Application, and neither needs to know the other exists. A new specialist learns the same vocabulary once and can immediately read everything the other six have written.
Set against the investigation we opened with, the change is narrower than it might sound. The alert still fires at 3:30pm, an engineer still has to weigh the evidence, and the fix still has to be written by someone who understands the code. The hour or two that used to go into assembling context — finding the owning team, reconstructing what shipped, correlating information from different parts of the system, recalling that something similar happened last quarter — all that work is the part the ontology and the graph absorbs. We are early enough that the ranked hypotheses are sometimes wrong, which is why each one carries the evidence that produced it. But when the engineer now starts the investigation, they are already holding the picture that used to take half an afternoon to assemble.
If you want to build one of these
A fair amount of what we have built, we learned by getting it wrong first. Based on this experience, these are the principles we would carry into a second attempt.
Model identity before anything else. Choose one authoritative catalog per entity kind and validate against it before creating an entity.
Treat the ontology as code. Version it, review it, test it, and let changes arrive as pull requests. Treating the ontology as code lends itself to agentic development and lets teams scale and contribute to it.
Separate concepts from instances. Concepts change on a human timescale and instances change continuously, conflating them makes both harder to evolve.
Build on open standards. RDF, OWL, SHACL and SPARQL are specified, implemented many times over, and portable. A bespoke graph model ties a future consumer to one implementation.
Constrain stochastic tools with a type system. This is the principle we would defend hardest. Models are useful precisely where determinism is impossible, and a sufficiently narrow output contract converts their output into something a query engine can trust.
Make enrichment best-effort and ordered. A slow dependency should utmost degrade the graph, never block it. And ordering enrichers so each builds on the last buys more accuracy and that’s worth the tradeoff with parallelism that buys speed.
Model human reasoning alongside machine telemetry. The most valuable edges in our graph came from discussion. Telemetry records what the system did, discussion records what people understood and interpreted. The reasoning is more valuable to store and serve as wisdom when we need it.
Keep humans as the approvers. Let agents propose fixes, enrichers, and ontology changes, and route every proposal through a review process with humans in the loop.
What comes next
The graph makes three things possible. First, automatic root cause analysis, where a regression arrives with a ranked set of graph-backed hypotheses attached. Next, automated remediation, where a sufficiently confident diagnosis carries a proposed action like a targeted rollback, a traffic shift, a code change. And eventually infrastructure that heals itself, acting on a predicted failure before members feel it.
Alongside this, we plan to integrate the Observability Ontology with Upper, the domain-modeling language at the heart of Netflix’s Unified Data Architecture (UDA). Upper lets teams describe business and system domains as interconnected, machine-readable models and preserve their meaning and constraints as those models are projected into concrete schemas and systems. This integration would connect operational concepts such as applications, alerts, deployments, and incidents to the broader set of domains modeled across Netflix, giving engineers and agents a shared semantic layer for querying, validation, and reasoning. We will begin with a proof of concept that models incidents in Upper and use what we learn to determine where the two systems should converge.
Acknowledgements
None of this would exist without the teams who share their data, expertise, and patience with an early system. We want to specially acknowledge the efforts of our own team members who made this possible from very early on in this lifecycle, Kaela Marsheck and Aditya Gaur. We are also especially grateful to our collaborators in Netflix’s ontology working group, Peter Royal, Alex Hutter, and to Alexandre Bertails and members of the Unified Data Architecture (UDA)/Knowledge Foundations team for sharing Upper and their domain-modeling expertise, and for helping us explore how the Observability Ontology can become part of a broader semantic foundation at Netflix.
The positive tests are coming mainly from Western states, including California, Washington and Arizona. Regardless of where you're located in the U.S., though, most people are recommended to get their annual flu shot in September or October, and the same goes for this flu season.
The annual flu shot is the best way to prevent hospitalization and death from seasonal influenza. Nearly everyone ages 6 months and older is recommended to get a flu vaccine, and older adults have several special options, including a high-dose shot, a vaccine containing an extra immune-stimulating ingredient called an adjuvant, and a brand-new mRNA vaccine.
Given that flu season is off to an earlier start this year, our latest poll asks whether you plan to get vaccinated or have gotten vaccinated already. Tell us your thoughts in the comments below.
With the growing number of SRv6 implementations that netlab supports, it made sense to create a set of topologies to help you get started exploring the SRv6 world. The sample topologies include:
Log in
