From Skills to Factories
A practical guide for incrementally adopting and scaling agentic engineering in a larger engineering org
The vast majority of CTOs I speak to are open to carefully increasing the autonomy of their agentic engineering, but they just don’t know exactly how to get there from where they are now. This is a brief summary of the good practices that I’m helping my advisory clients to use to safely accelerate their agentic engineering efforts.
Apologies in advance that this is a very abbreviated summary. I will be dropping articles over the next few weeks giving much more detail about how to actually do each of these things. But I wanted to ship an article, not a book :)
Assumptions
This article assumes you have at least 50 engineers. For smaller teams, adoption is not trivial, but it’s much easier.
It also assumes that you have executive alignment and support from security, audit, governance, compliance, risk and legal. If they’re not going to let you use the tools, adoption isn’t your biggest problem.
It also assumes that at least some of your engineers are starting to use skills to accelerate development.
And it assumes that you are not one of the very small number of companies with a mature scaled agentic SDLC already widely adopted.
Finally, it assumes that you have some interest in at least experimenting with accelerating and automating your software or wider product development life cycle.
Four steps for safely scaling agentic engineering
Here’s a four-step process for getting quick wins, strengthening the underlying infrastructure, starting to industrialize good patterns, and then scaling the factory. They are designed to make it easy to start and to allow you to bring the majority of your team along on the journey by supporting them in taking small incremental steps.
This piece is mostly focused on the technical infrastructure. I’ll drop a piece next week summarizing the cultural and change management pieces that make this work in a larger org.
1. Start with Skills
The first question everyone has is how to balance governance with experimentation. For skills, day one, you have to optimize for speed. It reduces the pressure for shadow AI, allows people to experiment with things and quickly shows you who your explorers or super-users are going to be. They are the people you will need to elevate and unblock to show the rest of the org what good can look like.
Process:
Start by allowing people to write their own skills - without forcing them to do so. At this stage, you are way too early in the game for AI usage mandates.
As soon as possible, build a shared skills registry. The only requirement to get into it should be a quick automated review for malicious or problematic instructions.
As specific skills start to become popular (3-4 users beyond the original author), provide DevEx or platform team support to improve their rigor and capabilities - uplifting (not excluding) the original author.
Over time, start to standardize on skills and provide a taxonomy so it’s easy for agents and people to find the right skills to use to do a given job.
Start to improve the rigor of skills - Supervisor patterns allow multi-step processes with deterministic and adversarial quality gates. Advisor patterns reduce token costs while keeping the quality similar. Intermediate artifacts reduce the intelligence required to deliver consistently high-quality outcomes.
By starting with skills, you can meet people where they are, allow them to start to encode tacit knowledge for the company, and incrementally introduce good practices - while you’re simultaneously hardening your infrastructure.
2. Underpin this with infrastructure
Right now, most companies are one poorly conceived prompt away from a critical security event. As soon as you have stood up the team to build the skills registry, you should pull together a tiger team to build out your agentic infrastructure.
Don’t worry about picking exactly the right vendors. Find ones that are good enough and ensure that you start every trial with a planned off-ramp. Things are moving too fast to pick winners, so it’s unlikely you will stay with most of your tools for more than a couple of months. Plan accordingly.
Key elements
Model router - you need the ability to route the appropriate work to the appropriate level of model. And also over time to failover across labs during outages. And to reduce costs by leveraging open-weight offerings.
Model sandboxes - you do not want agents running around on laptops that are probably logged in to Workday and that have access to proprietary IP - you need to get all of your agents running in sandboxes in the cloud.
Transcript capture - this has to be handled carefully, but everything you need to learn about both your agents and your teams is being captured in the transcripts. You need to find a safe way of persisting and accessing that information in a way that is supportive of the people whose work is being captured.
Start to build and run evals - as you start to classify types of model work, you should build evals, to avoid drift, to support auto-tuning of prompts for different model families and to experiment with cheaper models. Doing the same work with better-distilled context, this increases resilience and can materially reduce operating costs.
3. Industrialize proven flows
Start this early with a few pioneers, and scale it only once the first two steps are showing real progress.
Once people are used to the idea of skills leveraging a supervisor pattern to drive multi-step repeatable processes where code is generated, tested and adversarially reviewed, the obvious next step is to give them a deterministic multi-step orchestrator to more cheaply and reliably do the same job. Congratulations, you are now in the world of software factories!
There are lots of vendors entering this space, but most of the teams I see succeed are still building their own infrastructure, but using good practices that will make it easy to swap out for open-source or commercial offerings once they are sufficiently mature and capable.
Careful curation of skills over time will allow you to standardize on patterns for capturing specifications and verifications for units of work. Eventually those will become the inputs and outputs for your software factory allowing your engineering org and your citizen coders to ship anything from a quick prototype with only a very lightweight review to robustly architected and specified systems that can be trusted in production.
4. Scaling your factories
The factories are only the beginning, not the end. Vendors have not yet reliably solved the multi-player problems. Here are some of the active research areas that I find myself working on for my factory and my clients:
User experience - how can an engineer be given exactly the right context to provide training data and review code in the loop while confidence in a generation pipeline is still low? Equally how can you show progress and operational telemetry to ICs for the flows that they are on the loop and just keeping an eye on without blocking with a manual code review?
Interactive model output - in interactive mode, models will often emit a wall of text - burning a lot of engineer time and context to parse. Personally, I use a classifier with a custom UX that identifies the kind of model response (usually some combination of “done”, “problem” or “input required”). I then have a JavaScript UX that allows for progressive revealing of context if the summary is not enough - which nicely mitigates the problem that agents almost always give you too little or too much information when they report back.
Planning and capacity brokers - if you have tens or hundreds of engineers, each running tens or hundreds of agents, pretty quickly you are going to run into predictable constraints. Merge queues become hard to manage. CI becomes contended, and CD starts to require release trains to keep up with the load. I’m working now on a Capacity Broker, where approved features are planned to identify the likely files touched, and a PM agent is responsible for scheduling and approving builds to minimize merge conflicts. And CI and CD runs required across the fleet.
Looping the loops - you will also find pretty soon that there are established patterns for using agents to improve the infrastructure you’ve built for running agents! Think very hard about balancing complexity with performance and documenting rubrics, evaluations and identifying fundamental metrics that should not be degraded.
The best time to plant a tree
The problem with agentic engineering is it really takes a while to start to get material acceleration. We should probably all have started around June of 2025, but here we are.
There are, however, two pieces of good news. The first is that you can start with skills. It’s fine to be where you are right now. Add sandboxes, shared infrastructure, a shared skills library, and more of your team will start to get value out of your agents - with less friction. You don’t have to build the factory on day one.
The second is that you’re not going to run out of loops to build. For as far ahead as I can currently see, there will always be another interesting technical problem that will be uncovered as soon as you solve the one that you’re stuck on today!