Why AI courses for engineers don't work — and what does
Most AI training teaches prompts and tools. But the bottleneck is elsewhere: when generation is nearly free, the skill that matters is verification.
The usual story goes like this. A company buys licences for an AI assistant, runs a two-day course, and participants rate it 4.7 out of 5. A month later, a few people use the tool daily, some use it as clever autocomplete, and most are back to working the way they did before. The training report still looks like a success: everyone attended, everyone liked it.
I design learning programmes for engineering teams, and lately most of them are about AI. My conclusion is simple: a course about a tool does not change how people work. Here is why, and what I do instead.
Why the classic course fails
It teaches the tool, not the work
Most programmes are built around the product: here is the chat, here is autocomplete, here is agent mode, here are ten useful prompts. But tools change every quarter. What the course presented as “the right way” is outdated six months later; the course is not.
The skill that lasts is not “how to use a particular assistant”. It is “how to split a task so part of it can be delegated, and how to make sure the delegated part was done right”. Tool courses barely teach that.
Prompting is not the bottleneck. Verification is
Generating code, configuration, tests and documentation has become almost free. Verifying them has not got cheaper. Someone still has to read the diff, understand whether it does what it should, and whether it breaks something next to it.
There is an uncomfortable detail here: people are bad at judging how much AI actually helps them. In METR’s 2025 study, experienced open-source developers took 19% longer on tasks when AI was allowed. Before the experiment they expected a 24% speed-up; afterwards they believed they had been 20% faster. METR’s later data on newer tools shows a modest speed-up, but the authors themselves say honest measurement is getting harder and developers’ self-reports are unreliable.
The lesson for training: if an engineer cannot verify output quickly and critically, faster generation just moves the work from writing to review — and sometimes to production incidents.
Training is detached from real work
The course runs on a demo project: a todo app, an invented microservice, a clean repository. Then the engineer goes back to their monolith, internal libraries, legacy pipelines and security constraints. Transferring a skill from a clean example to that context is a separate piece of work, and usually nobody does it.
In learning design this is a well-known problem: knowledge transfers poorly between different contexts unless transfer itself is practised. AI training is no exception.
It measures satisfaction, not behaviour
Almost every AI programme reports on level 1 of the Kirkpatrick model: did people like it. Sometimes level 2: did they pass a quiz. Almost nobody looks at level 3 — did behaviour on the job change. That is the level the business actually cares about.
Without before-and-after measurements, you cannot tell a real effect from the enthusiasm of the first few weeks.
It trains individuals, but the team has to change
Even when one engineer learns to work well with AI, they return to a team whose rules have not changed. How do you review code an agent wrote? Do you label such changes? Who is accountable when a generated test does not actually test anything? Until the team agrees on this, individual skill hits a ceiling.
What works instead
I stopped building programmes around “how to use the tool” and started building them around a different question: how do you work when generation is cheap and verification is not?
Learning on your own backlog. Every session uses a real task from the participant’s current sprint. Not a demo project — a real ticket in a real repository, with all its constraints. Transfer happens immediately instead of “some time later”.
Practising verification, not generation. The most useful exercise I know: an engineer gets AI-generated output containing errors — from obvious to subtle — and has to find them. It calibrates trust: people quickly see where the model consistently gets their stack wrong and where it can be trusted more.
Decomposition and context as an engineering skill. A well-written task for an agent is essentially a small specification: goal, constraints, definition of done, what not to touch. Engineers who can write like that get better results regardless of the tool — and the skill does not expire with the next product version.
Knowing when not to use AI. A dedicated block on tasks where AI slows you down: unfamiliar code you cannot verify; security-critical changes; situations where explaining the context takes longer than doing the work.
Team agreements. At the end of the programme the team writes down its rules: how to review AI-generated code, what can be delegated to agents and what cannot, how to label such changes. It is a one-page document, but it is what turns an individual skill into a team one.
Measuring before and after. Not satisfaction surveys, but behavioural metrics the team already collects: time from starting a task to merge, how often changes bounce back from review, the share of changes that needed fixes after release. Collect self-reports too — as a separate signal, keeping in mind that they are systematically optimistic.
A minimal programme design
This is the shape I usually start from. It is built for one team, six weeks, alongside normal work.
| Week | Focus | Output |
|---|---|---|
| 0 | Baseline: team metrics, short survey, pick real tasks | Starting point |
| 1 | Decomposition and context: how to brief AI | Task-briefing template for your stack |
| 2 | Verification: error-spotting drills on AI output | List of the model’s typical mistakes in your code |
| 3–4 | Real tickets, weekly 90-minute clinic | Backlog items closed, cases reviewed |
| 5 | When not to use AI; risks and security | Team’s delegation boundaries |
| 6 | Team agreements, re-measure, retro | Team rules and before/after comparison |
The schedule matters less than the principles: real tasks instead of demos, verification instead of generation, the team instead of the individual, measurements instead of surveys.
What I’m still unsure about
A few questions I do not have good answers to yet:
- How to measure productivity honestly. Flow metrics show change, but too many other things affect them. Even METR is redesigning its methodology.
- Should juniors learn without AI first? You can only verify what you understand. But learning in isolation from the tools you will work with is odd too.
- How fast training loses value. If the principles are stable and the tools change every quarter, how much time should go to each?
If you have run or attended AI training in an engineering team, I would like to hear what worked for you and what did not.
This is the first issue of “AI Beyond the Demo” — field notes on bringing AI into real engineering work. Next issue is a lab: one DevOps task, several AI tools, real measurements.
Sources: METR: early-2025 AI and experienced open-source developer productivity · METR: uplift update, February 2026