Use AI · Chapter 7 of 13
Did AI save time, or just make work feel faster?
Use a real experiment and a later survey to build your own four-job scorecard: finished work, checking time, failed attempts and cost.
About 8 minutes · school-level background
Edition 2026-10-28.1 · Checked 2026-10-28
What you will understand
Measure the whole useful result, including checking and repair, and recognise when a study no longer answers your current question.
In this lesson
- The stopwatch starts before the impressive answer
- The real case, with its date attached
- A fast draft can move the work elsewhere
- Keep three clocks, not one vague impression
- The missing tasks can change the story
- Copy a four-job scorecard before opening the assistant
- A cheaper attempt can still cost more per accepted result
- Choose the next change from the expensive row
- Try a test small enough to finish
Build the idea
The stopwatch starts before the impressive answer
Imagine watching an assistant produce a page of code while you are still deciding what to type. It feels like a race you could never win. But the job is not finished when text appears. Someone must check that the code does the right thing, fits the project and survives a change tomorrow. The interesting question is how long it takes to reach that usable result. This distinction matters even if you never write code: the same question applies to a travel comparison, an event announcement or a spreadsheet.
Before you see the evidence, make a prediction. If an assistant writes a first draft almost instantly, which part of your own last task would still take time? Finding reliable inputs, deciding what the answer means and checking a surprising statement are all candidates. Naming that part gives you something more useful than a general opinion about whether AI is good. It gives you a specific place to look for a gain or a new bottleneck.
Without AI: 50 min making + 30 min checking = 80 min
With AI: 20 min making + 70 min checking = 90 min
Illustrative classroom example, not a measured result.
What the source reports
The real case, with its date attached
METR’s 10 July 2025 experiment involved 16 experienced developers and 246 real tasks, randomly assigned to allow or disallow AI. Work took 19% longer with AI allowed in that setting. Its 24 February 2026 follow-up described selection effects that made newer estimates unreliable. Keep those dates and populations attached to the findings.
A separate METR survey, published on 11 May 2026, analysed 349 technical workers’ responses. Median self-reported value gains ranged from 1.4 to 2 times across different questions. These are perceptions from a convenience sample, not independently measured output gains. The contrast gives you a useful question: what happened to your own checked deliverables?
Original report: METR: early-2025 developer productivity experiment · METR: changing the productivity experiment design · METR: self-reported technical-worker productivity survey
Worked teaching example
A fast draft can move the work elsewhere
Here is a fictional classroom example, not a METR measurement. Without AI, a student spends 50 minutes making a comparison table and 30 minutes checking it. The total is 80 minutes. With AI, preparing the request and producing the table takes 20 minutes, but checking sources and repairing errors takes 70 minutes. The new total is 90 minutes. The visible drafting stage is 30 minutes faster, while the completed job takes 10 minutes longer. Divide 10 by the original 80: the total increase is 12.5%.
Now change one part of the example. Suppose the student supplies cleaner source extracts and a precise table format. Drafting still takes 20 minutes, but checking falls to 25 minutes. The total is 45 minutes, saving 35 out of 80 minutes, or 43.75%. These invented numbers show why a workflow change can matter as much as a model change. They do not predict your saving. In both versions, the finish line must be the same: a table whose entries meet the agreed checks.
Build the idea
Keep three clocks, not one vague impression
Elapsed time is the time from starting to finishing. Active human time is the time you personally spend doing or supervising the work. Machine waiting time is the time the tool runs without needing your attention. They answer different questions. An overnight run might take hours of elapsed time but little active attention. A stream of quick answers might demand constant checking. If you care about having an answer before a meeting, elapsed time matters. If you care about freeing an afternoon, active attention matters too.
For a small personal trial, write down preparation, drafting, checking and repairs. Also record whether the output passed your checklist. A ten-minute result that is unusable is not a faster version of a usable result. Avoid silently dropping abandoned attempts. Put them in a separate unfinished row, with time already spent. That row can reveal the expensive failure mode you would otherwise forget when remembering the one spectacular success.
Worked teaching example
The missing tasks can change the story
Consider another invented experiment. You have 20 jobs, and AI is particularly helpful on 10 of them. If you refuse to put those 10 into a no-AI comparison, the remaining sample describes a different mixture of work. Even careful timing on the remaining jobs will not restore the missing comparison. This is selection bias: the process that chooses observations changes what they represent. It can happen without anyone lying about a single stopwatch reading.
A second trap is doing the identical task twice. After doing it once, you already know the sources and likely answer. The second attempt has an advantage whichever tool it uses. For a practice comparison, use similar but different inputs, set the checks in advance and vary the order. You will still have a small, imperfect personal trial. That is enough to improve a local decision, provided you do not turn it into a claim about every student, developer or model.
Use the idea
Copy a four-job scorecard before opening the assistant
Imagine offering a small service that checks community event listings. Before charging anyone, test the workflow on fictional listings. Prepare four different jobs, each containing five fields: event name, date, opening time, location and booking rule. Put the correct values in a separate answer sheet. Add one deliberate conflict to every job, such as two opening times, so a good result must mark the conflict rather than invent a resolution. These inputs are a practice kit, not real customer evidence.
For each job, make columns for method, preparation minutes, drafting minutes, checking and repair minutes, usage charge, and accepted or unfinished. Acceptance means that all supported fields match the supplied sources, each has a source reference, and the conflict is explicitly unresolved. Use the same checklist for both methods. A good-looking answer that guesses the opening time remains unfinished until corrected. Save the output that failed, as well as the eventual correction.
Use a coin to choose which method handles the first job; alternate methods afterwards. Try two jobs manually and two with your assistant, without repeating an identical input. This reduces an obvious practice advantage, but four jobs cannot balance every difficulty difference or establish a general effect. Keep the individual rows visible. You are finding where this workflow needs work before making a customer promise, not conducting a definitive model competition.
Worked teaching example
A cheaper attempt can still cost more per accepted result
Here is a fully invented ledger. Manual job M1 takes 3 minutes of preparation, 12 of drafting and 5 of checking: 20 minutes, accepted. Manual M2 takes 3, 14 and 7 minutes: 24 minutes, accepted. Assisted A1 takes 4, 2 and 6 minutes: 12 minutes, accepted, with £0.10 of usage. Assisted A2 takes 4, 2 and 18 minutes: 24 minutes, with £0.10 of usage, but its conflicting time is still unresolved when the time budget ends.
Predict the winner before calculating. The manual method used 44 active minutes and delivered two accepted results. The assisted method used 36 active minutes but delivered one. At an illustrative £12 per hour, each active minute is valued at £0.20. The manual effort is £8.80, or £4.40 per accepted result. The assisted effort plus usage is £7.40, or £7.40 per accepted result in this tiny batch. Lower total time did not produce a lower cost per completed job.
Keep A2 in the ledger. If a later repair takes eight more active minutes and requires no extra charge, the assisted batch reaches £9.00 for two accepted results, or £4.50 each. That is a different observation with a later stopping point. Preserve the original unfinished state and append the repair. Time values are planning assumptions, not wages actually paid; use actual charges and your chosen time value in a real project. These four rows are not measurements from METR or WRS users.
Use the idea
Choose the next change from the expensive row
The slow assisted row points to conflict checking. For the next small trial, change one thing: ask the assistant for a discrepancy table before it drafts the final notice. Keep the same acceptance rule and use fresh comparable inputs. Record whether it finds the conflict, how much checking remains and whether anything is still unfinished. A different model is another possible change, but changing the model, inputs and checklist together makes it hard to know what helped.
You can practise without extra software spend if your existing chat access permits it; a subscription still has a real cost and usage limits. The £0.10 charges above are invented calculation inputs, not current provider prices. Set a 45-minute personal practice cap and keep partial work when it expires. If you later offer the service, show a real recipient the checked sample and ask whether it solves their problem. Their decision to use it is stronger evidence of usefulness than the number of drafts your assistant generated.
Use the idea
Try a test small enough to finish
Choose a harmless recurring job: comparing public event listings, organising invented expenses or checking a short explanation against supplied sources. Define what a correct result must contain before opening the assistant. Keep the input sizes similar. Stop when a preset time budget runs out, preserving the incomplete result. Then ask where the minutes went. If most time went into source checking, the next useful change may be better inputs and source links rather than a more elaborate prompt.
Your final artifact can be a one-page comparison with the inputs, checklist, timing breakdown and remaining errors. Share only material you have permission to share. A useful conclusion sounds like this: for these inputs, this workflow reduced my checking burden, or it introduced an error I could not confidently repair. That is an actionable result. Repeating the exercise after a substantial tool change can show whether your earlier decision still holds.
Words you can now use
- Elapsed time
- Time from the start of a task to its finish, including waiting.
- Selection bias
- A distortion caused by which tasks or participants enter the observations.
- Baseline
- The comparison workflow used to judge a proposed improvement.
Check your understanding
The assisted batch takes 36 minutes and the manual batch 44. Why is that insufficient to declare the assisted service cheaper, and what happens if A2 is later repaired?
Show the explanation
The manual batch has two accepted results; the assisted batch initially has one. At the example time value plus usage, costs per accepted result are £4.40 and £7.40. Eight more minutes repair A2, bringing the assisted total to £9.00 for two results, or £4.50 each. Keep the first stopping point and append the repair. These are invented inputs; four unmatched jobs cannot establish a general productivity effect.
Read the original sources
- METR: early-2025 developer productivity experimentResearch organisation experiment report · Published 2025-07-10 · Checked 2026-10-28
- METR: changing the productivity experiment designResearch organisation follow-up · Published 2026-02-24 · Checked 2026-10-28
- METR: self-reported technical-worker productivity surveyResearch organisation survey report · Published 2026-05-11 · Checked 2026-10-28
Try it with your own assistant
Select and copy this bounded exercise. A normal chat assistant is enough to start.
Make four different fictional five-field event jobs, each with one unresolved source conflict. Set the same source-and-accuracy checklist before starting. Compare two manual and two assisted attempts, keeping preparation, drafting, checking, charges and unfinished rows. Recalculate the worked ledger, then change one expensive step on fresh inputs. Use a 45-minute practice cap and no real recipients or paid service activation.
Keep the task brief, supplied inputs, checked output and unresolved questions together. Using this exercise does not submit anything to WRS.