Transfer follows similarity
Practice must look like the job.
Design response: build every exercise from real tasks.
Research notes · AI awareness training problems
Why AI awareness training rarely changes work
My working notes on a problem I keep seeing: organizations train everyone on AI, and the work stays the same. Here is why I think that happens, what the learning research says, and the role-based ideas I have developed in response: Role Signature, BELGIC, and DRILL.
Organizations hand employees hours of videos about AI features and rules. Completion reports look excellent. The work does not change.
Training ends. Then the real questions start.
- Day 1Everyone completes the courseA talk on generative AI, a tool tour, a slide of rules, and a certificate. The completion report looks excellent.
- Week 1Back at the desk, four questionsWhich of my tasks can AI do well? What data may I put in? How do I check what comes back? Who is accountable if it is wrong?
- Month 1Uncertainty turns into habitsWithout answers, people make their own rules. Some guess. Some copy a colleague. Few have practiced on their real work.
- Month 3The workforce splits in twoOne group stops using approved tools, to be safe. The other uses unapproved tools quietly, because the work still has to get done.
Risk you cannot see: sensitive data in the wrong tool, unchecked output.
Productivity you are not getting: idle licenses, and no evidence either way.
AI helps on one task and hurts on the next
In a field experiment with 758 consultants, AI raised quality on tasks inside its capability, and lowered accuracy on a task just outside it. Researchers call this the jagged frontier.
+40%higher-quality work inside the frontier
−19 ptsin correct answers just outside it
The implication: if AI's usefulness changes task by task, training must be built task by task. That means role by role.
Dell'Acqua et al., 2023, Harvard Business School Working Paper 24-013.
Why awareness training does not change work
1Generic contentOne deck for everyone, examples about meal plans
Why it fails: skills transfer only when practice resembles the job.
A better design: Work-Twin scenarios built from each role's real tasks.
2Passive formatLecture and demo; people watch an expert prompt
Why it fails: watching builds familiarity, not skill.
A better design: DRILL: produce first, then critique and improve.
3Rules without practiceA slide of do's and don'ts from the AI policy
Why it fails: people recite a rule, then misapply it in a gray zone.
A better design: policy-to-practice drills: Go, Sanitize, or Stop.
4No verification skill"Always check the output," with no method
Why it fails: people over-trust AI outside its competence.
A better design: trap tasks with planted errors, and a measured catch rate.
5Tool tour, not judgmentFeature walkthroughs of one product
Why it fails: features change monthly; judgment lasts.
A better design: tool-agnostic technique plus the organization's approved tools.
6One and doneA single session, then nothing
Why it fails: unpracticed skills fade within weeks.
A better design: 30, 60, and 90-day micro-drills and launch check-ins.
7Measured by satisfactionCompletion rates and smile sheets
Why it fails: liking a session says little about what people can do.
A better design: Proof: before-and-after work samples.
Six laws of skills that stick
Doing beats hearing
People learn by producing, not watching.
Design response: start every session with a hands-on attempt.
Judgment needs exposure to error
People learn to catch mistakes by meeting them safely.
Design response: plant errors and score the catch rate.
Spacing beats repetition
Recalling a skill later makes it last.
Design response: micro-drills at 30, 60, and 90 days.
Intentions need if-then plans
Goals become action when tied to a cue.
Design response: end every cycle with a dated launch.
The environment decides transfer
Manager support and clear rules predict use.
Design response: leaders, IT, HR, and Legal co-own scenarios.
Six stages, four integrated frameworks
Define success before training starts
One idea I keep returning to: agree in writing, before any training, what success means for each group in measurable terms. For example, a target trap catch rate, the share of people who finish a real task with AI within 30 days, and the improvement in work-sample quality. Set the targets from a baseline, rather than promising them in advance.
What every participant should leave with
A launched work productA real memo, summary, or analysis, finished with AI under the organization's own rules.
A personal role playbookThe prompts, checks, and principles they wrote themselves during the lab.
A measured before and afterTheir own score on time, quality, and safety, on a task from their actual job.
And leaders should receive evidence: what changed, for whom, and what to do next.
Each framework answers one question
Role SignatureWho needs what, and where can AI safely help in this role?Role profile, task inventory, Delegation Grid, Work-Twin scenariosrolesignature.com
BELGICWhich AI capabilities matter most for this role?Weighted capability emphasis per role group, and learning objectivesbelgicframework.com
DRILLHow does a capability become a habit inside a session?Time-boxed practice cycles, principles written by learners, launched workdrillmethod.io
ProofDid it work, for whom, and how do we know?Before-and-after work samples, trap catch rate, launch ledger, evidence reportWrite-up coming
Role Signature: every role profiled on five lenses
WWorkflows
How much of the job is recurring documents, summaries, and analysis?
OObligations
Which laws, records rules, standards, and ethics codes apply?
RRisk of error
What happens if a wrong output slips through?
KKnowledge sensitivity
How often does the role touch personal, privileged, or nonpublic data?
SSystems
Which AI tools are approved, and what can they reach?
Delegation Grid: a rule for every task
DelegateAI drafts; human reviewsCo-pilotAI assists; human verifies every claimHuman-onlyAI may organize; judgment stays human
GreenPublic or non-sensitiveGoApproved tool; spot-checkGo with checksVerify sources and figuresAssist onlyStructure, not the decision
AmberInternal; approved tools onlyGo, sanitizedStrip names and identifiersSanitize and verifyTwo-person review if publishedAssist, sanitizedNo sensitive detail in the prompt
RedProhibited input per policyStopNever enter into AIStopNever enter into AIStopNever enter into AI
BELGIC: six AI capabilities, weighted for each role
BBuildPlan, structure, automate
Turns a recurring task into a reusable prompt or workflow.
EExploreResearch, trendspot, forecast
Gathers with AI, then confirms every source.
LLearnInteract, study, synthesize
Writes clear, requirement-rich prompts; grounded synthesis.
GGrowAssess, optimize, scale
Judges quality against criteria and shares what works.
IInnovateSolve, create, iterate
Generates options and drafts, then improves them.
CControlSecure, validate, audit
Decides what goes in, verifies what comes out, and leaves a record. Never below core for any role.
DRILL: act first, then learn why it worked
- DDo
A short, timed attempt on a real task, before any instruction
7 min - RReflect
Accurate? Relevant? Right tone? Safe?
8 min - IImprove
Change one lever at a time; predict, then test
12 min - LLearn
Write the principle in your own words
5 min - LLaunch
Commit to a real task and a date
3 min
The Control variant: stop before you paste. For safety drills the order changes. People inspect the input first, decide its data class, sanitize or stop, and only then use AI. Repeated with different documents, it becomes a reflex.
Proof: evidence instead of satisfaction scores
- The proof pair. A timed task from each person's job at the start, a matched task at the end, both scored blind.
- Trap catch rate. Planted fabricated sources, wrong figures, and hidden sensitive data. Count what is caught.
- Launch ledger. Every real task people commit to finishing with AI, checked again at 30 days.
- Evidence report. Results for leadership within two weeks, with a 90-day follow-up.
Sample participant result (illustrative)
Budget analyst: work-paper summary
| Before | After | |
|---|---|---|
| Minutes to a usable summary | 14 | 9 |
| Quality score (0 to 4) | 2 | 4 |
| Sensitive data removed before entry | No | Yes |
| Numerical error flagged | No | Yes |
Each participant sees their own result before they leave. Individual results are shown only to that person.
Awareness training compared with a role-based approach
| Awareness training | Tool training | Technical bootcamps | Role-based approach | |
|---|---|---|---|---|
| Built from | Generic examples | Product features | Technical curriculum | The organization's roles, tasks, and policy |
| Learner activity | Watch and listen | Follow along | Code | Produce, critique, improve, launch |
| Customization | Logo and examples | None | By skill level | Role profile, Delegation Grid, Work-Twins |
| Risk and policy | A slide of rules | Product settings | Varies | Practiced decisions with trap tasks |
| Audience | Everyone, the same way | Tool users | Developers | Every role, differently |
| Proof of impact | Completion, satisfaction | Completion | Projects | Before-and-after work samples, catch rate |
| After the session | Nothing | Help articles | Alumni network | 30/60/90 reinforcement and 90-day review |
Each category has a place. Awareness training builds vocabulary, tool training builds familiarity with one product, and bootcamps build engineers. My argument is narrower: none of them is designed to build and show role-specific capability across a whole workforce.
About these notes. Role Signature, BELGIC, and DRILL are ideas I have developed through my own research and writing. They draw on established work in training transfer, active learning, spaced practice, and human–AI collaboration. I share them here as a researcher and practitioner, for discussion. They are not a product or a service offered on this site.