Ask a physician what they do after clinic and the answer is documentation. Notes written in the evening, notes written at weekends, notes written during the visit while a patient watches the top of their doctor's head.
An AI medical scribe listens to the encounter and drafts the note from it. The clinician reviews, edits, signs. The visit stays a conversation, and the documentation is largely done before the patient has left the room.
The pipeline is not complicated. The accuracy requirements are brutal, and that is what most of this article is about.
What documentation is actually costing
Four separate problems, worth naming because they need different fixes:
- Time, the obvious one. Hours per week that are neither clinical nor paid.
- Attention. A clinician typing is a clinician not observing. Patients read it accurately as divided attention, and satisfaction reflects it.
- Burnout. Documentation burden is one of the most consistently cited drivers, and it is cumulative in a way that a single bad day is not.
- Data quality. Notes written from memory at 9pm are thinner and less consistent than notes written from the encounter. Everything downstream — coding, quality reporting, continuity of care — inherits that thinness.
Note that the last one is a business problem, not a wellbeing problem. Under-documented encounters under-code, and under-coded encounters under-bill.
The pipeline
1. Capture, with consent. Ambient audio from the room. Consent comes first and is covered below, because getting it wrong is a legal problem rather than a product problem.
2. Transcribe with speaker separation. Diarisation matters more than in general transcription: a symptom the patient reports and a finding the clinician states are different clinical objects. If the transcript cannot tell you who said something, the note built on it will attribute wrongly.
3. Understand the clinical content. Extracting what is clinically meaningful from a conversation that includes scheduling talk, small talk, interruptions, and the patient's dog. This is the hard stage.
4. Generate the structured note. SOAP or your template, populated — subjective from the patient's account, objective from examination findings stated aloud, assessment and plan from the clinician's reasoning. Structured fields, not a wall of prose, so the data is usable downstream.
5. Push to the EHR. Via FHIR where available — DocumentReference for the note, Encounter for context, Condition and Observation for discrete data. A note that lands in a separate app has not saved anybody anything.
6. Clinician reviews and signs. Not a step to be optimised away. See below.
Where clinical accuracy breaks
Negation, first and worst. "No chest pain, no shortness of breath" and "chest pain, shortness of breath" differ by one word and invert the clinical meaning. Negation and uncertainty handling — "denies", "ruled out", "cannot exclude", "unlikely" — is the single most important correctness property of a clinical NLP system, and generic transcription-plus-summarisation gets it wrong in ways that read perfectly fluently.
Drug names and dosages. Sound-alike medications and mis-transcribed numbers are direct patient safety issues. Constrain extracted medications against a real drug vocabulary and flag anything that does not match rather than accepting a plausible-sounding string. Never let a dosage through on transcription confidence alone.
Attribution. A symptom the patient denies, recorded as a finding. A family history item recorded as the patient's own. These come from diarisation failures and they are hard to spot in a fluent note.
Confident invention. The defining failure mode of generative models in this context. Asked for a complete note, a model will smooth over gaps with plausible clinical content that was never said. The mitigation is architectural: the note must be grounded in the transcript, every generated statement traceable to the utterance that produced it, and a gap left visible as a gap. A note that says "not documented" is useful. A note that invents a normal finding is dangerous.
Ambient conditions. Multiple speakers, overlap, background noise, accents, code-switching. Test on recordings from your actual clinic.
Consent is not a checkbox
You are recording a medical conversation. Two obligations, and they are separate.
Recording consent. US wiring and eavesdropping laws vary by state — some require all-party consent, meaning the patient must agree, not merely be informed. Know your jurisdiction, tell the patient plainly what is being recorded and why, capture that they agreed, and make declining frictionless. A patient who says no gets a visit documented the old way, without discussion.
HIPAA. The audio is PHI, the transcript is PHI, and so is anything derived from them. Encryption in transit and at rest, BAAs with every processor including speech and model vendors, audit logging, and a retention policy with actual deletion. Decide early whether you keep audio after the note is signed — the safest answer is usually no.
Review and sign, always
The clinician is accountable for the note, so the clinician reads it. Design for that rather than around it:
- Draft presented for editing, never auto-filed.
- Uncertain extractions visibly flagged, not silently smoothed.
- The transcript accessible beside the note so a questionable line can be checked in one click.
- Every edit logged — that is your quality signal and your audit trail.
Measure time-to-signed-note and edit volume, not "accuracy". If clinicians are rewriting half of every draft, the tool is generating work while appearing to save it, and only edit volume will tell you.
FAQ
Does it work for all specialties? Templates and vocabulary are specialty-specific. Primary care and general medicine are well served; highly specialised or procedural documentation needs tuning. Validate on your own specialty mix.
Can it code the encounter? It can surface the codes the documentation supports, for a coder or clinician to confirm. Auto-submitting codes from a generated note is how a model error becomes a compliance event.
Do patients object? Most accept it when told the reason plainly — the clinician looks at them instead of a screen. Objections tend to come from being recorded without a clear explanation.
How is this different from document extraction? Input. This turns conversation into a record; extraction turns files into records. Same destination, different failure modes.
Cloud or on-premise? Cloud with a BAA is the usual answer. On-premise reduces the compliance surface and raises the cost and latency — a real trade-off, not an obvious one.
Where we fit
We build documentation systems with the correctness properties first: diarisation, negation and uncertainty handling, vocabulary-constrained medication extraction, grounded generation with traceability, and FHIR/HL7 write-back into the EHR you already run — the same interoperability bench behind our hospital platform.
If your clinicians are documenting after hours, book a call. Bring five anonymised encounter recordings and we will show you exactly where a generic scribe would get them wrong.
