Most businesses record their calls and read almost none of them. The audio is retained for compliance and is useless for anything else, because nothing in it is searchable.

Making it searchable is a pipeline problem, and each stage has a failure mode worth knowing before you start.

Transcription: expect worse audio than the benchmarks

Published accuracy figures are measured on clean, wideband recordings. Telephony audio is narrowband, compressed, and full of crosstalk, hold music and interruptions. Accuracy on your actual recordings is the only number that means anything, so measure it during discovery on a real sample.

Whisper handles this well; WhisperX adds the two things that make output usable — accurate word-level timestamps and speaker diarization.

Diarization is the point

An undifferentiated wall of text is nearly worthless for a two-party conversation. Knowing that the customer said a thing rather than the agent changes its meaning entirely — a complaint from a customer and an agent quoting the complaint back are opposite signals.

[00:04:12] agent    — that model is on a four week lead time
[00:04:19] customer — four weeks is too long, what else fits
[00:04:26] agent    — the 50mm is in stock today

In one production system, an average call yields around 412 separately transcribed and attributed segments. That granularity is what the analysis layer works on.

Why call-level sentiment is misleading

A single sentiment score for a whole call averages away everything interesting. A call that starts angry and ends resolved is a success; one that starts pleasant and ends with a cancellation is a failure. Both average to neutral.

Per-segment scoring, with trajectory across the call, actually answers the question. The direction of change is the signal.

  • Score sentiment per utterance, then look at the trend
  • Extract intent per segment — customers raise several in one call
  • Flag specific moments: competitor mentioned, discount promised, escalation requested
  • Summarise from the segments rather than the raw transcript

Queue everything

Transcription and analysis are far too slow to run inline. Completed calls are queued, and the pipeline processes them as capacity allows — which also means a spike in call volume delays insight rather than dropping it.

call ended → queue(transcribe)
            → queue(diarize)
            → queue(analyse per segment)
            → queue(summarise)
            → write structured record to CRM

Each stage is separately retryable. A failure in summarisation should not force a re-transcription of an hour of audio.

Why self-hosted, specifically here

Call recordings are among the most sensitive data a business holds: names, addresses, payment discussions, complaints. Most of the clients we do this work for had already rejected hosted transcription for exactly that reason.

Ollama running a local model handles the analysis and summarisation stages perfectly adequately. Combined with WhisperX, the entire pipeline sits inside the client’s own network, which is usually the difference between the project happening and not.