One person became four

A two-person call that showed up as four speakers, and what fixing it taught me about not trusting the machine’s confidence.

Here is a failure that looks absurd until you see why it happens. I recorded a call with one other person on it. The finished transcript showed four speakers.

The step that assigns each slice of audio to a speaker — diarization — does not know in advance how many people are in the room. Left unbounded, it over-splits. One person’s voice, across changing mic distance, a cough, a bad moment on the network, gets broken into several “speakers.” So a clean two-person conversation fragments into Speaker 1 through Speaker 4, all of them really the same two people.

The first instinct is to merge anything that sounds similar. That helps, but it is not enough, because the fragments that survive are exactly the ones that do not sound similar — short, low-quality junk. Note that the machine reports all four with the same flat confidence. It does not know it is wrong. That is the real lesson, and it runs through the whole product: a record you cannot trust is worse than no record, because the machine’s confidence tells you nothing about whether it is right.

So the fix is not one clever line, rather a set of conservative rules — merge same-voice fragments within a meeting, fold away the small low-coherence junk, and where I actually know the number of people, enforce it.

None of this is exciting. But “who said what” is the entire value of the transcript, and getting it to stop inventing people is the difference between notes I can act on and notes I have to re-check line by line.

More soon.