No speech recogniser is the best at everything. One engine handles Polish digits better, another copes better with Austrian German on a noisy mobile line, a third is faster to return a result. So we do not standardise on a vendor. We choose the recogniser for each language and process, tune how it listens, and check its work every day against real recordings.
This explainer covers what a recogniser actually does in a telephone conversation, why performance varies so much, which settings we tune per process, and how we keep checking. It ends with the questions worth asking any supplier who tells you their speech recognition is excellent.
What the recogniser does, and what it does not#
Automatic speech recognition, ASR for short, turns audio into text, or more precisely into a set of likely interpretations with confidence scores. On a phone call it works under hard conditions: narrowband audio, compression, mobile networks, traffic noise, speakerphones, people talking while walking or driving. It hears “twenty-five” and must decide whether the caller said a date, an amount or part of an account number.
In our architecture the recogniser is only the first step. What the person meant is decided by our own understanding layer, Brilliance, which interprets the utterance in the context of the dialogue and the process. That separation has been deliberate since our first prototype in 2013: understanding matters more than recognition. But understanding cannot recover what recognition lost. If the engine turns “Friday” into “fine day”, the best intent model in the world is working from the wrong words. Recognition sets the ceiling.
Why there is no single best engine#
Engines are trained on different data, tuned for different uses and updated on different schedules. The consequences are predictable once you have run enough calls:
- Languages differ. An engine that is excellent in one language may be ordinary in another, simply because of the data it learned from.
- Content types differ. Recognising free speech, a yes or no, a postcode, a date and a sum of money are different problems. Engines rank differently on each.
- Populations differ. Older callers, regional varieties and people speaking a second language challenge engines in different ways.
- Versions change. A release that improves one language can quietly make another worse. Yesterday’s ranking is not today’s.
We learned this early. Before 2016, the recognition engines available to us were dictionary-bound, slow and poor in real time, and they held back everything we built. In 2016 a cloud engine reached a usable beta and unblocked our commercial work. A year later, for the passenger assistant at Gdańsk Lech Wałęsa Airport answering in Polish and English, CCNEWS reported that we used two separate recognition technologies to improve results. The habit of comparing engines rather than marrying one dates from then.
What we tune for each process#
Choosing the engine is only the first decision. Each process gets its own settings, because a bank’s collections call in Polish and a utility’s meter-reading confirmation in German ask very different things of the recogniser.
| Setting | What it controls | What goes wrong if it is off |
|---|---|---|
| Engine per language | Which recogniser hears this language in this process | Systematic errors that no dialogue design can fix |
| Listening latency | How long the system waits before deciding the person has finished | Customers cut off mid-sentence, or silences that feel like a dropped line |
| Accent and regional variety | Which variety of the language the engine expects | Misrecognition concentrated in one region or age group |
| Vocabulary and hints | Product names, institution names, local terms the engine should expect | The words that matter most for the process are the ones most often missed |
| Expected answer type | Whether this dialogue state expects a date, an amount, a yes or no, or free speech | Numbers heard as words, words heard as numbers |
Typical settings in Vee’s processes; the exact configuration differs per process.
Accent is not a footnote#
German spoken in Vienna is not German spoken in Hamburg, and the difference shows up in recognition. The same is true of regional speech in any language, and of people who speak the language of the call as their second language. When one region or one age group shows more misrecognitions than the rest, that is usually not bad luck. It is a signal that the engine or its settings suit one population better than another, and it is fixable.
Vocabulary is process knowledge#
A general engine does not know the name of an insurer’s product, the colloquial name customers use for a type of loan, or the way patients refer to a clinic’s departments. These are often exactly the words the process depends on. Giving the engine the right hints for each process, and updating them as we hear new phrasings, is routine work for our dialogue designers.
Numbers need their own care#
Dates, amounts and reference numbers are where recognition errors cost most. A misheard word in small talk is harmless; a misheard payment date in a collections call becomes a broken promise that was never made. For these answers we tell the engine what type of answer to expect in that dialogue state, and the dialogue confirms the key value back to the customer before it is recorded. A short confirmation costs a few seconds. A wrong date written into a bank’s system costs much more.
Listening latency deserves its own decision#
Of all the settings, listening latency is the one people notice without knowing what it is. Wait too little after the customer stops speaking, and the system interrupts someone who was only pausing to find their card or remember a date. Wait too long, and the line goes quiet in a way that feels broken; in an outbound call, a quiet line is an invitation to hang up.
There is no correct value for a whole process, let alone for a language. Different questions and different customers need different patience, so we set latency per dialogue state, watch where interruptions and silences cluster, and adjust. It is a small parameter with a large effect on whether a conversation feels natural.
Checking every day against recordings#
Recognition metrics from the engine itself are useful but incomplete. An engine reports its own confidence; it does not report when it was confidently wrong. The only reliable check is to compare what the system heard with what the customer actually said, and that requires people listening.
Every day, in every project, our people listen to selected calls, especially unusual ones, to hear with their own ears what technology still cannot. – Dawid Wójcicki, Founder & CEO of Vee
When our quality team reviews a call, one of the questions is whether the recognition error was understandable: was there traffic noise, a second voice, a word nobody could have caught? Errors the engine had every right to make are accepted and handled in the dialogue, for instance by confirming a key answer. Errors it had no right to make are logged, grouped and fixed, through a vocabulary change, a latency change, a different engine for that state or, occasionally, a different engine for the whole language. Transcripts help us find calls to listen to; they are never treated as a verbatim record.
- Select calls daily: unusual outcomes, low-confidence turns, unexpected hang-ups.
- Listen and compare what was said with what was recognised.
- Decide whether each error was reasonable given the audio.
- Group unreasonable errors by cause: engine, accent, vocabulary, latency, answer type.
- Make the change, release it and measure its effect on live calls.
Why we accept the extra work#
Standardising on one vendor would be simpler for us. One contract, one integration, one set of behaviours to learn. We accept the extra work because the customer on the line does not care which vendor we chose; they care whether they were understood. The integration burden of running several engines sits with us, inside the managed service, not with the client, who needs no telephony or speech infrastructure of their own.
Switching an engine is never done on faith. When a new engine or a new version looks promising for a language, we compare it with the current one on the same kind of audio, state by state, and look at where each one fails rather than only at an average. An engine that is slightly better overall but worse on dates may be the wrong choice for collections and the right one for surveys. Only when the comparison holds up do we move live traffic, usually for one process first.
In regulated sectors the engine choice also has to fit the client’s data-protection arrangements, including where audio is processed, and that is part of the decision from the start. A technically better engine that cannot be used under the client’s rules is not an option. Within those rules, we choose by results.
What to ask a supplier about speech recognition#
- Which engine will hear our customers in our languages, and how was it chosen?
- Can you change the engine for one language or one dialogue state without rebuilding the process?
- How do you set listening time, and does it differ by question?
- How do you handle regional accents and older callers, and how will we know if one group is misheard more often?
- Who listens to real calls to check recognition, how often, and what happens with what they find?
- Where is the audio processed, and under which data-processing arrangements?
A supplier who answers with a single vendor’s name and a headline accuracy figure has answered a different question. Recognition quality is not a property of an engine. It is a property of an engine, configured for a language, a process and a population, and checked against real calls every day.
Follow The First Second
Occasional issues on the mechanics and economics of AI-powered customer contact. No noise, no vendor news.
Follow via RSS