We have already written about the product itself in "An AI secretary with a convincing voice and telephony": someone calls the number owner, a voice agent picks up, finds out who is calling and why, and the recording and transcript land in the owner's Telegram. This article is about something else. Why "just plug a neural network into a phone" does not work, how many milliseconds one natural pause costs, and how we built ourselves a lab where those milliseconds can be caught, compared and tamed.
There will be no prompts or algorithms here, that is the closed part of the project. Instead you get the system's composition, the research, the tools, the charts and an honest account of where voice agents usually trip.
The key problems of a live conversation
Text chat forgives everything. The answer took three seconds? Fine, the user was busy with coffee anyway. On the phone, three seconds of silence sound like "hello, can you hear me?", and a secretary who cuts a person off mid-word sounds like a badly raised robot. Voice runs on different physics, and it has three main problems.
1. Latency: what a pause is made of
People hand the turn to each other in a couple of hundred milliseconds. Anything longer than a second already feels like a hiccup. A voice agent, though, has a whole pipeline between the caller's last word and the first sound of the reply, and every stage takes its share:
- Audio capture. A microphone or a phone line delivers sound in frames. A large frame is convenient, but every extra 4096-sample block adds tens of milliseconds of waiting.
- Network. Audio travels from the phone through the carrier to us, from us to the model and back. Jitter (uneven packet arrival) forces a buffer, and the buffer adds delay.
- Deciding "the person has finished". The trickiest stage. Until the system is sure the turn is over, it must not reply. A half-second silence window is half a second of delay even if everything else is instant.
- Recognition and the model. The model has to hear the turn, understand it and produce the first words of the answer. If the answer arrives as one chunk rather than a stream, speech synthesis waits for the whole text.
- Speech synthesis. Time to first sound differs by an order of magnitude between voice models: from a fraction of a second for fast models to several seconds for expressive ones.
- Playback. The player keeps a pre-buffer so audio does not stutter. Our first version held 900 ms, today it is several times smaller.
In the lab every reply is stamped with a measurement. In one of the test runs the line reads: pause to answer 1124 ms, of which silence window 500, model 720, synthesis 255. This is not our working configuration, just an example of how one second breaks down. You see at once what is expensive and what is nearly free.
2. VAD: who decides the person has finished
VAD (voice activity detection) answers a simple question: is the person speaking right now or not. The trouble is that for a conversation this is the wrong question. "The person went quiet" is not "the person has finished". People hesitate ("well, there, um..."), search for words, take a breath mid-sentence and then add the important bit: "...and tell him I'll call back after lunch". Over the project we tried every major family of detectors:
- Energy-based (classic). Compares frame loudness with a floating noise floor. Fast, free, runs on our server. It does not understand meaning: a thinking pause longer than the threshold, and the turn is closed.
- Local neural. A small model tells speech from noise, coughs and dial tones far better than loudness. Silero VAD, for example, weighs about 2.3 MB and runs on an ordinary CPU.
- Provider-side server VAD. Server VAD in a realtime API: the provider cuts the stream on silence of a set length. Convenient and predictable, but it is still silence, just someone else's.
- Semantic (semantic VAD, turn detection). On top of the audio runs an end-of-turn model that estimates whether the person finished the thought and decides how much longer to wait. An eagerness parameter shifts the balance: the patient mode lets people finish but can wait for seconds, the eager one answers fast but sometimes barges in mid-word.
- Audio end-of-turn model. A separate class: the model listens to the intonation and rhythm of the last fragment and returns the probability that the turn is complete. We cover it in detail below, it is the very "five-megabyte library from HuggingFace".
3. Interruptions and "uh-huh"
A live listener is not silent while you talk. They drop in "uh-huh", "right", "okay", clear their throat while thinking and say "hello?" when the line goes quiet. This is called backchannel, and without it even an agent that is perfect in content sounds like an answering machine. The other side of the coin is barge-in. If the caller starts talking while the secretary is answering, the secretary should stop. On the phone, however, the agent's own voice comes back as echo, and a naive detector starts interrupting itself. Telling a real "wait, I haven't finished" from echo and from the caller's casual "uh-huh" is an engineering task of its own.
That model from HuggingFace
A word about the model we get asked about most. It is Smart Turn v3 by the Pipecat team: an open model that listens to the last seconds of speech and decides whether the person has finished or is just pausing. People call it "the five-megabyte one", we weighed it honestly: the 3.2 CPU file (int8, ONNX) is 8.7 MB, and it works in tandem with Silero VAD at 2.3 MB. About 11 MB together, less than one phone photo. No GPU needed: tens of milliseconds per decision on an ordinary laptop.
We run it as a separate service next to the voice core, checked our feature extraction against the reference implementation to the seventh decimal and found a surprise: plain linear resampling from 24 to 16 kHz noticeably changed the model's answers. A proper filter fixed it. Details like this are what eat weeks in voice projects.
How good is it on Russian and Ukrainian phone speech? That question is exactly why we built the comparison bench, more on it below.
What the system is made of
The secretary grew into a family of voice runtimes on a shared core rather than a single program. Each runtime answers three questions in its own way: who thinks, who speaks and where the audio comes from. Everything else (speech queue, detectors, synthesis, phrase bank, billing, Telegram, analytics) is shared and reused.
- Telephony bridge. Takes the forwarded call, decides whether to pick up (favourite contacts are not answered, they hear normal ringing), runs the conversation, records it and sends the report.
- Voice core. End-of-turn detectors, a realtime dialogue model, speech synthesis, a bank of short inserts, barge-in logic.
- Web secretary (the lab). The same secretary, but the call comes from a browser. Here we try models, detectors, voices and roles without a phone line.
- Fast pipeline. A variant tuned for minimal delay: streamed answer, phrase-by-phrase synthesis, a short pre-buffer, barge-in.
- Experimental runtimes. A scenario classifier (the model only picks a scenario number, the phrases are prerecorded), a cascade "recognition, third-party language model, synthesis", and a full-duplex mode where the model listens and speaks at the same time.
- Monitor. Calls, charts, events, recordings, a table of turns.
- Detector comparison bench. Runs recordings through every mode in real time and counts errors.
- Telegram bots. Version one, "Robo-secretary", and version two, "Телефонний секретар": the owner's account, settings, balance, call reports.
- Smart Turn service. A separate lightweight process with the end-of-turn model.
The data is deliberately simple: all the complexity lives in the conversation, not in the tables. The number owner, their calls, the turns of every call with audio fragments, favourite contacts, reminders and payments. Voice core settings, which the lab changes on the fly, live separately.
The lab
The project's main tool. It is a web app where you can talk to the secretary straight from the browser and change almost everything on every call: model, detector, voice, role, pace. Under the hood it is the same core that answers the phone, so whatever we learn in the lab carries over to the line without translation.
What can be tuned
- Agent voice and language. Male or female, Russian or Ukrainian. The voice is more than timbre: it has its own bank of short inserts.
- End of turn. Three modes: the provider's semantic VAD, our loudness-based audio VAD and Smart Turn v3. In the last two we switch the provider's detection off and tell the model "answer" ourselves.
- Wait after an open question. An extra 0 to 1200 ms. After "what is it about?" people think longer than after "shall I put you through?", and the detector benefits from knowing that.
- VAD pause (eagerness). Auto, low, medium, high: how quickly the semantic detector closes the turn.
- Pause reaction in the fast pipeline. Server VAD with a 250 to 700 ms silence window, or the semantic modes.
- Realtime model. Several generations of dialogue models, including the lightweight "mini" ones. Faster is not always better: a small model answers sooner but asks an unnecessary question more often.
- Synthesis model. Expressive (slower to first sound), fast (about 0.2 s), balanced, and in the fast pipeline also the realtime model's native voice with no separate synthesis.
- Voice from the catalogue. Pulled from the synthesis catalogue and filtered by model compatibility and Russian or Ukrainian support.
- Tone tag. For the expressive model: warmly, gently, friendly, calm, cheerful.
- Speech pace. From 0.8 to 1.2. A funny finding: on the expressive model the parameter barely works, its pace wanders on its own.
- Role. "Secretary" or "an acquaintance asked to answer". Not cosmetic: people talk to a secretary formally and briefly, to an acquaintance more freely and at length, which means they pause differently.
- Background sounds and inserts. "Uh-huh" while the caller speaks, "hmm" or "okay" while the model thinks, "hello?" on silence, a lead-in before the answer.
- Barge-in. Whether to listen to the caller while the secretary speaks and stop on their turn.
- Instructions. An addition to the role description for new connections: "be even shorter today", "don't mention email".
How the settings affect each other
The fun starts when you realise no setting lives on its own. A few patterns we took away from hundreds of test calls:
- Detector and synthesis add up. A patient detector plus an expressive voice model gives you a pause long enough to make tea. Fast synthesis lets the detector be more patient, slow synthesis demands decisiveness, and a decisive detector interrupts more often.
- Inserts mask latency, but not for free. An "uh-huh" in the pause makes waiting feel natural. But an insert cut off halfway by the model's answer sounds worse than honest silence. So inserts, model and detector are only ever tuned together.
- The dialogue model changes the person's pauses. A long secretary reply invites interruptions, a question with options shortens the caller's answer, an open question lengthens their thinking.
- Language and voice shift the thresholds. Ukrainian and Russian, male and female voice, a phone line and a browser microphone: detectors behave a little differently on every combination.
- Barge-in without echo protection becomes self-interruption. It can only be enabled together with proper handling of the agent's own voice.
The prompt is a setting too, and one of the strongest. How the role is described drives reply length, the number of clarifying questions, the habit of confirming what was heard and how readily the model says goodbye. The same set of models with different role descriptions sounds either like a living person or like a questionnaire. We do not publish the texts themselves.
Which combination we consider the working one is closed project information. The screenshots show test states of the lab, not production settings. In short: there is no "best VAD", there is a good combination of detector, model, voice, inserts and role description for a specific scenario and specific people on the line.
Research and tools
The monitor and the conversation chart
Every call and every test session leaves a trace: what the server heard, which events happened and at which millisecond. The monitor lays this out on a timeline: microphone level, sharp volume drops, the moments the detector closed the turn, the model started and finished answering, synthesis started and ended. Phone calls add jitter and reply buffer fill. Phase two calls also get a table of turns: who said what and why (model answer, insert, greeting, farewell).
The same chart lives in the lab in real time: while you speak you see the volume rise, where the threshold is, how much audio sits in the player buffer and when the detector made its call. It saves a lot of arguing: instead of "I think it interrupted me" there is a picture showing who and when.
The "uh-huh" inserts panel
We do not generate short sounds on the fly, they would arrive late. Each voice has a prerecorded bank: "uh-huh" and "yeah" for listening, "hmm", "okay", "umm" for thinking, "hello?" for silence. The panel shows every insert with its waveform and length, lets you listen, drop the odd one and build a lead-in queue that plays before answers in rotation so the secretary does not repeat itself.
The phone version has a separate voice core panel: dialogue model, pause reaction, synthesis model and tone, voices, a switch for every kind of insert, barge-in, a farewell before hang-up, a soft limit on call length. Everything applies to new calls without a restart.
The VAD comparison bench
You can argue about detectors by feel forever, so we built a bench. It holds 81 minutes of real audio: 13 phone calls and 110 web sessions, 394 turn ends, 190 pauses where the person hesitates and 99 pauses of the "finished, thought, added more" kind. Reference labels come from word-level timestamps of speech recognition, then every recording runs through every mode at once and in real time, as if the calls were live.
We count what the person on the line hears: how often the detector cuts off someone searching for words; how many turns get cut at least once; how often the system "sticks" and stays silent for over a second and a half after a finished sentence; median and 90th percentile decision delay; false triggers on noise. Every number comes with audio examples on a timeline showing the moment each mode decided "finished".
One finding we can share: Smart Turn v3 really is the fastest when it is confident (a decision about half a second after the end of a phrase versus almost eight tenths for the other modes). But on our Russian and Ukrainian recordings it noticeably more often confuses "went quiet to think" with "finished the thought". The small model has a great ear for intonation, and our callers love thinking out loud. Which mode, and with which parameters, went into production stays behind the scenes.
Telegram bots
The owner never opens an admin panel, their whole account lives in Telegram. There are two bots, and they are two eras of the project.
Version one, Robo-secretary. The user's account: connection, call reports with recording and transcript, favourite contacts, text reminders and a reminder call for when you need to politely escape a meeting ("pretend to be busy"), voice messages instead of typing, buying tokens.
Version two, "Телефонний секретар" (Phone secretary). Rebuilt from scratch for phase two: Ukrainian by default, the whole interface on buttons. Onboarding with a sample conversation, choosing the secretary's voice and name, a free trial, connecting a number and step-by-step call forwarding setup (the code is dialled with one button), status check, a second number, introduction profiles, favourite contacts, balance and payment, support. After every call there is one message: who called, why, the recording and the transcript. There is also a "share the bot" page for SMS and messengers.
The hardest part, which we will keep quiet about
Everything above is the tip of the iceberg. Below the waterline are three things that took most of the engineering, and they stay inside the project.
- Decision algorithms. Between the caller and the model sits our own layer that decides when to listen, when to answer, when to drop in an "uh-huh", when to stop and what to do when two detectors disagree. It is what separates "a neural network on the phone" from a conversation partner. How it works we will not tell even over tea and biscuits.
- Admin panel. Users, calls, tokens, payments, analytics, bot texts, profiles. Everything that makes the service a business rather than a demo.
- Infrastructure. Telephony, switching contours without dropping calls, a safety net for when the new contour does not know the client, bots isolated from each other, service health monitoring. Sounds boring, fixes merrily at 3 a.m.
If you are building your own voice agent, this is where you will lose the most time. We have already lost it for you, and that is our value.
Summary
A realistic voice conversation rests not on one "smart model" but on milliseconds, pauses and small things like a well-timed "uh-huh". Working with them takes measurement: a lab where any combination is assembled in a minute, a monitor where every call is visible on a timeline, and a bench where detectors compete on live audio rather than on README promises. That is the main result of the project: not only a secretary that answers calls, but a method we use to bring any voice interface to life.
Need a voice agent for inbound calls, client booking, first-line support or your own SaaS? Let's start with a free consultation with an architect: we will go through your scenario and calculate the price after the call. More about our approach at systems.ingello.com.
Where to see it
The character behind it, a voice agent with an avatar that drives Codex and Claude Code: "INGELLA: a voice AI agent".
The first, deterministic version of the secretary, with its pros and cons: an AI secretary that follows scenarios, where every word is known in advance.
A similar solution: An AI agent that sells by itself.
