There is the kind of programming people write books about, record courses on and argue over in comments. And there is the kind nobody talks about, because almost nobody has done it. Real-time stream processing belongs to the second kind: video from browser to browser, a live voice on a phone line, a sensor stream you cannot pause. There is no “refresh the page and try again” here: the person has already said the phrase, the frame has already flown, the heart has already beaten.
In this case we explain in plain words what stream processing is, why it is one of the hardest and least popular jobs in software, how the peer-to-peer internet works with its NAT, STUN and TURN, and how we went through all of it on real projects. Spoiler: at times it hurt, but the projects shipped and they work =)
A free consultation with an architect, and a price estimate after it. If you have a task involving streaming, video calls, voice or sensors, we will go through it on a call and send you a timeline and cost estimate by phase. Book a consultation
What stream processing is: a river, not a lake
Most business software treats data like a lake. Orders, payments and reports pile up in a database first and get processed later: hourly, nightly, at the press of a button. This is called batch processing. If the nightly report takes five minutes longer, nobody even notices.
Stream processing treats data like a river. Data flows non-stop in small chunks: a video frame, a 20 millisecond audio packet, a sensor reading. Each chunk has to be received, checked, processed and passed on before the next one arrives. If processing is late, the chunk is useless: a frame shown a second after its time is a slideshow, not a video, and a voice agent answering after five seconds feels frozen.
Five enemies lie in wait along the way, and we have met each of them personally:
- Latency. The whole path from the microphone to the other person’s ear has a budget of a few hundred milliseconds. Capture, network, processing and playback share it, and each one wants a bigger bite.
- Jitter. Packets arrive unevenly: in bursts, then with gaps. Play them as they come and the sound gurgles and stutters. You need a jitter buffer that holds the stream back a little and releases it evenly.
- Loss. Live video and voice run on UDP, which never asks for lost packets again. Rightly so: a lost frame is useless a second later. So the system has to live with holes.
- Ordering. Packets overtake each other on the network and must be put back in order by sequence numbers and timestamps. Otherwise the other side hears “no, yes” instead of “yes, no”.
- Backpressure. If processing cannot keep up, data piles up in queues, latency grows, and a minute later the conversation lags ten seconds behind. A proper stream knows when to drop the excess instead of hoarding it.
Why it is the hardest and least popular job
The ordinary web lives by “request and response”. The browser asks, the server answers, done. You can reproduce a bug, set a breakpoint, inspect variables. A stream does not allow that: while you sit on a breakpoint, the stream has flowed away, the connection has timed out and the caller has hung up. You debug on a live stream, through logs and metrics, sometimes frame by frame.
- The network is unpredictable. One user has home Wi-Fi, another has mobile data in a lift, a third sits behind an office firewall that lets only web pages through. All of it has to work the same.
- Bugs are intermittent. “It works on my machine” sounds especially sad here: the bug shows up only on one carrier, on one phone, with a weak signal.
- The stack is huge. Codecs, sample rates, RTP and SIP, WebRTC, WebSocket, NAT, certificates, browser audio APIs. Every topic deserves its own book, and those books are mostly written for telecom engineers.
- There are almost no specialists. People who have actually run streaming in production are few, and they usually work at large companies. On the services market they are practically impossible to hire, even for good money.
Today you can ask a language model some of these questions and it may even help. But most of our real-time experience was gained when there was nobody to ask. WebRTC documentation described how things should work, forums described how they did not work for someone, and between the two lay a gap we had to jump ourselves. We looked for a paid WebRTC consultant and honestly found none: those who knew how were busy, and those who were available knew the topic from the same docs we did. So everything below we went through by hand: experiments, logs and sleepless nights with the browser console.
The peer-to-peer internet: why two computers cannot just talk
The internet seems like a network where any computer can reach any other. In theory that was the plan. In practice there were not enough IPv4 addresses for everyone, and nearly every device today sits behind a router that hides it behind one shared address. This is called NAT (network address translation).
NAT: the guard at the door
A router works like a security guard in an office building. You walk out freely, the guard remembers who went where and lets the reply back in. But nobody gets in from outside without an invitation: an unknown incoming packet is simply thrown away. For websites that is perfect, you always “call first”. For a browser-to-browser video call it is a disaster: both sides sit behind their own guards, and each waits for the other to call first.
UDP vs TCP
TCP is registered mail: delivery guaranteed, order guaranteed, anything lost is resent. Ideal for pages and files. Bad for live audio: while TCP re-requests a lost packet, the whole queue behind it waits and the sound freezes. UDP is postcards: send and forget. Something gets lost, but nothing waits. That is why real-time video and voice live on UDP, and codecs and the jitter buffer deal with losses.
STUN, hole punching and ICE
For two browsers to connect directly they need three things. First, a signaling server, usually on WebSocket, to exchange “business cards” (offer, answer and address candidates). The video itself does not pass through it. Second, a STUN server that answers a simple question, “how do I look from outside”, meaning the public address and port. Third, hole punching. Both browsers send UDP packets to each other at the same time, each guard believes its tenant “called first”, and lets the reply in.
The ICE protocol conducts all of this. It gathers candidates (local address, public address from STUN, relay address) and tries them until one works.
TURN, symmetric NAT and mobile networks
Now the bad news. Symmetric NAT hands out a new public port for every new destination, which makes the STUN hint useless. Mobile carriers use CGNAT: NAT behind NAT, with thousands of subscribers behind one address. Corporate networks may block UDP entirely. In all these cases a direct path is impossible, and TURN takes the stage: a relay that receives the stream from one side and forwards it to the other. It works almost always, including over TLS on port 443, which looks like an ordinary website to a firewall. But all the traffic goes through your server, and you pay for it.
The good news: all of this can be set up so the user notices nothing. The bad news: to set it up, you have to go through every one of these cases with real people.
Experience 1: in-browser streaming for a premium social platform
Our first big real-time project was a premium social platform with video chat and streaming right in the browser. A user turns on the camera and goes live, viewers watch without installing anything, and time in chat and on stream is paid for with internal credits. So every second of video is also money: billing has to know exactly when a stream started and when it dropped.
A mesh (“everyone to everyone”) does not work for streaming: with a hundred viewers the streamer’s browser would have to send a hundred copies of the video, and the laptop would take off. So we wrote our own SFU (Selective Forwarding Unit) on Node.js. The streamer sends one stream to the server, the server distributes it to any number of viewers, and the streamer’s browser keeps exactly one connection.
- WebSocket signaling. Exchanging offer, answer and ICE candidates between browser and server, streamer and viewer roles, automatic reconnect on drops.
- Our own STUN/TURN on coturn. With a TLS certificate, dedicated ports and logs. Without it, some viewers on mobile data just saw a black screen.
- Connection lifecycle. When the streamer leaves, all their viewers disconnect cleanly; when a viewer vanishes from the network, the server frees their resources without waiting for a polite goodbye.
- WebSocket chat next to the video, watched by a supervisor that restarts the daemon after a crash.
- Mobile wrappers for Android and iOS, and recording stories straight from the device camera.
- Frame-by-frame diagnostics. We logged every frame (number, size, time) and the ICE connection state every second. It sounds paranoid, but that is how you catch bugs that hit one viewer in twenty.
A story of its own: switching cameras on the fly. A user plugs in an external camera or yanks it out mid-broadcast, the browser fires an event, and the stream has to be rebuilt without dropping the broadcast. In the docs it is one line; in real life it is a week of experiments. Later, in other projects, we also tried the off-the-shelf Janus media server, but our own SFU gave us the main thing: we understand every line of the packet’s path and can fix it ourselves.
Experience 2: real-time voice and telephony
The second big area is voice. Our AI voice secretary answers ordinary phone calls, understands the caller and replies in a natural voice. From the outside it is just a call. Inside it is a whole set of modules and builds working with a live audio stream. The Secretary itself has its own case; here we cover the streaming problems we had to solve.
- Telephony. A call arrives over SIP at an Asterisk PBX, and audio flows over RTP in 20 millisecond frames using an 8 kHz telephone codec. A Node.js server takes the stream, decodes it, resamples it for speech recognition and sends synthesized speech back at the same rhythm, frame by frame.
- Response latency. Early versions answered after several seconds, which is an eternity on the phone: the caller has time to say “hello, are you there?”. We broke the latency budget down by stage, moved recognition, generation and speech synthesis to streaming mode (we speak the first phrase while the second is still being written), shrank the buffers and brought the pause on a live call below one second.
- Echo. On a mobile line the secretary heard itself and politely answered itself. We had to learn to tell its own echo from the caller’s real speech.
- Barge-in. Real people interrupt, and the agent has to go quiet within a fraction of a second, then figure out whether it was an “uh-huh” or a new thought.
- End of turn. How do you know the person has finished rather than paused to think? We compared several approaches on an hour and a half of real recordings and chose the best balance between “interrupts” and “lags”.
- RTP quirks. For example, the address for sending audio back into the call is only known from the first incoming packet. If you do not start playing silence immediately, the caller simply will not hear the secretary. Textbooks do not cover this; you find it in logs.
- Ordering and stalls. One stalled chunk of synthesized speech could freeze the whole conversation. Now chunks play strictly in order, slow requests are backed up by spare ones, and a dropped WebSocket to the telephony side reconnects automatically.
- Switching without dropping. A call can be moved between versions of the voice pipeline mid-conversation, and the caller will not notice.
There are browser builds as well: microphone via getUserMedia with noise and echo suppression, playback through an AudioWorklet with our own jitter buffer, and a full-duplex WebRTC mode. Speech synthesis and recognition as a standalone service is covered in the NaturalTTS case.
Experience 3: sensors, biometrics and monitoring
A stream is not only video and voice. We built our first real-time temperature and humidity chart back in 2019: a sensor on an Arduino, a serial port, Python and an event stream straight into the browser. Later it grew into an IoT loop on cheap hardware that records into the sensor’s own memory while the computer is off and uploads without duplicates on reconnect.
Our densest stream is in the SHIELD project: a wearable device on an ESP32 continuously captures biometrics and sends them to the server in batches once a second. If the connection drops, the device stores data on a memory card and later resumes from where it stopped. We made the storage format roughly ten times more compact than JSON and load-tested the server with a hundred virtual devices. The browser dashboard draws live charts and redraws only what changed.
What this gives you
We do not sell “streaming in a box”. We bring experience that is hard to buy: we know where a stream breaks and design for it from day one instead of discovering it at launch.
- Video calls and broadcasts in the browser with no apps to install: consultations, training, video sales, paid live sessions billed by time.
- Voice agents on ordinary telephony: answering calls, outbound calling, call recording and transcription, CRM integration.
- Real-time monitoring of sensors and equipment: from warehouse climate to wearables, with buffering when the connection drops.
- Your own infrastructure: STUN/TURN, media servers, WebSocket gateways on your servers, with no dependence on someone else’s service and its per-minute pricing.
- An honest assessment: where an off-the-shelf service is enough and where custom development is needed. Sometimes the right answer is “do not write your own SFU”, and we will say so.
Let’s talk about your stream
If you have data that flows rather than sits, be it video, voice, calls or sensors, let’s talk. We will go through the task, show the risks and send a price and timeline estimate by phase after a free consultation. Leave a request on systems.ingello.com
