Client project · Voice AI · September–October 2026

AI Voice Sales Agent

A lead sheet goes in. The system calls each lead, holds a scripted but adaptive conversation, records what it learns, and hands qualified leads to a human sales team. Built with a teammate for a client's lead-qualification campaign.

Role: Co-developer, team of two. I owned the FreJun telephony integration, call lifecycle, latency work and the browser demo; my teammate Karthik Nambiar built the core engine.
Stack: Python, FastAPI, PostgreSQL, WebSockets, OpenAI speech-to-text, LLM and text-to-speech
Telephony: FreJun (Teler) active; Exotel and Twilio adapters behind the same interface
Links: Source on GitHub ↗

~2.9 s

median from the prospect finishing a sentence to the agent's first audio

121

automated tests: providers, routes, call lifecycle, engine regressions

telephony providers behind one interface: FreJun, Exotel, Twilio

DNC first

do-not-call is checked before every dial and every other rule

ARCHITECTURE

Two loops that never share a thread of logic

A phone call is two different problems. The media loop moves audio in real time: frames from the carrier, voice-activity detection, barge-in, and paced playback back to the line. The agent loop decides what to say: a state machine, a prompt, the model's proposal, and validation. They meet at one narrow seam: text in, text and an end-of-call flag out. That separation is what made it possible to swap carriers, and to run the same agent in the browser.

Campaigns & calls

Lead CSV import, normalised to E.164
Postgres work queue (FOR UPDATE SKIP LOCKED)
One row per call attempt, unique per (campaign, lead, attempt)
FreJun: initiate, call-flow endpoint, signed status webhooks

media WebSocket

Voice session

Speech-to-text on each utterance
Voice-activity detection and barge-in with a grace window
Text-to-speech streamed in small pieces, paced in real time

text in, text out

Agent

Script as a state machine (config, not code)
LLM proposes a reply, facts and tool calls as structured output
Validator: schema, allowed transition, allowed tool, else fallback
Rules compute the qualification outcome

DECISIONS

The parts worth explaining

The model proposes; the application decides

Every turn the LLM returns structured output: what to say, which state to move to, facts it heard, and any tool it wants to call. Nothing acts on that until a validator has checked the JSON, the schema, that the state transition is allowed by the script, and that the tool is in the registry. Anything malformed or disallowed falls back to the script's own line for the current state, so a bad model response costs one awkward sentence, never an ungoverned action. The model also never decides whether a lead qualifies: it reports facts, and configured rules compute the outcome.

Do-not-call always wins

Opt-outs write to one suppression table that every dial checks first, whether they came from the agent mid-call or from an operator. Scripts, qualification thresholds and prompts live in versioned YAML, so changing a client's pitch or bar is a configuration change, never a code change.

Plain, idempotent plumbing

No Kafka, Redis or workflow engine: a Postgres queue with SKIP LOCKED hands leads to workers, and each call attempt is its own row with a unique constraint, so a retry is a new attempt, never a duplicate dial. Status webhooks are verified with HMAC-SHA256 and a five-minute replay window, and hang-ups carry an idempotency key.

MY PART

Getting it onto a real phone line, and making it fast

FreJun (Teler) integration, built from FreJun's docs and official SDK: outbound initiation, the call-flow endpoint that answers with a stream flow, a media WebSocket that translates Teler's audio messages to the voice session and back, signed JSON status webhooks, and hang-up.
Call lifecycle: leads now leave “in progress” when the media stream ends or a terminal webhook arrives, and the attempt is committed before dialling so the carrier's callbacks can always find it.
Bugs found in live testing: the prompt context stuck on the first state, a failed tool call crashing the turn, the model never seeing tool argument schemas (so qualification was never recorded), duplicate callbacks, and the agent hanging up when asked who was calling.
Latency: models re-chosen by benchmark on this project's own prompt (the LLM's time to a speakable reply fell from 1.96 s to 0.84 s), one pooled client, a cache-friendly prompt order, a single deadline per turn instead of two stacked timeouts, and smaller audio chunks. The browser demo now plays speech as it streams, with barge-in, at a measured ~2.9 s median.

The FreJun call path is built and tested end to end in code; the first real phone call through it waits on the client's carrier account setup (Indian numbers need KYC).
Speech-to-text, the LLM and text-to-speech are three sequential network calls, so sub-second replies need a different design: streaming the caller's audio during speech, or a speech-to-speech model, which would speak before the validator can check the reply. That trade-off is a decision for the client, not a tweak.
Client-specific scripts, numbers and credentials are kept out of the repository; it ships placeholder scripts.

Next case studyClueCodeA desktop AI assistant with subscriptions, shipped end to end.