Let the AI Out: Audient — An Ambient Audio Perception Layer That Grows
This post is part of the Let the AI Out series on giving AI agents direct access to the physical world. Start here for the overview.
This series has been about giving agents hands — BLE, serial, the debug probe. Channels the agent opens and closes when it’s done.
Agents already consume vision, text, and speech fine. You hand them an image, they tell you what’s in it; speech they get too, once a speech-to-text model has done its job. All of it is requested input: someone pointed, someone asked, someone hit record.
What about audio that isn’t speech?
A bird singing outside while you read. A glass breaking two rooms away. A smoke-alarm ringing. Whether any of that matters depends on context, and no fixed classifier gets you there. What gets you there is a vocabulary of what you’ve heard before, and a way to grow it.
That’s what audient is: an experimental audio perception layer for AI agents — an MCP server plus a browser viewer, one repo, one install. Continuous audio in, event-gated recognition out, a concept memory that grows through use, and a registry of specialized audio models the agent chains as it needs them.
Useful anywhere an agent should notice audio it wasn’t told to listen for: a robot that reacts to a crash it wasn’t watching for, an assistant that hears the oven’s still beeping, an industrial monitor that catches a bearing before it fails, an ecological deployment tracking a species that showed up where it shouldn’t. Mostly, I built it to see if the loop worked.
How an ear becomes a perception
Roughly four stages, each doing one thing.
- A peripheral gate. The cochlea and brainstem don’t pass everything upstream. Onsets, transients, energy changes (the things that might be signal) get amplified; steady background gets attenuated. The first “decision” is “is this worth attending to?”, and it is made before any recognition has happened.
- A fingerprint. The signal gets encoded as a distinctive pattern of neural activity, organized by frequency, changing over time. Not a stored recording; a shape the rest of the system compares against past patterns.
- A working-memory match. The fingerprint gets compared against everything you’ve heard before that’s still organized enough to retrieve. Familiar → strong match, fast recognition. Unfamiliar → no match, attention flares.
- Specialized modules, when they apply. Speech, music, and environmental sound each get routed to different cortical regions, each tuned to its domain. Recognition isn’t one classifier: it’s a set of specialists the system picks from.
There’s a fifth thing, the part that makes the whole loop close: you can look or ask. You hear something you don’t know, and you turn your head, or you ask the person next to you “what was that?” Then the next time it happens you know.
The same four stages, in code
The audient growth loop mirrors the biology stage-for-stage:
- Event gate. Energy, novelty, and voice-activity detection trim the continuous signal down to candidate regions. Most of the audio never reaches the rest of the system.
- Fingerprint. Every candidate gets a CLAP embedding (the dense, comparable form) and a panel of symbolic features (spectral shape, peaks with prominence, harmonic content, temporal stability, energy and noise floor, etc.). The embedding is for math; the symbolic features are for the agent to reason about.
- Concept memory. A sqlite-vec store of every concept the agent has learned. Each candidate’s embedding gets a top-k cosine lookup. Strong match against a calibrated concept (above its threshold) fires recognition server-side, optionally invoking the concept’s linked specialist to enrich the event. No agent in the loop. Anything weaker surfaces to the agent.
- Specialists. A registry of pre-trained classifiers. Whisper for speech. BirdNET for bioacoustics. AST (AudioSet, 527 coarse labels) as the broad-coverage fallback the agent reaches for first when memory misses. Each declares, in plain English, when to use it, its failure modes, and how to interpret its confidence.
- Unhandled bucket. Events the agent couldn’t recognize land here, waiting for a teacher.
Two paths. For familiar, calibrated concepts, the server recognizes directly: fast, automatic, no agent involved. For everything else, the MCP server exposes the tools and the agent sequences them itself. The dispatcher is the LLM whenever there’s a decision worth making.
Three ways to reason about a sound
The agent reasons over symbolic representations, never raw audio. The architecture gives it three layers to reason across.
The signal. Analysis tools (e.g., spectrum) return plain numbers: roughly two dozen features spanning energy, spectral shape, harmonic content, peaks, and temporal drift. The agent reads them as JSON and reasons about them the way an audio engineer would. Strong peak at 1019Hz, 38dB prominence, harmonic ratio 0.91. That’s tonal, not broadband. Low temporal drift: it’s sustained, not a transient.
The embedding. The agent never sees the CLAP vector itself. It sees the similarity score against everything in concept memory. query_memory returns the top-k matches with their cosine similarity, the concept’s threshold, and the recognizer kind. 0.91 against smoke-alarm, threshold 0.80: confident match. 0.62 against kitchen-timer, 0.58 against microwave-beep: neighbors, not matches.
The labels. Specialists return structured labels. Whisper transcripts. BirdNET species with confidence. AST’s coarse AudioSet guess across 527 categories. The agent hands a sound off via invoke_specialist and reads back the structured answer alongside the specialist’s plain-English contract for how to interpret it.
The agent can disagree with the specialist if the signal features don’t fit. It can override the embedding similarity if the labels suggest the categories diverged. No single layer is authoritative.
When you hear something you don’t know
The unhandled bucket is the turn your head and ask primitive.
When concept memory has no good match and no specialist comes back confident, the event lands in a queue of pre-trimmed audio clips with their fingerprints and whatever the system thinks it heard, waiting for a teacher. The teacher pulls from it and labels the sound with a name. That name becomes a concept the store can find on the next similar event.
Today the teacher is a human. The natural extension is cross-modal: an agent that hears something unfamiliar and looks toward the source with a camera, then labels the sound with what it saw. Crash in the kitchen → vision sees broken glass → labels the sound as glass-break.
Handing off to specialists
The auditory cortex has specialized regions for different domains: Wernicke’s area for speech, lateral belt for environmental sound, regions for music. The system mirrors that: no monolithic audio model, a registry of specialists instead, each declaring what it’s for in plain English the agent can read.
Three specialists ship today. The agent picks who to delegate to. AST is what it reaches for first when memory misses (broad coverage, fast), and the result tells it whether to follow up with BirdNET, Whisper, or nothing. The contract for each specialist (when_to_use, failure_modes, confidence_interpretation) is what lets an LLM reason about a classifier it didn’t train.
Adding a new specialist is registering a spec.yaml and an adapter.py. The agent reads the spec and decides when to use it. The same slot can hold specialists trained from the store itself as vocabulary accumulates; a system that can self-extend.
How a concept grows
The clearest way to see what this system does is to follow one sound through its life in the store. A simple example, no specialist involved.
Prefer to watch the whole thing? Full 90-second demo on YouTube.
Step 1. Fresh run, empty store. You whistle into the mic. The gate fires. query_memory returns nothing — the store has never seen this sound. The agent reaches for AST as its broad-coverage fallback. AST returns Whistling @ 0.906, tier=very_high. Confident enough to skip the unhandled path. Agent calls record_event(label="whistling", confidence=0.906). First example of a new concept.
Step 2. You whistle again. Gate fires. query_memory returns whistling @ 0.98 — well above the concept’s default 0.80 threshold. Strong match. Agent reads the score, confirms the label, records the event. The concept extends. The Memory sidebar’s counter ticks up with each whistle; the centroid tightens.
Step 3. After six examples, a green PROMOTE? badge lights up on the whistling concept card — audient’s hint that the concept has cleared the minimum for calibration to succeed. Click into the concept, click Promote concept. Calibration runs and picks the smallest threshold that clears 0.95 precision. Here it lands at 0.958 — high, because with only one concept in the store there’s no cross-family variance to work with; calibration has to be conservative.
Step 4. From that point on, the fast path takes over. Next whistle: cosine reads 0.964, clears 0.958, and the server records it directly; no MCP round-trip, no LLM call, no reasoning turn. The Live Transcript renders a compact auto-recognized · whistling · 0.99 row. Then another. Then another. The panel’s character changes: full classifier turns give way to slim acknowledgment rows.
Two rows worth pausing on. Mid-run, a random noise landed in the gate (an accidental tap); low energy, impulsive character, no memory match, AST tier=low. The classifier read the features, reasoned faint transient or percussion, and punted to unhandled. That’s the system doing what it should: a slot for “I don’t know,” and a path someone can label later.
A weaker whistle came in at 0.910: above the default 0.80 but below the calibrated 0.958. The fast path bailed. The classifier picked it up, queried memory (0.910 centroid), asked AST for a second opinion, saw both converge on whistling, and recorded it. Even after promotion, the LLM is still there for the edge cases.
The LLM’s role shrinks each beat: from doing all the work, to confirming memory’s guess, to nothing at all, except on the events where it should be there. No model was trained. The “model” is a centroid in a vector store, a name attached to it, and a calibrated threshold. The expensive ML work is what CLAP, AST, and BirdNET already did. The rest is bookkeeping.
Getting started
Setup instructions are in the README. Two ways to use it: through the browser viewer that ships with audient, or through Claude Code (or any MCP-capable host) that talks to the same server. Both hit the same SQLite concept memory.
The browser viewer
Three surfaces on the same data.
Live. What’s happening right now. Continuous audio scope, event chips dropping in as the gate fires, the classifier’s reasoning in a side panel, the memory grid pulsing when a concept extends.
Timeline. What happened. Chronological events, playable, with the label, confidence, and which recognizer decided.
Memory. What the agent has learned. Concepts in the store, their examples, how their centroids have moved over time, when they were taught.
From Claude Code
The default entry point speaks stdio MCP. Point your client at the audient command in the server directory and you get the tool surface.
Note: Claude Code doesn’t listen for MCP push notifications, so it’s more of a pull-based approach.
Known limitations
audient is experimental. It’s been lightly evaluated on recorded data during development; no formal benchmark suite or public-dataset numbers yet.
Single-source assumption. Each detected region is treated as one event. Overlapping sounds — a bird call over music, a doorbell mid-conversation — get classified as whichever signal dominates. The other is lost. Source separation isn’t wired up.
macOS-first. Everything has been tested on macOS. The dependencies (torch, transformers, faster-whisper, sounddevice/portaudio) all have Linux wheels, so Linux should work with minimal fuss; I just haven’t verified. Windows needs a Windows branch added to the BirdNET dep gate in pyproject.toml and probably a look at the mic-source path; nothing about the models themselves is Windows-incompatible.
Prompt tuned for Anthropic. The classifier runs through fast-agent so any supported provider (Anthropic, OpenAI, Gemini, local via Ollama) will run, but the prompt was written and tested against Anthropic models. Switching may need prompt adjustments.
Open directions
A few directions from here:
- Agent-grown specialists. The registry doesn’t have to be hand-curated. The agent could discover them (searching Hugging Face for a pretrained model that fits a concept it’s struggling with and registering it on the fly) or train them from the store itself: a small learned classifier over the CLAP embeddings the store already has, a family disambiguator, a speaker-ID head.
- Per-subclass concept splitting. Concepts live at the family level today: “bird vocalization” holds examples across species, with the species as
subclassmetadata per event. When variance under a family is high, the store could promote subclasses to leaf concepts under the family root. - Voice beyond words. Speech goes to Whisper for transcript only. Tone, emotion, speaker identity, prosody, stress; the architecture supports adding these as specialists under the “speech” concept, each declaring its own contract; none exist today.
- Cross-modal teaching. The unhandled bucket assumes a human teacher. Pair it with a vision channel and the system labels its own audio by looking: hear the crash, turn the camera, see the broken glass, label the sound.
- Spatial awareness. A mic array and beamforming give events a where: baby crying, back of the room; glass break, kitchen. Pairs naturally with vision-pointing.
Closing thought
The series has been about giving agents direct, stateful access to interfaces that used to require humans: BLE, serial, debug probes. Hands.
Ears are different. There’s no protocol to wrap. The problem is a vocabulary problem: how do you build a working map of what your environment sounds like, without one to start from? The answer ends up looking a lot like what a brain does.