This post is part of the Let the AI Out series on giving AI agents direct access to the physical world. Start here for the overview.

This series has been about giving agents hands — BLE, serial, the debug probe. Channels the agent opens and closes when it’s done.

Agents already consume vision, text, and speech fine. You hand them an image, they tell you what’s in it; speech they get too, once a speech-to-text model has done its job. All of it is requested input: someone pointed, someone asked, someone hit record.

What about audio that isn’t speech?

A bird singing outside while you read. A glass breaking two rooms away. A smoke-alarm ringing. Whether any of that matters depends on context, and no fixed classifier gets you there. What gets you there is a vocabulary of what you’ve heard before, and a way to grow it.

That’s what audient is: an experimental audio perception layer for AI agents — an MCP server plus a browser viewer, one repo, one install. Continuous audio in, event-gated recognition out, a concept memory that grows through use, and a registry of specialized audio models the agent chains as it needs them.

Useful anywhere an agent should notice audio it wasn’t told to listen for: a robot that reacts to a crash it wasn’t watching for, an assistant that hears the oven’s still beeping, an industrial monitor that catches a bearing before it fails, an ecological deployment tracking a species that showed up where it shouldn’t. Mostly, I built it to see if the loop worked.

How an ear becomes a perception

Roughly four stages, each doing one thing.

1. A peripheral gate. The cochlea and brainstem don’t pass everything upstream. Onsets, transients, energy changes (the things that might be signal) get amplified; steady background gets attenuated. The first “decision” is “is this worth attending to?”, and it is made before any recognition has happened.
2. A fingerprint. The signal gets encoded as a distinctive pattern of neural activity, organized by frequency, changing over time. Not a stored recording; a shape the rest of the system compares against past patterns.
3. A working-memory match. The fingerprint gets compared against everything you’ve heard before that’s still organized enough to retrieve. Familiar → strong match, fast recognition. Unfamiliar → no match, attention flares.
4. Specialized modules, when they apply. Speech, music, and environmental sound each get routed to different cortical regions, each tuned to its domain. Recognition isn’t one classifier: it’s a set of specialists the system picks from.

There’s a fifth thing, the part that makes the whole loop close: you can look or ask. You hear something you don’t know, and you turn your head, or you ask the person next to you “what was that?” Then the next time it happens you know.

The same four stages, in code

The audient growth loop mirrors the biology stage-for-stage:

• Event gate. Energy, novelty, and voice-activity detection trim the continuous signal down to candidate regions. Most of the audio never reaches the rest of the system.
• Fingerprint. Every candidate gets a CLAP embedding (the dense, comparable form) and a panel of symbolic features (spectral shape, peaks with prominence, harmonic content, temporal stability, energy and noise floor, etc.). The embedding is for math; the symbolic features are for the agent to reason about.
• Concept memory. A sqlite-vec store of every concept the agent has learned. Each candidate’s embedding gets a top-k cosine lookup. Strong match against a calibrated concept (above its threshold) fires recognition server-side, optionally invoking the concept’s linked specialist to enrich the event. No agent in the loop. Anything weaker surfaces to the agent.
• Specialists. A registry of pre-trained classifiers. Whisper for speech. BirdNET for bioacoustics. AST (AudioSet, 527 coarse labels) as the broad-coverage fallback the agent reaches for first when memory misses. Each declares, in plain English, when to use it, its failure modes, and how to interpret its confidence.
• Unhandled bucket. Events the agent couldn’t recognize land here, waiting for a teacher.

Two paths. For familiar, calibrated concepts, the server recognizes directly: fast, automatic, no agent involved. For everything else, the MCP server exposes the tools and the agent sequences them itself. The dispatcher is the LLM whenever there’s a decision worth making.

Three ways to reason about a sound

The agent reasons over symbolic representations, never raw audio. The architecture gives it three layers to reason across.

The signal. Analysis tools (e.g., spectrum) return plain numbers: roughly two dozen features spanning energy, spectral shape, harmonic content, peaks, and temporal drift. The agent reads them as JSON and reasons about them the way an audio engineer would. Strong peak at 1019Hz, 38dB prominence, harmonic ratio 0.91. That’s tonal, not broadband. Low temporal drift: it’s sustained, not a transient.

The embedding. The agent never sees the CLAP vector itself. It sees the similarity score against everything in concept memory. query_memory returns the top-k matches with their cosine similarity, the concept’s threshold, and the recognizer kind. 0.91 against smoke-alarm, threshold 0.80: confident match. 0.62 against kitchen-timer, 0.58 against microwave-beep: neighbors, not matches.

The labels. Specialists return structured labels. Whisper transcripts. BirdNET species with confidence. AST’s coarse AudioSet guess across 527 categories. The agent hands a sound off via invoke_specialist and reads back the structured answer alongside the specialist’s plain-English contract for how to interpret it.

The agent can disagree with the specialist if the signal features don’t fit. It can override the embedding similarity if the labels suggest the categories diverged. No single layer is authoritative.

When you hear something you don’t know

The unhandled bucket is the turn your head and ask primitive.

When concept memory has no good match and no specialist comes back confident, the event lands in a queue of pre-trimmed audio clips with their fingerprints and whatever the system thinks it heard, waiting for a teacher. The teacher pulls from it and labels the sound with a name. That name becomes a concept the store can find on the next similar event.

Today the teacher is a human. The natural extension is cross-modal: an agent that hears something unfamiliar and looks toward the source with a camera, then labels the sound with what it saw. Crash in the kitchen → vision sees broken glass → labels the sound as glass-break.

Handing off to specialists

The auditory cortex has specialized regions for different domains: Wernicke’s area for speech, lateral belt for environmental sound, regions for music. The system mirrors that: no monolithic audio model, a registry of specialists instead, each declaring what it’s for in plain English the agent can read.

Three specialists ship today. The agent picks who to delegate to. AST is what it reaches for first when memory misses (broad coverage, fast), and the result tells it whether to follow up with BirdNET, Whisper, or nothing. The contract for each specialist (when_to_use, failure_modes, confidence_interpretation) is what lets an LLM reason about a classifier it didn’t train.

Adding a new specialist is registering a spec.yaml and an adapter.py. The agent reads the spec and decides when to use it. The same slot can hold specialists trained from the store itself as vocabulary accumulates; a system that can self-extend.

How a concept grows

The clearest way to see what this system does is to follow one sound through its life in the store. A simple example, no specialist involved.

Prefer to watch the whole thing? Full 90-second demo on YouTube.

Step 1. Fresh run, empty store. You whistle into the mic. The gate fires. query_memory returns nothing — the store has never seen this sound. The agent reaches for AST as its broad-coverage fallback. AST returns Whistling @ 0.906, tier=very_high. Confident enough to skip the unhandled path. Agent calls record_event(label="whistling", confidence=0.906). First example of a new concept.

Step 2. You whistle again. Gate fires. query_memory returns whistling @ 0.98 — well above the concept’s default 0.80 threshold. Strong match. Agent reads the score, confirms the label, records the event. The concept extends. The Memory sidebar’s counter ticks up with each whistle; the centroid tightens.

Step 3. After six examples, a green PROMOTE? badge lights up on the whistling concept card — audient’s hint that the concept has cleared the minimum for calibration to succeed. Click into the concept, click Promote concept. Calibration runs and picks the smallest threshold that clears 0.95 precision. Here it lands at 0.958 — high, because with only one concept in the store there’s no cross-family variance to work with; calibration has to be conservative.

Step 4. From that point on, the fast path takes over. Next whistle: cosine reads 0.964, clears 0.958, and the server records it directly; no MCP round-trip, no LLM call, no reasoning turn. The Live Transcript renders a compact auto-recognized · whistling · 0.99 row. Then another. Then another. The panel’s character changes: full classifier turns give way to slim acknowledgment rows.

Two rows worth pausing on. Mid-run, a random noise landed in the gate (an accidental tap); low energy, impulsive character, no memory match, AST tier=low. The classifier read the features, reasoned faint transient or percussion, and punted to unhandled. That’s the system doing what it should: a slot for “I don’t know,” and a path someone can label later.

A weaker whistle came in at 0.910: above the default 0.80 but below the calibrated 0.958. The fast path bailed. The classifier picked it up, queried memory (0.910 centroid), asked AST for a second opinion, saw both converge on whistling, and recorded it. Even after promotion, the LLM is still there for the edge cases.

The LLM’s role shrinks each beat: from doing all the work, to confirming memory’s guess, to nothing at all, except on the events where it should be there. No model was trained. The “model” is a centroid in a vector store, a name attached to it, and a calibrated threshold. The expensive ML work is what CLAP, AST, and BirdNET already did. The rest is bookkeeping.

Getting started

Setup instructions are in the README. Two ways to use it: through the browser viewer that ships with audient, or through Claude Code (or any MCP-capable host) that talks to the same server. Both hit the same SQLite concept memory.

The browser viewer

Three surfaces on the same data.

Live. What’s happening right now. Continuous audio scope, event chips dropping in as the gate fires, the classifier’s reasoning in a side panel, the memory grid pulsing when a concept extends.

Timeline. What happened. Chronological events, playable, with the label, confidence, and which recognizer decided.

Memory. What the agent has learned. Concepts in the store, their examples, how their centroids have moved over time, when they were taught.

From Claude Code

The default entry point speaks stdio MCP. Point your client at the audient command in the server directory and you get the tool surface.

Note: Claude Code doesn’t listen for MCP push notifications, so it’s more of a pull-based approach.

Known limitations

audient is experimental. It’s been lightly evaluated on recorded data during development; no formal benchmark suite or public-dataset numbers yet.

Single-source assumption. Each detected region is treated as one event. Overlapping sounds — a bird call over music, a doorbell mid-conversation — get classified as whichever signal dominates. The other is lost. Source separation isn’t wired up.

macOS-first. Everything has been tested on macOS. The dependencies (torch, transformers, faster-whisper, sounddevice/portaudio) all have Linux wheels, so Linux should work with minimal fuss; I just haven’t verified. Windows needs a Windows branch added to the BirdNET dep gate in pyproject.toml and probably a look at the mic-source path; nothing about the models themselves is Windows-incompatible.

Prompt tuned for Anthropic. The classifier runs through fast-agent so any supported provider (Anthropic, OpenAI, Gemini, local via Ollama) will run, but the prompt was written and tested against Anthropic models. Switching may need prompt adjustments.

Open directions

A few directions from here:

• Agent-grown specialists. The registry doesn’t have to be hand-curated. The agent could discover them (searching Hugging Face for a pretrained model that fits a concept it’s struggling with and registering it on the fly) or train them from the store itself: a small learned classifier over the CLAP embeddings the store already has, a family disambiguator, a speaker-ID head.
• Per-subclass concept splitting. Concepts live at the family level today: “bird vocalization” holds examples across species, with the species as subclass metadata per event. When variance under a family is high, the store could promote subclasses to leaf concepts under the family root.
• Voice beyond words. Speech goes to Whisper for transcript only. Tone, emotion, speaker identity, prosody, stress; the architecture supports adding these as specialists under the “speech” concept, each declaring its own contract; none exist today.
• Cross-modal teaching. The unhandled bucket assumes a human teacher. Pair it with a vision channel and the system labels its own audio by looking: hear the crash, turn the camera, see the broken glass, label the sound.
• Spatial awareness. A mic array and beamforming give events a where: baby crying, back of the room; glass break, kitchen. Pairs naturally with vision-pointing.

Closing thought

The series has been about giving agents direct, stateful access to interfaces that used to require humans: BLE, serial, debug probes. Hands.

Ears are different. There’s no protocol to wrap. The problem is a vocabulary problem: how do you build a working map of what your environment sounds like, without one to start from? The answer ends up looking a lot like what a brain does.

Links

• audient on GitHub
• Let the AI Out series

Obsidian didn’t win on features. It won as a second brain: the place your thinking accumulates and stays yours, in plain files you control. And it’s already everywhere I am. Obsidian Sync, or iCloud, or a self-hosted setup keeps the vault mirrored across my laptop, my desktop, my phone. As a place to read and write my own notes, the reach problem was solved years ago.

What changed is how I use it. A second brain used to be a one-way dump. A thought shows up, I drop it into my notes raw, promise to tidy it later, never do. The notes fills with fragments I have to clean up before they’re worth anything. Now the thought goes to Claude first. I think it through in the conversation, the agent helps me sharpen it, and the version that reaches the notes is already clean. The AI slid in between me and my notes and became the way I write to them. I don’t dump into my second brain anymore. I dump into Claude, and Claude files the refined result.

At my desk this mostly works. The vault files sit right there on disk, and you can wire a local tool to read them. But that’s not where the thinking happens. The ideas show up when I’m away from the machine: walking somewhere, on a train, halfway through a book. In those moments Claude is on my phone and my whole vault is synced to the same phone, two apps an inch apart, and they have no way to reach across to each other.

So I went looking for a way to give the AI access from anywhere I am. There were a couple of options, but all came down to the same deal: leave your laptop always-on. That works, technically. It also makes your second brain hostage to whether you remembered to leave a computer on. I wanted it to work with every device off. Nothing did. So I built it a few months ago, and I’ve used it every day since.

Why the AI couldn’t get in

The sync tools all keep your vault for you. They move notes between your own devices and stop there. None of them is something an outside agent can read from.

Obsidian Sync is a closed, end-to-end pipe between copies of Obsidian. There’s no API to reach into it from anywhere else. iCloud is a file-sync layer between your Apple devices; Claude on the web has no path into it at all. The Local REST API plugin does expose an HTTP endpoint, but only while Obsidian is running, on a machine that’s awake and reachable. Every option is either a sealed box or needs a device of mine on.

Self-hosted LiveSync is the exception, and the reason is that it’s open source. The community plugin, 600k+ downloads, syncs your vault into a CouchDB database that you own and control, in a documented format that its own shared library can read and write. That’s not a sealed pipe. It’s a database with a door. And because the database is yours, you decide where it runs: locally on your own machine if that’s all you need, or on any always-on host you point it at, including a Fly.io box that costs close to nothing. Put it somewhere that stays up and your notes are reachable no matter which of your devices are asleep, chunked and end-to-end encrypted. Everything an agent would need was sitting in place. The one missing piece was the thing that lets the agent in.

The build

Obsidian Sync MCP gives any AI agent read and write access to your vault. It runs in one of two modes.

Filesystem mode reads .md files straight from the vault folder. Simple, no database, but the machine has to be on. CouchDB mode reads and writes the database that LiveSync syncs to, which is what lets the vault answer with every one of your devices off.

CouchDB mode is the one that fixed my problem, and it’s the harder one to get right. LiveSync doesn’t store plain Markdown. It stores documents split into content-addressed chunks, optionally encrypted, optionally with obfuscated file paths. To read or write a note that the plugin will accept, you have to speak that format exactly. Rather than reverse-engineer it, the server imports livesync-commonlib, the same library that powers the plugin. What the MCP server writes, the plugin reads, because underneath they’re running the same code.

Both modes expose the same MCP tools over HTTP, so any MCP-compatible agent connects: Claude on desktop, web, or mobile, Copilot, custom agents, anything that speaks the protocol.

I run mine on Fly.io, the MCP server and CouchDB sharing one small box. It’s less than $5/month, which Fly.io doesn’t collect as of today. Getting there is one command: a setup script that deploys either the MCP server on its own, next to a CouchDB you already have, or CouchDB and the MCP server together from scratch. It generates the credentials and wires them up as it goes, so the whole thing is a few prompts rather than an afternoon of config.

Once it’s connected, the agent sees the vault as a set of tools.

That “needs approval” column matters: every read and every write waits for me to say yes. It’s the first line of defense for handing an agent the keys to your notes.

How I actually use it

Most days it starts on my phone. A half-formed thought goes into the chat while it’s still fresh, the agent and I knock it into shape, and only the cleaned-up version gets written to a note. The messy thinking stays in the conversation. The vault gets the result, waiting in Obsidian the next time I open the laptop, which never had to be on for any of it.

The same move works for anything worth keeping. A technical back-and-forth, a debugging session, a decision I argued myself through: instead of letting it scroll out of a chat window forever, I have the agent write it into a note in a clean, readable format. That quietly turns the vault into shared memory between agents. A thread I had with Claude on the web becomes context I can hand to Claude in VS Code, not by replaying the session but by pointing the next agent at the note. cc-tap bridges live sessions; this is the structural version, where the durable artifact does the handoff and survives every session that touched it.

Books are their own habit, and the richest use of all of this. I read, then I take the book to Claude. Sometimes it’s just a quote worth keeping. More often it turns into a real conversation: a thread I want to pull on, a concept I dig into until it opens onto other writers, the history underneath it, the social questions it runs into. That thinking is worth more than the highlights, so the whole thread gets distilled into notes and tagged as I go. Months later I can ask what I took from a given author, or where a particular idea led me, and it’s there, in their words and mine.

And I stopped doing the bookkeeping. The agent picks the tags and wires up the [[links]], so notes connect to each other without me stopping to think about taxonomy. Tagging and linking was the part of Obsidian I always let rot, the chore that quietly kills a vault. Handing it to the agent is what finally made the graph worth having.

That graph is one example: my entire reading history from Libby, pulled in and filed by the agent through the server. Every book became a note, tagged and linked into the web, and I didn’t place a single connection by hand.

Two settings make me trust it with all of this. MCP_INSTRUCTIONS bakes my vault’s conventions into the server itself: folder structure, naming rules, where daily notes go, which tags exist. Every client follows them without per-app config, and the agent stops guessing where things belong. READ_ONLY exposes only the read tools, so an agent can lean on my vault as context without ever touching it. Both of those started as requests from other people using it, which was the first sign I wasn’t the only one with this gap.

What it is, and what it isn’t

It’s a tool I use every day that other people now use too. It isn’t Obsidian Sync with a support desk. A few honest edges:

• CouchDB mode rides on LiveSync, which is powerful but rough around the edges. The initial setup takes some patience, and the plugin gives you a lot of knobs to get right. The saving grace is that it’s a one-time job: once it’s running, you stop thinking about it.
• One vault per instance. Multiple vaults means multiple servers on different ports.
• Last write wins. If an agent and Obsidian edit the same note at the same instant, there’s no conflict resolution yet. Issue #2 tracks adding _rev checks for that.
• Text only. Binary attachments aren’t exposed.
• You’re giving an agent write access to your second brain. Keep backups, set MCP_AUTH_TOKEN, use tool approval deliberately, and reach for READ_ONLY when you don’t need writes.

One blunt warning before you point this at a vault you care about. Most of the heavy lifting here is LiveSync’s, and LiveSync can go sideways, especially once E2E encryption is in the mix. A mismatched passphrase or obfuscation setting can fail to sync or mangle notes without a single error message. On top of that, an agent with write access can delete or overwrite the wrong thing in one bad call. So: back up your vault before you connect any of this, keep backing it up, test against a throwaway vault first, and run READ_ONLY whenever you only need reads. This is MIT software provided as-is, with no warranty. I take no responsibility for lost or corrupted notes. Your vault, your risk.

Closing thought

The MCP server is the small part. The real shift is in how we use a second brain at all. It used to be a place you filed things by hand and came back to read. Now we think out loud with an AI, and the worthwhile parts should just settle into the vault on their own. The strange thing is how little had to be built for that. The notes were already in an open database. The agents already spoke a protocol for tools. Nobody had wired the two together.

I connected them because I wanted to drop an idea into a note from a bus and have Claude file it properly. It’s a small project, but it turns out I wasn’t the only one who wanted their notes within reach of an AI, and the handful of people using it have already made it better than I would have alone.

Links

• Obsidian Sync MCP (source): GitHub
• Self-hosted LiveSync: GitHub
• Model Context Protocol: modelcontextprotocol.io

In the previous article, I benchmarked Apple’s on-device 3B model for shell command generation and got it to ~80% accuracy using retrieval (dynamic few-shot examples from a 21k-entry bank). That was the ceiling I could reach without touching the model’s weights, or growing the bank size.

Apple’s FoundationModels framework also supports custom adapters. You can train LoRA weights, export them as .fmadapter files, and load them at inference time. The model stays frozen; you’re adding a small set of correction matrices that steer its outputs. This is the next logical step — not because I plan to ship adapters (distributing ~160MB adapter files for a lightweight CLI isn’t practical), but to understand the training experience: what does Apple’s toolkit look like, what does it take to get training running on accessible hardware, and do the resulting adapters actually work?

The short version: the toolkit assumes a beefy GPU, but with some work I got training running on a free Colab T4 and eventually on a Mac. The adapters work — and the training method and platform don’t matter much. Along the way I found a caching bug in Apple’s inference service that silently consumed 269GB of disk space.

Disclaimer (April 2026): There is a known bug where loading adapters from a command-line tool leaks ~160MB per invocation to a SIP-protected cache with no cleanup. Apple has confirmed this is specific to CLI tools — app bundles are not affected. I detail the bug below. The shipped version of hunch does not use adapters, so this doesn’t affect normal usage.

A quick primer on LoRA

If you’re not familiar with how adapter training works, here’s the short version.

Transformers are the architecture behind modern LLMs. They process text through layers of attention (deciding which parts of the input to focus on) and feed-forward networks (transforming representations). Each layer has large weight matrices, millions of parameters that encode what the model has learned.

Fine-tuning means updating those weights on new data so the model learns new behavior. Full fine-tuning updates every parameter, which for a 3B model means moving ~12GB of weights through the optimizer. It’s expensive in compute, memory, and storage.

LoRA (Low-Rank Adaptation) is the shortcut. Instead of updating the full weight matrices, you freeze the original model and attach small “correction” matrices to specific layers. If a weight matrix is 2048×2048 (~4M parameters), LoRA decomposes the update into two small matrices: 2048×32 and 32×2048 (~131K parameters). The rank (32 in this case) controls how much capacity the adapter has. You train only these small matrices. Across all 56 transformer layers in Apple’s 3B model, that works out to ~66M trainable parameters, about 2% of the full model. The result is a ~160MB adapter file instead of a 12GB model copy.

QLoRA goes one step further. It quantizes the frozen base model to 4-bit precision (NF4 format), compressing it from ~12GB to ~2GB in GPU memory. The LoRA matrices themselves stay in higher precision for training stability. The practical effect: you can train on GPUs with much less memory, including free-tier ones.

Apple’s toolkit and the hardware wall

Apple ships an Adapter Training Toolkit, a Python package (~12GB including base model weights) that handles data loading, LoRA configuration, training loops, and export to the .fmadapter format. The official requirements are “Mac with Apple silicon and at least 32GB memory, or Linux GPU machines.”

In practice, the memory requirements are steep. The base model is 12GB in fp32. The toolkit loads it without memory mapping, so during loading the checkpoint and the model parameters both live in system RAM — ~24GB peak. The GPU side needs ~15GB for the training loop. But the system RAM spike is the real bottleneck: a Colab T4 has 16GB VRAM but only 12GB system RAM, so training OOMs before it even reaches the GPU. I used an A100 (80GB system RAM) where this isn’t an issue. A 32GB Mac might technically fit, but it’s tight. My 24GB MacBook Air M4 OOM’d immediately.

The memory breakdown is straightforward once you look at the loading pipeline, and each bottleneck has a fix. Getting it to run on a free Colab T4 and eventually on a Mac took three iterations.

Getting training onto a free T4

A Colab T4 has 16GB VRAM and only 12GB system RAM. The ~15GB training footprint barely fits on the 16GB GPU, but the ~24GB loading spike doesn’t fit in system RAM — training OOMs before it starts. My first attempt was patching Apple’s pipeline directly. Three changes:

1. Memory-mapped loading. Replace torch.load with mmap=True to read weights from disk on demand instead of materializing 12GB in RAM. System RAM drops from ~24GB to ~6GB.
2. fp16 model creation. Force model_config.dtype = torch.float16 so the model takes 6GB on GPU instead of 12GB, with adapter weights cast back to fp32 for gradient scaling.
3. Gradient scaling and dtype fixes. Apple’s training script only enables GradScaler for a precision mode that isn’t exposed as a CLI option. Without it, fp16 gradients overflow to NaN. A separate fix in rms_norm casts weight tensors to match input dtype in mixed precision.

With all three patches, fp16 LoRA trains on a T4: ~8.5GB GPU, ~6GB RAM. It works, but it’s three patches to Apple’s code and still uses over half the T4’s memory.

QLoRA is cleaner. It quantizes the frozen base model’s Linear layers to 4-bit NF4 via bitsandbytes, dropping the base model to ~2GB and total training footprint to ~5GB. The rms_norm dtype patch is still needed (mixed fp16/fp32/quantized tensors flow through norm layers), but only one patch instead of three. I wrote a custom training script around it since the toolkit doesn’t support quantized training natively.

Getting training onto a Mac

Standard LoRA OOM’d on my 24GB Mac — the ~24GB RAM peak during model loading leaves nothing for training. QLoRA with mmap loading fits comfortably at ~5GB GPU, but trains ~9x slower than a free Colab T4. The bottleneck: bitsandbytes has no native MPS kernels in its current release (0.49.2), so NF4 dequantization falls back to CPU.

bitsandbytes recently merged native Metal kernel support into main, expected in v0.50.0. Installing from git (pip install git+https://github.com/bitsandbytes-foundation/bitsandbytes.git) brings lower GPU memory usage and ~2x faster training on the short override dataset that produces the best adapters, cutting the T4 gap from ~9x to ~4x. Slower, but fully local — no cloud, no uploads, no Colab session management. And the adapter quality is identical.

Do the adapters actually work?

With the training infrastructure in place, I needed to verify the adapters actually improve the model. I trained on two datasets: the full 21k tldr bank, and a small set of curated overrides (~96 total, reduced to ~57 after excluding benchmark prompts to avoid data leakage) I’d hand-written to fill gaps in the model’s knowledge (macOS-specific commands, flag corrections, common mistakes).

The full bank made things worse: the adapter learned noise from 4,500 different commands and interfered with retrieval. The overrides worked. Not a surprising result (targeted data beats noisy data), but it confirmed the pipeline produces usable adapters.

The important comparisons for validating the training pipeline:

Adapters work. Standalone, they lift the bare model from ~41% to ~74-76%. Combined with retrieval, accuracy pushes into the ~85-86% range. Retrieval alone averages ~80% (with variance up to ~83% across runs), so the gain is modest but consistent across all training methods and platforms.

LoRA ≈ QLoRA ≈ Mac. All three paths land in the same range: ~85-86% with retrieval, ~74-76% standalone. QLoRA on a free T4 produces the same quality adapter as full LoRA on a paid A100, and Mac-trained adapters match both.

Pushing accuracy higher wasn’t the goal here. The point was to verify that adapters trained across different methods (LoRA, QLoRA) and platforms (A100, T4, Mac) all produce equivalent results, and that the cheaper paths don’t sacrifice quality.

The bug: invisible disk consumption

While running adapter benchmarks, my 500GB SSD dropped to 10GB free, but du showed only ~230GB used. The missing space was invisible to every standard macOS tool, even sudo.

The cause: each call to SystemLanguageModel.Adapter(fileURL:) triggers Apple’s inference service to write a full ~160MB copy of the adapter to a SIP-protected cache directory (/private/var/db/AppleIntelligencePlatform/AppModelAssets/). Each copy gets a unique hash and there’s no cleanup. Over ~300 benchmark runs: 1,684 cached copies, ~269GB. Only visible from Recovery Mode (which bypasses SIP).

Apple has confirmed the bug is specific to command-line tools — app bundles are not affected. To reclaim space, boot Recovery Mode and delete the cache directory contents.

Where this leaves us

The toolkit works. You can go from dataset to working on-device adapter in an afternoon. But the experience is firmly an ML engineer’s workflow: Python scripts, PyTorch, bring-your-own-GPU, debug your own dtype mismatches. If you’ve fine-tuned models before, you’ll feel at home. If you’re an iOS developer who’s never touched a training loop, there’s a steep ramp.

Apple has a track record of eventually abstracting these workflows. Create ML turned image classification and object detection training into a drag-and-drop Xcode experience. It’s easy to imagine a similar path here — a “Create LLM” that wraps the toolkit into something Xcode-native, with managed training on Apple silicon and one-click export to .fmadapter. The pieces are all there; the developer experience just hasn’t caught up yet.

I’m not shipping adapters with hunch; ~160MB per adapter doesn’t fit a lightweight CLI, independently from the bug. But the training pipeline, patches, and notebooks are in the hunch repo with a training guide if you want to try it yourself:

• LoRA notebook — end-to-end pipeline around Apple’s toolkit (needs a 24GB+ GPU)
• fp16 LoRA notebook — same pipeline with three patches to run on a free Colab T4
• QLoRA notebook + training script — 4-bit quantized training via bitsandbytes, runs on a free T4 or locally on Mac
• MPS setup instructions — installing bitsandbytes from main with native Metal kernels
• Bug workaround — batch mode for the adapter caching leak, and Recovery Mode cleanup

I use Claude Desktop for brainstorming: exploring ideas, thinking through architecture, working out strategy. I use Claude Code for implementation: planning in the codebase, writing and testing code, running builds. Different modes of thinking, different tools. Like wanting a whiteboard for one task and a terminal for another.

The problem is they can’t see each other.

A brainstorming session on Desktop might surface a design idea that I want the Code session to act on. Or a design question on Desktop needs to know how something was actually implemented but the answer lives in the Code session’s context. The information flows both ways: Desktop pushes ideas down, and needs to pull implementation details up.

But there’s no way to do either. I copy-paste fragments, re-explain things, re-establish context that already exists in another window. Two AI agents, both with deep context on my project, completely unaware of each other.

Anthropic recently introduced Remote Control, a way to access your local CC session from the web or mobile. It solves a real problem: your laptop’s session, reachable from your phone. But it’s the same you, driving the same session, from a different screen. One agent, two windows.

What I want is different. I want the Desktop conversation (a separate agent, with its own context) to reach into a running Code session, see what’s there, and send it a message. Not multi-device single-user. Multi-agent. That axis isn’t covered.

So I mapped the session protocol that Claude Code’s web UI uses, and built an experimental MCP server that lets Claude Desktop read and interact with running Code sessions. It uses undocumented APIs that could change at any time, but it works today and the protocol itself is the interesting part.

The API is already there

Both Remote Control and Claude Code on the web use the same session protocol under the hood. I captured the browser traffic with HAR exports and mapped the endpoints. The protocol has three layers:

HTTP REST for session management. List sessions, read event history, send messages. Uses your existing OAuth token against api.anthropic.com. This is the workhorse: straightforward REST, well-structured responses, and it works from any HTTP client.

WebSocket for real-time streaming and tool approval. The web UI at claude.ai/code uses a WebSocket to receive events as they happen and to relay tool approvals back to the session. This is behind Cloudflare’s bot protection: TLS fingerprinting, JS challenges, browser verification. A non-browser client gets a 403.

Polling as a fallback. Without WebSocket access, an external client can poll the event history endpoint every 1-2 seconds. You get ~1.5s latency instead of real-time, but it works with just an OAuth token.

The session model already supports multiple clients connecting to the same session. That’s not an accident. It’s how the web UI works.

The full protocol is documented in PROTOCOL.md in the repo.

The build

cc-tap is an MCP server that exposes Claude Code sessions as tools. Install it in Claude Desktop or another Claude Code session, and you can see into and interact with running CC sessions.

pip install cc-tap
claude mcp add cc-tap -- cc_tap

Requires claude /login and Remote Control enabled.

Six tools:

With this, a Claude Desktop session can ask “what are my running Code sessions?” and get a list. It can read what a Code session has been working on. It can send a message to a Code session (“run the test suite,” “what’s the status of the refactor?”) and get back the response. A second Code session can coordinate with a first one.

It uses the same OAuth credentials you already have from claude /login. No additional auth, no API keys. cc-tap reads them from your macOS Keychain (or ~/.claude/.credentials.json on Linux) and talks to the same API the web UI does.

Where it stops working

It mostly works. But there’s one place where it doesn’t, and that’s the most interesting part.

When a Claude Code session needs to run a tool (read a file, execute a command, make an edit) it asks for permission. In the terminal, you approve it by pressing Enter. In the web UI, you click a button. That approval travels to the session runtime via WebSocket.

cc-tap can see these pending tool requests through polling. But it can’t approve them.

The reason isn’t just that the WebSocket at claude.ai is behind Cloudflare’s bot protection (though it is — a non-browser client gets a 403). The deeper issue is architectural: control responses sent via HTTP POST are stored in event history but never relayed to the session worker. The worker only picks them up from an active WebSocket connection. So even if you got past Cloudflare, the HTTP path wouldn’t work for approvals. The control channel is WebSocket-shaped by design.

There are WebSocket endpoints on api.anthropic.com too, but they appear to require a session-scoped ingress token issued through a worker registration flow. I tested three different endpoint/auth combinations, all returned 403. The details are in the Open Questions section of the protocol doc.

The practical effect: the read path (listing sessions, reading events, sending messages) works over plain HTTP with OAuth. The control path (approving tool use) is WebSocket-only, and those endpoints aren’t reachable from external clients.

What this is — and isn’t

This is an exploration, not a production tool. The API is undocumented and could change tomorrow. Polling adds ~1.5 seconds of latency. Tool approval requires the CC terminal or the web UI. The protocol documentation is reverse-engineered from one person’s traffic captures — there’s almost certainly more surface area I haven’t found.

Things worth exploring further:

• Real-time streaming. The WebSocket endpoints exist for real-time event delivery. Getting access from a non-browser client would eliminate the polling latency.
• Structured coordination patterns. Right now cc-tap is a bridge: read and write. But the primitives are there for higher-level patterns: one session delegating subtasks to another, a Desktop session acting as a supervisor for multiple Code sessions, or an orchestrator that routes work based on session state.
• Tool approval relay. The biggest limitation. If the WebSocket becomes accessible, or if Anthropic exposes a REST endpoint for control responses, cc-tap could go from observer to full participant.

The value right now is the proof that it works and the protocol documentation. Both show that multi-client session access isn’t hypothetical. It’s implemented, just not exposed as a platform primitive.

Closing thought

Anthropic built MCP, an open protocol so that any agent can talk to any tool. They also built multi-client session infrastructure so that their web UI, mobile app, and CLI can all attach to the same running session.

What they haven’t done is connect these two things. There’s no MCP surface for session-level operations. No way for one Claude instance to discover or interact with another through the protocol Anthropic designed for exactly that kind of interoperability.

cc-tap bridges the gap from the outside, using the same session API the web UI uses, exposed as MCP tools. It works because the pieces are already there. The session model supports multiple clients. The event stream is readable over HTTP. Messages can be injected by any authenticated client. The architecture is ready. It just hasn’t been unbundled.

If anyone at Anthropic is thinking about exposing CC sessions through MCP, the tool list in cc-tap is one shape that surface could take.

Links

• cc-tap (source): GitHub
• cc-tap (package): PyPI
• Protocol documentation: PROTOCOL.md

Apple quietly shipped a 3B-parameter LLM on every Mac running macOS Tahoe. It sits on the Neural Engine, powers Siri and Writing Tools, and is accessible through the FoundationModels Swift framework, but there’s no direct way to use it outside of building a custom app.

apfel changed that. A Swift CLI that wraps FoundationModels and gives the on-device model a command-line interface. It hit the front page of Hacker News and proved the model was usable from the terminal. That got me thinking: what if I built something more specialized?

macOS’s default shell, zsh, has hooks that fire before you run a command, when a command isn’t found, and after a command fails. They’ve always been there, but the options were either deterministic (Levenshtein-based “did you mean”), or an LLM that’s either too slow locally or requires a cloud roundtrip. A 3B model on the Neural Engine responding in under a second changes the tradeoff.

I wired all three hooks to the on-device model. It worked for simple things, then started hallucinating flags. So I benchmarked multiple approaches across 100 prompts to find out what actually helps, built the winning approach into a CLI called hunch, and learned something about where the 3B wall actually is.

The three hooks

zsh has three built-in extension points that fire at specific moments in the command lifecycle. Each one is a natural place to put an LLM call:

The Ctrl+G hook is the main one. Type a description, hit Ctrl+G, the buffer gets replaced:

The key safety property: Ctrl+G never runs anything. It fills the buffer. You always read before you execute.

The 40% baseline

Simple commands work. Typo correction is reliable. But the moment you ask for anything with specific flags, the model hallucinates.

find files changed in the last hour → find . -mtime +1h

This is wrong in three ways (-mtime counts days, + means “more than,” the h suffix doesn’t exist). The correct command is find . -mmin -60.

I ran 100 prompts (31 simple, 51 flag-heavy, 18 composed) and scored each result. Baseline: 40% usable. 60% wrong.

The model is also oblivious to macOS. “Show my IP address” returns ip a, a Linux command that doesn’t exist on macOS. It doesn’t know pbcopy, caffeinate, mdfind, pmset, or sips. It reaches for systemctl, lsusb, iwconfig every time.

Some hallucinations are dangerous. Asked for a soft git reset, it generated git reset --hard HEAD~1, which destroys uncommitted changes. This is why Ctrl+G never executes anything.

What I tried

I tried a few approaches to improve accuracy. They fall into four categories.

Help the model reason better. Give it reference material so it can look up the right flags. I tried parsing man pages into flag indexes, fetching tldr pages as documentation, grepping man pages for relevant keywords, and hardcoding cheat sheets in the system prompt. The model sees the right flag in the docs but can’t reason about it and apply it.

Let the model self-correct. Self-critique (“is this correct for macOS? if not, fix it”) and Apple’s @Generable constrained decoding both failed. The model “fixes” correct commands into wrong ones. Self-consistency (majority vote) didn’t help on its own either, though it becomes useful later when combined with the example bank.

Give it solved problems to copy. Instead of documentation, inject Q/A pairs the model can pattern-match against, like “find files larger than 100MB” → find . -size +100M. Static few-shot uses 8 fixed examples. Dynamic few-shot picks the 8 most similar from a bank using token-overlap similarity. Accuracy depends on bank size. I went back to tldr, this time using it as solved examples instead of documentation: 20k+ community-written Q/A pairs for ~3k commands, parsed into a SQLite FTS5 index. That alone got accuracy to ~68%. Adding targeted macOS overrides, tiered retrieval (overrides → macOS-specific → common), a tuned prompt, and lightweight command validation pushed it to ~83%.

Add sampling diversity. At temperature 0 the model is mostly deterministic, but not fully: accuracy swings 64–76% across runs even with the same prompts. Without the example bank, raising temperature to 0.3 with 3 samples actually makes things worse (39% vs 41%). On top of the example bank, self-consistency kills the variance: 67–69% range (2pp) instead of 64–76% (12pp). Same average, but you know what you’re getting. For a tool you use dozens of times a day, predictability matters. Tradeoff: 1.3s instead of 0.4s.

The full benchmark suite (100 prompts, all approaches, raw results) is in the hunch repo.

The 3B wall

The benchmark results tell a specific story about what this model can and can’t do.

It can’t reason from documentation. I gave it man page flag indexes, tldr pages as context, cheat sheets, targeted doc sections. It finds the right flag name (it sees -mmin) but outputs -mmin 1 instead of -mmin -60. The +n/-n/n semantics are beyond what it can derive from a description. Man page keyword grep fails too: man pages don’t use words like “hour” or “changed,” so naive search returns nothing. None of these approaches beat a bare prompt.

It can’t self-correct. Self-critique dropped accuracy to 33%: worse than baseline. The model “fixes” correct commands into wrong ones. Chain-of-thought is even more revealing: forced to show its reasoning, the model confabulates. When it outputs makepasswd instead of openssl rand, the reasoning field says “makepasswd is a macOS-specific command.” It isn’t. The 3B doesn’t reason through CoT: it invents plausible justifications for wrong answers.

It can copy patterns. Give it "find . -mmin -60 finds files in the last hour" as a literal example and it outputs correctly, because it copies, not reasons. That’s why few-shot examples work and documentation doesn’t. Self-consistency doesn’t improve accuracy, it improves reliability. Without examples, sampling explores variations around a wrong center. With examples, it narrows the spread around a roughly-right center. Same average, less variance.

The precise claim: Apple’s 3B on-device model can classify intent and copy patterns but cannot reason over documentation to derive correct usage. The model knows what tool to reach for but doesn’t know how to hold it.

The meta-lesson: the right question for a small on-device model isn’t “can it do this task?” but “can I decompose the task so the model only does the parts it’s strong at?”

hunch

The dynshot-tldr approach worked well enough that I built it into hunch, a Swift CLI that calls FoundationModels directly (no apfel dependency), with FTS5 search and the tldr bank baked into a single binary.

brew install es617/tap/hunch
source ~/.local/share/hunch/hunch.zsh  # add to ~/.zshrc

~83% accuracy in 0.5s. In practice, the basics are reliable: simple commands, macOS-specific tools (pbcopy, caffeinate, pmset), git operations, network diagnostics, file operations. The example bank is extensible: you can add your own overrides for tools or workflows the community examples don’t cover. Temperature and sample count are configurable via CLI flags or environment variables. Everything runs on the Neural Engine. No cloud, no API keys, no data leaves your Mac.

A couple of things to know: FoundationModels is Tahoe only (macOS 26, Sequoia and earlier don’t have it), and Apple’s guardrails are inconsistent. kill whatever is using port 3000 returns empty because the word “kill” triggers the safety filter. hunch uses permissiveContentTransformations to avoid false positives.

What this means

The model is bad at generating syntax but good at classifying intent and picking from options. That’s exactly what tool-calling requires, and FoundationModels supports it.

A 3B model that picks the right tool, passes the right arguments, and summarizes the result is more useful than one that tries to generate correct code from scratch. The right question for a small on-device model isn’t “can it do this task?” but “can I decompose the task so the model only does the parts it’s strong at?”

And it all runs locally. No roundtrip, no API key, no data leaving the machine. For anything latency-sensitive or privacy-sensitive, that matters. Every Tahoe Mac already has this capability sitting idle on the Neural Engine. hunch is one way to use it. There will be others.

This post is part of the Let the AI Out series on giving AI agents direct access to hardware. Start here for the overview.

Most AI integrations today are conversational. Connect an LLM to a database, an API, a sensor, and you can ask questions in natural language. “What’s the temperature?” “Any anomalies?” It’s a better interface than a dashboard, but the model is the same: you ask, it answers, nothing happens until you ask again.

That’s the presentation layer.

But what happens when the system stops waiting for you, observes and acts on its own?

That’s the shift to a control layer.

This post is about what that looks like in practice. I gave AI agents access to real BLE hardware through a BLE MCP server and asked them to build and operate a monitoring system.

The demo focuses on scanning and alerting, but the same tools support reading sensors, writing commands, and controlling devices directly. The pattern applies anywhere there’s hardware generating data — BLE devices, industrial sensors, fleet trackers.

I tried two approaches.

• First, I architected a two-agent system myself using fast-agent — it worked, but I had to design everything.
• Then I gave Paperclip a single prompt and let it hire its own team. One prompt, zero code, 4 active agents, 44 devices tracked, 198 alerts — and an Analyst agent that started exercising real judgment: tracking missing devices across review cycles, flagging signal anomalies, and recommending fixes to its own monitoring system.

Approach 1: Architect the agents yourself

Most AI tools today are built for interaction - they’re driven by a human at the keyboard. They don’t run unattended (yet), and they don’t support MCP notifications (still!), so they can’t react to hardware events on their own.

A control layer needs to run continuously. So I built one — two agents, clear roles, shared state:

This was built with fast-agent, a framework for wiring agents and MCP together. Each agent is defined with a system prompt that describes its role, what MCP servers it can access, and how it should behave. I wrote detailed instructions for both: the controller knows how to create plugins, verify they work, fix bugs, and manage rules in SQLite. The user agent knows to delegate control tasks and only query data directly. That’s the architecture work: defining who does what, how they interact, and what each agent is allowed to touch.

One prompt:

“Scan for BLE devices every 30 seconds and log everything found.”

What happened: the user agent stored the rule in SQLite and delegated to the controller. The controller read the rules table, created a devices table, wrote a custom scanner plugin with a background asyncio loop, loaded it, and verified it was working. All autonomously. Zero code written by hand.

The plugin runs inside the BLE server process at zero token cost. The controller gets called only when something needs judgment.

A second prompt pushed it further:

“Monitor 3 devices. Track when they appear and disappear. If any goes missing, check the time of day before alerting — not everything is an emergency.”

The controller wrote a 400-line plugin with absence windows and reliability scoring. The plugin tracked each device’s history: how often it appeared, how long it typically stayed, when it usually went offline. “95% reliability” meant the iPhone had been consistently present in 95 out of 100 scans. When it disappeared for 6 minutes, at a time it’s not usually absent, the system flagged it as a real alert, not noise. A 6-minute gap for a device that comes and goes would have been ignored. The agent encoded that distinction without being told to.

This worked. Agents can react to hardware and build their own monitoring logic. But it was still designed. I defined every piece: two agents, their roles, their MCP connections, their interaction pattern.

Approach 2: Define the goal, not the agents

The fast-agent approach worked, but I had to design the whole thing: agents, roles, prompts, wiring. What if I just described the problem and let the system figure out the rest?

Operate a BLE device monitoring system. Scan, track, analyze devices, and alert on notable events.

I used Paperclip for this - a multi-agent platform where you describe a company goal and a CEO agent hires agents and coordinates the work. It gives you a dashboard with full auditability and control: who’s working on what, task status, agent conversations, hiring decisions.

I used Claude Code agents under the hood, orchestrated by Paperclip. Simple setup, and cheaper. But no MCP notifications here, so the agents worked around it by writing plugins with background tasks.

The BLE MCP server was running on HTTP with plugins enabled. A SQLite MCP server was available for persistence. Both accessible from Paperclip running on my machine.

One prompt

I created a company with one goal:

“Operate a BLE device monitoring system. Scan, track, and analyze BLE devices. Create and maintain monitoring plugins. Execute monitoring rules and alert on notable events.”

And one task for the manager:

“The BLE MCP server is running at http://localhost:8001/mcp. Plugins are enabled. A SQLite database server is also available. For continuous monitoring, create BLE plugins with background tasks. Hire your team and start executing the company goal.”

~90 minutes later: a team of 4 agents, 7 completed tasks, one continuous monitoring task always running — 46 scans, 44 unique devices discovered, 198 alerts generated. Zero lines of code written by a human.

The Engineer

The Manager’s first move was hiring a BLE Engineer.

The Engineer built the entire monitoring infrastructure in about 20 minutes:

A database schema — three tables (devices, scan_history, alerts), properly normalized with indexes for query performance.

A scanner plugin (ble_scanner.py) — background asyncio task that scans every 60 seconds, upserts the devices table, logs scan history, sends notifications when new devices appear. The plugin runs inside the BLE server process. Autonomous, zero token cost.

An alerter plugin (ble_alerter.py) — background task evaluating four rules every 30 seconds: new device detection, device disappearance, RSSI threshold, and device count spike. Deduplication, acknowledgment flow, alerts persisted to SQLite.

Both plugins use the BLE server’s background task system. The Engineer studied the plugin template and applied the patterns. No human wrote any code.

QA and the Analyst

Once the infrastructure was running, I gave the Manager a second task:

“We need someone to validate the BLE engineer’s work and someone to monitor alerts every 2 minutes.”

The Manager hired two more agents: a QA engineer and an Analyst.

QA validated the full stack: database schema, scanner plugin (35 scans, 41 devices tracked), alerter plugin (335 rule evaluations, 76 alerts triggered, all 4 rules firing), and MCP server health. Everything passed.

The Analyst used judgment, not rules. Across four review cycles, it:

• Tracked devices across cycles (Gatto missing for 4 consecutive reviews — no worries, he was just upstairs)
• Differentiated transient disappearances from concerning ones
• Escalated based on device type (pet tracker vs. unknown)
• Flagged infrastructure anomalies (lock signal degradation)
• Recommended improvements to its own monitoring system

Cycle 1: 52 alerts. Baseline. The Analyst noticed 14 devices disappeared simultaneously and correctly identified it as a scanner outage. (I’d paused for dinner.)

Cycle 2: 23 alerts. Flags Gatto (pet tracker) and Keys ES1 (key finder) as “watch next cycle.”

Cycle 3: 28 alerts. Gatto and Keys ES1 absent for 3 consecutive cycles. Analyst escalates: “These are known persistent devices. Their prolonged absence may indicate a real-world concern — out of range, powered off, or lost.”

Cycle 4: 32 alerts. Gatto absent for 4th cycle. Analyst also flags Schlage lock weak signal (-91 dBm): “Fixed infrastructure device. Weak signal is unusual — could indicate interference, low battery, or physical obstruction.”

The Analyst also recommended system improvements: “The rule engine continues to re-fire… Recommend reviewing the disappearance rule’s re-alert logic…”

Safety and guardrails

I didn’t set any guardrails for this demo. No rate limits, no alert suppression, no noise filtering. The Analyst figured out on its own that the alerter plugin was re-firing the same alerts every cycle, called it noise, and recommended fixing the re-alert logic.

A guardrail I never wrote, discovered by an agent that was just doing its job.

It also points at a real question: an agent that controls physical systems needs constraints. Someone will ask — “didn’t you just replace one set of rules (the automation) with another (the constraints)?”

Not quite. The rules moved up a level. The old rules were implementation-level: if device_count > threshold, then alert; specific instructions for a specific scenario. Guardrails are intent-level: don’t flood the operator with noise, a principle the agent applies across every scenario it encounters.

In practice you’d want both — hard limits on the control surface (rate limits, write opt-in, destructive action confirmation) and intent-level guidance that the agent reasons about.

This is a version of what I called the governor module problem in an earlier post. Every time we’ve delegated more to machines (compilers, cloud infrastructure, CI/CD), the control plane moved up a layer. We didn’t lose control. We moved where it lives.

That’s still unsolved for physical control. But the fact that the Analyst independently identified a noise problem and recommended a fix — without being told to — suggests the capability is already emerging. The question is whether we can build infrastructure robust enough to trust it.

Closing thought

This post is about a different kind of problem. Not “help me build this” but “watch this and tell me when something matters.” The agent isn’t assisting a human at a terminal. It’s running autonomously, making judgment calls about the physical world.

The pattern isn’t specific to BLE. Anywhere you have sensors generating data and dashboards displaying it, there’s room for an agent that actually does something about it.

Nobody should rip out their working automations for this — today. Rules handle known scenarios well. They’re fast, cheap, and predictable. But the world isn’t all known scenarios. The question isn’t whether agents can replace rules. It’s what becomes possible when the automation layer can handle situations nobody thought to write a rule for. And as models get faster, cheaper, and more trustworthy — maybe the rules do go away, and all that’s left is intent and guardrails.

But for now, the gap between “imagine if…” and “let me show you” is small enough to close in a weekend project.

Links

• ble-mcp-server: GitHub · PyPI
• ble-agent-gateway (Approach 1): GitHub
• fast-agent: GitHub
• Paperclip: GitHub
• MCP (Model Context Protocol): modelcontextprotocol.io

This post is part of the Let the AI Out series on giving AI agents direct access to hardware. Start here for the overview.

An ML engineer can train a keyword detection model in an afternoon. Deploying it on a microcontroller — Zephyr RTOS, CMSIS-NN kernels, memory alignment, tensor arena sizing, fixed-point FFT — takes weeks and a completely different skillset.

This post is about what happened when an AI agent with debug probe access was pointed at that problem. One Claude Code terminal session. 90 minutes. No code written by hand. No hardware physically touched. Never left the terminal. The result: 98ms end-to-end latency and 94.6% accuracy on real-world recordings.

This is a toy example — a known model, a known training sample. But the workflow is real. And what it demonstrates about the speed of iteration is the point.

Getting from zero to a working deployment in hours lets you focus your expert attention on the hard part — quantization tuning, power optimization, production robustness. The first 70% shouldn’t take weeks.

What this post covers

This post builds directly on the debug probe MCP server. If you haven’t read that one, the short version: the agent can flash firmware, halt the CPU, set breakpoints, read registers and memory — all through a J-Link debug probe over SWD/JTAG.

Here, the agent uses that foundation to deploy and iterate on an edge AI model. Along the way, it builds two custom plugins that turn the debug probe into an edge-AI development environment.

The embedded session replays show the actual agent sessions — edited for length, but with real tool calls and responses.

This is written for ML engineers who want to test models on real hardware — and for embedded engineers looking to accelerate their development workflow with agents.

The setup

The model is micro_speech — a TFLite Micro keyword spotting model from the TensorFlow examples. 18.8KB, int8 quantized, four classes: yes, no, silence, unknown. Designed specifically for microcontrollers.

The target is a Nordic nRF52840 development kit — ARM Cortex-M4F at 64 MHz, 1MB flash, 256KB RAM, running Zephyr RTOS. A common board, well-supported by Zephyr and CMSIS.

The agent is Claude Code with the debug probe MCP server installed — though any MCP-capable agent would work.

The starting point is Nordic’s Peripheral UART sample — a clean Zephyr project that exposes a UART interface over both serial and BLE. TFLite Micro was added on top of this existing firmware.

End-to-end inference looks like this: one second of audio goes in, preprocessing extracts MFCC features (49 frames × 40 coefficients, the spectral fingerprint of the audio), those features feed the neural network, and the network outputs confidence scores for each class.

For simplicity, there is no microphone or PDM/I2S input in this setup. Audio is injected through the debug probe — the agent writes raw PCM data directly into the device’s memory and triggers the on-device preprocessing and inference pipeline.

The session

The entire deployment happened in a single Claude Code session, roughly 90 minutes. No code written by hand, no hardware physically touched, never left the terminal. A few steering interventions made the difference — step back, simplify, and redirect when the agent was overcomplicating things. No predefined instructions were used — no CLAUDE.md, no custom skills. With them, the agent can skip the discovery phase and go straight to building.

1. Initial setup & debugging (~24 min)

The agent set up the project, built and flashed firmware, hit a hard fault, and worked around it by disabling optimized kernels. First inference: 865ms — slow but correct. It then built a custom debug probe plugin — the TFLite Micro inspector — to interact with the model at a higher level — inspect tensors, inject audio, read predictions — instead of raw memory addresses.

2. Performance optimization (~7 min)

With the plugin in place, the agent went back and fixed the root cause. CMSIS-NN — ARM’s optimized neural network kernels, the embedded equivalent of cuDNN — uses SIMD instructions that require aligned memory access. The model data array had no alignment constraint, so the linker placed it at an arbitrary address and CMSIS-NN’s LDRD instructions faulted. One-line fix: alignas(16) on the model array. Inference dropped from 865ms (reference kernels) to 31ms with CMSIS-NN — a 28x speedup.

3. Preprocessing implementation (~47 min)

The model expects MFCC features — a spectral fingerprint of the audio — not raw waveforms. To test with real recordings, those features need to be extracted somewhere: either on the host in Python, or on the device in firmware. The agent started with Python (librosa), which is faster to iterate on. It ran, but classifications were wrong — training/serving skew. The model was trained with a specific fixed-point preprocessing pipeline that includes spectral subtraction and per-channel automatic gain control — steps that don’t exist in standard MFCC extraction. The features looked right but were semantically wrong.

Pivoted to implementing the exact training pipeline in firmware. Correct classifications, but preprocessing took 5.3 seconds. Switching the compiler to optimize for speed cut it to 2.6 seconds. Per-step profiling revealed the FFT was 98% of the remaining time — replacing Kiss FFT with ARM’s CMSIS-DSP arm_rfft_q15 brought preprocessing to 67ms.

The agent then tried to optimize the remaining preprocessing steps with CMSIS-DSP intrinsics — each attempt either broke accuracy or ran slower, so all were reverted. The compiler’s -O2 was already doing a good job on those small loops. The tradeoff: flash grows from 342KB to 442KB. On a chip with 1MB of flash, that’s fine — but on a tighter budget you’d weigh that differently.

4. Final profiling & verification (~15 min)

Using the plugin’s arena_info tool, the agent found only 6,948 bytes used out of 16,384 allocated — and right-sized the tensor arena to 7KB, saving 9KB of RAM. The agent initially instrumented the firmware with timestamps for profiling. I steered it toward using the Cortex-M4’s DWT cycle counter instead — hardware profiling with zero firmware changes. That led to a second plugin — the Cortex-M profiler — non-invasive cycle counting through the debug probe. Final validation — automated through the plugin’s accuracy_test tool — against 130 real-world recordings from the Google Speech Commands dataset: 94.6% accuracy.

Final numbers:

The agent also documented the entire process as a technical setup guide — build instructions, troubleshooting, per-step profiling data, and optimization history — so a human engineer can reproduce or continue the work.

The two plugins

Two Python files — the TFLite Micro inspector and the Cortex-M profiler — turned the debug probe from a byte-level tool into an edge-AI development environment.

These are the third stage of the poke→spec→plugin arc — but here, the agent built its own plugins during the session, then used them to accelerate everything that followed. The plugin creation itself took 6 minutes. The time saved by having structured tools instead of raw memory access was far greater.

TFLite Micro inspector (9 tools) — model-level interaction. The agent reasons about the ML model, not memory addresses.

All addresses are resolved automatically via ELF symbol lookup inside the debug probe MCP server — the agent never deals with raw addresses, and the tools keep working across rebuilds. Output scores are dequantized for human-readable results:

predicted_class: "yes"
confidence: 99.6%
categories: silence=0.0%, unknown=0.0%, yes=99.6%, no=7.0%

Cortex-M profiler (3 tools) — non-invasive hardware cycle counting, zero firmware changes.

Both plugins work with any TFLite Micro model on any Cortex-M target — class labels, model symbols, and CPU frequency are configurable. Available in the dbgprobe-mcp-server and nrf52840-edge-ai repositories.

What this means for ML engineers

If you’ve trained a model and want to test it on real hardware — not ship it to production, but try it and learn the embedded side along the way — this workflow gets you there in hours instead of weeks.

You don’t need to learn Zephyr or CMake. You don’t need to understand linker scripts or debug hard faults manually. You don’t need to write fixed-point DSP code by hand or instrument firmware for profiling. The agent handles the embedded plumbing. You focus on what you know: is the model working? Are the classifications correct? Is the latency acceptable?

Think of it as running a model in a Jupyter notebook before building the serving infrastructure — except the notebook is running on real silicon. The tensor arena is like a pre-allocated memory pool for intermediate activations (similar to how TensorFlow manages GPU memory, but with a fixed budget set at compile time). CMSIS-NN is the embedded equivalent of cuDNN — optimized kernels for common neural network layers.

This is not production deployment. The firmware the agent writes needs review. The memory layout needs hardening. Power management is nonexistent. But as a prototyping and validation tool — getting to “yes, this model runs on this chip, here are the real numbers on real silicon” — it collapses the timeline dramatically.

Getting from zero to 70% fast lets you spend your expert attention on the hard 30% — power optimization, edge cases, production robustness, certification. The piping, the configuration, the build-flash-test iteration — that’s what the agent accelerates.

Closing thought

The earlier posts in this series gave the agent access to hardware surfaces — BLE, serial, debug probes. This post is different. It isn’t about building a new MCP server. It’s about what happens when you combine an existing tool — the debug probe MCP server — with domain-specific plugins and point them at a real problem.

The session took 90 minutes. Doing the same work manually — Zephyr setup, TFLite Micro integration, CMSIS-NN debugging, fixed-point FFT conversion, profiling — typically takes days. Not because it’s conceptually hard, but because every step involves build systems, toolchain quirks, and hardware debugging. The agent handles the plumbing. The engineer directs the intent.

The interesting shift isn’t “AI that writes firmware.” It’s AI that iterates on the physical behavior of a real system — flashing, observing, measuring, adjusting — in a tight loop. Edge AI deployment is one example. The pattern applies anywhere the gap between software intent and physical behavior needs to close.

Links

• dbgprobe-mcp-server: GitHub · PyPI
• tflite_micro plugin: README
• cortex_m_profiler plugin: README
• TFLite Micro: GitHub
• claude-replay: GitHub · npm

Claude Code sessions are great for development, but hard to share.

Screen recordings are bulky, static, and you can’t search or skip ahead. Copy-pasting raw transcripts is noisy — tool calls, thinking blocks, and responses all run together. The most interesting part of a session isn’t just the final output — it’s the reasoning: which tools the agent called, in what order, and why.

I’ve been writing a series of posts about giving AI agents direct access to hardware through MCP, and every article includes demos of the agent interacting with real devices. I wanted to show the full sessions — not just cherry-picked screenshots — but there wasn’t a good way to do that.

The logs are already there

Claude Code stores complete session transcripts as JSONL files in ~/.claude/projects/. These logs already contain everything needed to reconstruct a session:

• User prompts
• Assistant responses (with markdown formatting)
• Tool calls and their results
• Thinking blocks
• Timestamps for every interaction

The data is all there. It just needs a player.

So I built one

claude-replay is a small CLI tool that converts these logs into interactive HTML replays. It works with sessions from Claude Code — both the terminal CLI and the VS Code extension — and also supports Cursor transcripts.

npx claude-replay session.jsonl -o replay.html

The output is a single self-contained HTML file — no external dependencies, no framework, no server. You can email it, host it on GitHub Pages, or embed it in a blog post, or link to it. For platforms that don’t support iframes (like dev.to or Medium), you can host the replay and embed it via CodePen.

Here’s what that looks like:

Hit play, step through with arrow keys, or click the progress bar to jump around. Expand tool calls to see inputs and results. Collapse thinking blocks if you want to focus only on the conversation.

Use cases

The obvious one is blog posts and documentation — embed a session instead of stitching together screenshots. But the use case I didn’t expect was knowledge sharing in teams.

When you’re working with AI agents, the interesting artifact isn’t just the code that got written — it’s the session that produced it. How did the agent approach the problem? What tools did it reach for? Where did it struggle? A replay captures all of that in a format someone else can actually step through.

Instead of “hey, look at this cool thing the agent did” in Slack followed by a wall of text, you send a replay.

Features

• Interactive playback with speed control (0.5x to 5x)
• Collapse/expand tool calls and thinking blocks
• Bookmarks and chapters for long sessions
• Keyboard shortcuts (space, arrow keys)
• Multiple themes (dracula, monokai, github-light, and more)
• Automatic secret redaction
• Self-contained — zero external dependencies

Getting started

# Install globally
npm install -g claude-replay

# Or run directly with npx
npx claude-replay session.jsonl -o replay.html

# Find your session logs
ls ~/.claude/projects/*/

A few useful options:

# Only include turns 5-15, start at 2x speed
claude-replay session.jsonl --turns 5-15 --speed 2.0 -o replay.html

# Add chapter markers
claude-replay session.jsonl --mark "3:Flash firmware" --mark "7:BLE scan" -o replay.html

# Use a different theme
claude-replay session.jsonl --theme dracula -o replay.html

Links

• claude-replay (source): GitHub
• claude-replay (package): npm

This post is part of the Let the AI Out series on giving AI agents direct access to hardware. Start here for the overview.

The previous posts in this series focused on what the agent can access — BLE, serial, debug probes. Each one gave the agent a new hardware interface.

This post is about where the agent runs — and why that shouldn’t matter.

The agent doesn’t need MCP to talk to hardware. It could generate a pyserial script, shell out to GDB, or write bleak code directly.

But every model reasons about that differently, and every client executes scripts in its own way. The integration ends up being reimplemented over and over.

For hardware, this is especially painful. Tools like pyocd, probe-rs, or SEGGER’s J-Link CLI aren’t well represented in training data. The agent is more likely to hallucinate flags or generate outdated code. And hardware interfaces are inherently stateful — you can’t open a BLE connection, read a characteristic, and subscribe to notifications across three separate script invocations. The session has to stay alive.

MCP changes that. The hardware knowledge lives in one place - the server — with stateful sessions, standard discovery, structured parameters, and typed responses. The client just calls tools. That’s what makes interop possible.

Instead of embedding hardware knowledge inside the agent or the IDE, MCP moves it into reusable servers. The agents can change, but the hardware interface stays the same.

This post tests that claim. The same three MCP servers — BLE, Serial, Debug Probe — running across Claude Code, VS Code + Copilot, and Cursor. Same servers, same tools, different hosts.

The setup

All three MCP servers — BLE, Serial, and Debug Probe — configured in three different clients, talking to real hardware.

Claude Code

Terminal-native. MCP servers are registered from the CLI:

claude mcp add ble -e BLE_MCP_ALLOW_WRITES=true -e BLE_MCP_PLUGINS=all -- ble_mcp
claude mcp add serial -e SERIAL_MCP_PLUGINS=all -- serial_mcp
claude mcp add dbgprobe -e DBGPROBE_JLINK_DEVICE=nRF52840_xxAA -- dbgprobe_mcp

Tools are discovered automatically. Everything — tool calls, results, reasoning — flows through the same terminal session. For permissions, /permissions supports wildcards: mcp__ble__* allows all BLE tools, mcp__dbgprobe__* allows all debug probe tools.

VS Code + Copilot

IDE-native. MCP servers are configured via .vscode/mcp.json:

{
  "servers": {
    "ble": {
      "type": "stdio",
      "command": "ble_mcp",
      "env": {
        "BLE_MCP_ALLOW_WRITES": "true",
        "BLE_MCP_PLUGINS": "all"
      }
    }
  }
}

Serial and debug probe servers are configured the same way.

VS Code provides a “Start” button inline in the JSON file. Servers also start automatically when you open Copilot Chat. Tool calls appear in the chat panel with approval prompts — you can allow a specific tool or all tools for a server, scoped to the current session, workspace, or everywhere.

Cursor

Also IDE-native, but with some differences. Configuration lives in .cursor/mcp.json:

{
  "mcpServers": {
    "ble": {
      "command": "ble_mcp",
      "args": [],
      "env": {
        "BLE_MCP_ALLOW_WRITES": "true",
        "BLE_MCP_TOOL_SEPARATOR": "_"
      }
    }
  }
}

Serial and debug probe servers follow the same pattern.

Servers are managed in Settings > Tools & MCP, where each one shows a status indicator and a list of discovered tools. Tools are disabled by default — you need to enable them manually. For permissions, Cursor offers three modes: “Ask every time,” “Use allowlist” (approve once per tool), and “Run everything.”

Note the TOOL_SEPARATOR env vars. The MCP spec allows dots in tool names (ble.scan_start, serial.open). Claude Code and VS Code + Copilot handle this fine, but Cursor converts dots to underscores and sends the mangled name — which the server doesn’t recognize. The fix: a configurable separator. Set TOOL_SEPARATOR=_ and the server registers ble_scan_start instead. This required new releases across all three servers (ble-mcp-server 0.1.4, serial-mcp-server 0.1.2, dbgprobe-mcp-server 0.1.1).

A small thing — but exactly the kind of interop edge case that shows up when you test a protocol across real clients.

The experiment

To test whether the same MCP servers behave consistently across different agent clients, I ran the same workflow against a Nordic nRF52840 development kit running Zephyr’s Peripheral UART sample.

It exposes a UART interface over both serial and BLE (via Nordic’s UART Service, NUS) — so you can send a message over BLE and see it arrive on the serial console, and vice versa.

This makes it a convenient test device: it lets the agent exercise three independent hardware interfaces — silicon, serial console, and wireless — and verify that data flows correctly between them.

Each step below shows the same operation in Claude Code, VS Code + Copilot, and Cursor.

Programming the silicon

The agent starts by interacting with the debug probe.

It discovers the connected J-Link, performs a full chip erase, and flashes the Zephyr firmware.

Left: Claude Code · Center: VS Code + Copilot · Right: Cursor

This is the lowest layer of the system: direct access to the microcontroller itself.

Accessing the device console

Next the agent opens the serial console.

At this point the agent has access to the device’s wired interface — the same console a developer would often use for debugging.

Discovering the wireless interface

The firmware also exposes the same UART service over BLE.

The agent scans for nearby devices and connects to the Nordic UART device.

This step demonstrates a completely different interaction surface: wireless discovery and GATT communication.

End-to-end validation

Finally the agent verifies that the firmware is bridging the two interfaces correctly.

It sends a message over BLE and confirms it appears on the serial console, then performs the reverse test.

The agent is now interacting with the device through three independent surfaces:

Debug probe  → silicon control
Serial       → device console
BLE          → wireless interface

What the transcripts show

The screenshots tell you the workflow is the same. The transcripts tell you the reasoning is different.

Each agent took a different path through the same set of tools. Claude Code was methodical — it found the hex file with a single search, performed full BLE service discovery before writing, and flushed the serial buffer proactively. VS Code + Copilot hedged — it ran two parallel BLE scans (one filtered, one broad) and struggled with its own file search before falling back to a directory listing. Cursor was the most direct — it picked merged.hex when zephyr.hex wasn’t in the expected location, used a name filter to narrow the scan, and moved through the steps with minimal narration.

Different strategies, different trade-offs, same MCP tool calls, same results. The protocol is the stable layer. How the agent plans and sequences is up to the model and the client. What the server exposes is always the same.

The only client-specific accommodation was the tool separator fix for Cursor. Everything else — the tool names, the parameters, the responses — was identical across all three.

Here’s the full Claude Code session — every tool call, every response:

Why this architecture matters

MCP decouples the capability from the client.

The BLE server doesn’t know if it’s being called from a terminal, an IDE, or a script. It exposes tools — scan, connect, read, write — and the client decides how to present them. The same is true for serial and debug probe.

This is what a protocol buys you. The hardware integration is written once in the server, and every MCP-compatible client can use it. No plugins, no adapters, no client-specific code.

For hardware tools specifically, this matters more than it might seem. Embedded developers don’t all live in the same editor. Some work in VS Code, some in the terminal, some in specialized IDEs. The hardware interface shouldn’t depend on the editor choice.

Seen this way, MCP servers start to look less like plugins and more like infrastructure.

Each server exposes a surface of the physical system:

• BLE exposes the wireless interface
• Serial exposes the debug console
• A debug probe exposes the silicon state

These servers translate hardware interactions into structured tools an agent can reason about.

Once that interface exists, any MCP-compatible client can use it — a terminal agent, an IDE assistant, or something that hasn’t been built yet.

Closing thought

The previous posts were about giving the agent eyes and hands — BLE for the wireless surface, serial for the debug console, and a debug probe for the silicon itself.

This post shows something different: those capabilities aren’t tied to a specific tool or editor. Because they’re exposed through MCP, they move with the agent. Terminal, IDE, or something else entirely — the same servers work the same way.

It’s also the first time all three ran together in a single workflow against the same device: debug probe, serial, and BLE. Flash the silicon, open the console, connect over the air, verify the bridge. Same tools, same hardware, different clients.

That’s the point of a protocol.

Write the server once. Any agent can use it.

Links

• ble-mcp-server: GitHub · PyPI
• serial-mcp-server: GitHub · PyPI
• dbgprobe-mcp-server: GitHub · PyPI

This post is part of the Let the AI Out series on giving AI agents direct access to hardware. Start here for the overview.

Serial MCP gave the agent the developer’s view — boot banners, debug logs, CLI commands. BLE MCP gave it the user’s view — scanning, connecting, reading characteristics.

Both are curated surfaces. Both are views the firmware chooses to present.

A hardware debug probe bypasses all of that.

It connects directly to the silicon — the core, the memory, the registers. If the firmware lies, the probe tells the truth. If the firmware crashes, the probe still answers.

This post is about giving that level of access to an AI agent — the same access you have when you’re at your most powerful: halted at a breakpoint, staring at the stack.

When abstractions fail, you drop to the metal. Now the AI agent can too.

What this looks like in practice

End-to-end demo: connecting to a real target, flashing firmware, halting execution, setting breakpoints, reading memory, using ELF for symbol resolution, and attaching SVD for register-aware inspection.

The server currently supports J-Link probes only. The architecture is backend-agnostic — OpenOCD and pyOCD backends are planned — but today, you’ll need a J-Link and the SEGGER J-Link Software installed separately.

What this is (and what to read first)

This is the fifth post in the series:

• What Is MCP? — the protocol underneath. Start here if MCP is new to you.
• BLE MCP Server — wireless interface access. Covers the poke→spec→plugin arc in detail.
• Serial MCP Server — serial console access. Covers PTY mirroring and line-oriented I/O.
• BLE + Serial Demo — using both interfaces together to debug a real device.

The Debug Probe MCP Server follows the same pattern: stateful MCP server, stdio transport, plugins, tracing. If you’ve read the earlier posts, the architecture should feel familiar.

A quick primer on debug tools

If you’ve done embedded development, you know these tools well. If you haven’t, here’s the minimum context to follow the rest of the post.

The debug port

Most modern microcontrollers have a dedicated debug port built into the silicon — a hardware interface designed specifically for development. The two common ones are:

• SWD (Serial Wire Debug) — two wires (clock + data). Used by most ARM Cortex-M chips.
• JTAG (Joint Test Action Group) — four or five wires. Older, more general.

Both provide direct access to the CPU’s internal state — a hardware feature that works even if the firmware is broken, stuck, or not yet flashed.

The debug probe

The debug port on the chip speaks a low-level wire protocol. To use it from a computer, you need a debug probe — a small piece of hardware that bridges USB to SWD or JTAG.

Common ones:

• J-Link (SEGGER) — the workhorse. Fast, reliable, widely supported. What this MCP server uses today.
• ST-Link — ships with every STM32 development board.
• CMSIS-DAP — an open standard. Used by many low-cost probes.

The debug software stack

The software that drives the probe is usually GDB (GNU Debugger) or a tool that speaks GDB’s protocol. The typical stack looks like this:

This MCP server replaces GDB in that stack — it talks directly to the GDB server, so the AI agent drives the debug protocol itself.

What you can do with it

Once connected — probe and software stack in place — you can:

• Control execution — halt, resume, single-step, breakpoints
• Inspect and modify state — read/write memory, registers
• Program and reset — flash firmware, soft/hard reset
• High-speed logging — RTT (Real-Time Transfer) over the debug wire itself, no UART needed

The agent’s development flow

Here’s what a typical session looks like when the AI agent has access to a hardware debug probe — from connecting through flashing, halting, and setting breakpoints.

Connect and orient

The agent lists attached probes, picks one, and connects — specifying the target device, interface (SWD or JTAG), and clock speed. Behind the scenes, the server starts a JLinkGDBServer subprocess and opens a persistent GDB RSP connection over TCP. That connection stays open for the entire session — low-latency, stateful, ready for interactive debugging.

Flash firmware

The agent can program the target directly — .hex, .elf, and .bin files are all supported. When you flash an ELF, the server automatically parses the symbol table (more on that below).

Flashing uses JLinkExe under the hood rather than going through GDB — it’s more reliable and handles vendor-specific unlock sequences. From the agent’s perspective, it’s a single tool call: flash and continue.

Halt, step, inspect

The agent can freeze the CPU and see exactly where execution stopped — the program counter, the current function, the offset within it. From there, it can single-step through instructions, or read memory directly: RAM, flash, peripheral registers, anything in the address space.

Breakpoints

The agent can set breakpoints by address or by function name (when an ELF is attached). Resume execution, and the target runs until the breakpoint fires. The server reports why execution stopped — software breakpoint, hardware breakpoint, manual halt, or something else.

When a workflow involves multiple steps — flash, wait for boot, set breakpoints, resume, check state — plugins let the agent run it as a single operation. The BLE post covers the poke→spec→plugin arc in detail.

ELF integration: the agent knows your symbols

ELF (Executable and Linkable Format) is the standard binary format produced by embedded toolchains — it contains compiled code, section layout, and the symbol table that maps addresses to function and variable names.

Raw addresses are useful. Function names are better.

When you flash an ELF file or attach one to a session, the server parses the symbol table and enriches every response with symbol context. Halt the CPU and you don’t just get a raw program counter — you get main+12. Set a breakpoint by function name instead of a hex address. Look up a symbol and get its address, size, and type. Bidirectional, always available.

The agent reasons about your firmware in the same terms you do. Not “halted at 0x00003400” but “halted at sensor_read.” Not “breakpoint at 0x00002100” but “breakpoint at ble_on_connect.”

When the agent can map addresses to function names, it can form hypotheses about what the firmware is doing and why — not just report raw state.

SVD integration: memory with meaning

An ELF file tells the agent about code — functions, symbols, addresses.

An SVD file tells it about hardware.

SVD (System View Description) files are XML descriptions of a microcontroller’s peripheral register map — peripheral base addresses, register names and offsets, bitfields, enumerations, access permissions.

Without SVD, reading 0x40014000 returns raw bytes. With SVD attached, the agent can ask for TIMER0.TASKS_START by name and get back the value, the bitfields, and what they mean.

The agent doesn’t just see memory — it sees intent. Instead of guessing what 0x40014000 means, it can reason: TIMER0 is running. TASKS_START is asserted. That explains the interrupt storm.

SVD support turns the debug probe from a byte-level tool into a hardware-aware reasoning surface.

RTT: logging without a UART

RTT (Real-Time Transfer) uses the debug connection itself to stream data between the target and the host. The firmware writes to a small buffer in RAM; the probe reads it over SWD/JTAG in the background, without halting the CPU. No UART pins, no extra wiring, no baud rate configuration.

It sits in an interesting middle ground: not curated like a serial console, but not invasive like halting the core. The agent gets continuous visibility into firmware behavior while execution continues in real time.

Getting started

Install:

pip install dbgprobe-mcp-server

The MCP server does not include J-Link tools — you’ll need to install the SEGGER J-Link Software separately. The server auto-detects JLinkExe and JLinkGDBServer on your system. For personal projects and education, SEGGER offers the J-Link EDU Mini at a lower cost.

Not affiliated with or sponsored by SEGGER — it’s just what I had on my bench.

Register with Claude Code:

claude mcp add dbgprobe -- dbgprobe_mcp

Optionally set a default target device and enable plugins:

claude mcp add dbgprobe \
  -e DBGPROBE_JLINK_DEVICE=nRF52840_xxAA \
  -e DBGPROBE_MCP_PLUGINS=all \
  -- dbgprobe_mcp

For VS Code + Copilot, add to .vscode/mcp.json:

{
  "servers": {
    "dbgprobe": {
      "type": "stdio",
      "command": "dbgprobe_mcp",
      "env": {
        "DBGPROBE_JLINK_DEVICE": "nRF52840_xxAA"
      }
    }
  }
}

The tools will be available in Copilot Chat automatically.

Safety

The BLE post covers the safety philosophy in detail — anything that can affect real hardware is opt-in. A debug probe raises the stakes:

Memory writes affect real hardware. A write to the wrong address can corrupt device state, overwrite calibration data, or put the chip in an unrecoverable state. Be aware of what and where you let the agent write.

Flash operations are destructive. Flashing overwrites the chip’s program memory. If the new firmware is wrong, the device won’t boot. A mass erase will wipe all flash contents, including any data or keys stored on the device.

Plugins execute code with hardware access. Same as the other servers — plugins are opt-in, contained to the project directory, and should be reviewed before enabling.

Closing thought

Serial gives you what the firmware says. BLE gives you what it advertises. A debug probe gives you what it’s actually doing — instruction by instruction, register by register, byte by byte.

That changes what kind of problems it can help with. Not just “write the firmware” — but “the device hangs after five minutes, find out why.” Set a breakpoint, wait for it to trigger, inspect the stack, read the peripheral registers, form a hypothesis, test it. The same loop an engineer runs — but automated, repeatable, and stateful.

This doesn’t replace the engineer. It extends the engineer — with structured tools that drive the debug protocol directly, not text scraped from a CLI.

It gives the agent the same level of access you have when you’re at your most powerful — at the probe, halted at a breakpoint, staring at the stack.

When abstractions fail, you drop to the metal. Now the AI agent can too.

Links

• dbgprobe-mcp-server (source): GitHub
• dbgprobe-mcp-server (package): PyPI
• SEGGER J-Link Software: Downloads
• Serial MCP Server: GitHub
• BLE MCP Server: GitHub