Dear AlmostHadYa, I got live previews working with a batch ASR model, and it's more accurate than the real-time ones

I went to check my Stripe account the other day and I discovered I had a handful of subscribers. Insane! How did they even find me? I procrastinated a bit then sent them personal emails thanking them and asking if they had any bug or feature requests.

One reply:

Hi Marshall,

[A cancellation request, and a reminder that California has digital subscription laws.]

P.s. I like the app but the real-time transcription left something to be desired. I’d include non-NE [Neural Engine] GPU real-time transcription models with more accuracy.

Ouch. First: I’m not running a national gym chain over here. Billing is through Stripe and I refund freely. But then the real-time transcription feedback. Do I completely suck? Did I miss an obviously better solution? Time to spend tokens on ego recovery, I mean a local ASR model bake-off.

To be fair to him: the live view he saw was rough. Words went missing where two decoding windows met, phrases showed up twice, and numbers got split in half. It would have been reasonable to conclude that a batch model just can’t do live previews. Conceptually, though, merging overlapping windows should be straightforward, but the model writes some numbers as digits on its own, so two windows hearing the same “twenty twenty” could disagree on how to spell it, and the merge tore the number apart.

Hard requirement:

• Strong punctuation: This lets me collect cleaner voice embeddings since humans usually finish their thoughts in sentences. Less punctuation = contaminated speaker samples and janky attribution.

Nice to have:

• Live transcript text popping up within a few seconds of being spoken, so you can follow the meeting as it happens.
• No re-transcription after the call: the transcript you watched is the one that gets saved.

The short version: over 18.8 hours of meetings and earnings calls, MimicScribe got 15.5% of words wrong (next best: 16.9%) and 96.9% of spoken numbers right, with text about 2 seconds behind live. Every model I tried is compared at the bottom. *

Path 1: Models Built for Live Transcription (The subscriber’s suggestion)

Short version: they show words fast, but they drop more of them or miss the sentence ends I need to tell speakers apart. Running the same model after the call doesn’t fix it.

Models tested: Parakeet-unified-en-0.6b, Voxtral Realtime 4B, Moonshine Streaming.

The obvious idea was to skip windowing and pick a model built for live transcription that spits out tokens as it goes.

• The promise: Words on screen almost immediately, and no window-stitching code to write.
• Why it failed:
  • Missing sentence ends: Parakeet-unified marked only 20% of the sentence ends where one speaker hands off to the next, against 89% for mine. Moonshine marked 72%. I use sentence punctuation to isolate speaker embeddings. Without reliable sentence ends, speaker attribution gets a lot worse. *
  • Dropped words: Voxtral is the exception on punctuation (it marked sentence ends best of anything I tested, 94%), but it skipped nearly 3x as many real words as mine on clean speech, at both 4-bit and full precision, so it’s the model and not the compression. Moonshine skipped twice as many words as mine, failed on every first decode on the Apple Neural Engine, and on CPU kept 2.25 cores busy for a single stream. A meeting runs two at once (mic + system audio). *

Is it the streaming, or the model? Parakeet-unified can also run after the call, reading each stretch of audio with full context in both directions. Run that way, it scores about the same:

The same one-meeting scoring as the table at the bottom.

Live and after-the-call Unified land within half a point of each other. The model just writes differently. It leaves out ums, repeats and false starts, and in back-and-forth conversation it stops punctuating (“morning everyone hope everyone’s doing well so could we just talk…”). Fine for captions, bad for splitting speakers. *

Path 2: Re-transcribing with a Bigger Model

Short version: the big models wrote down less of what was said, not more, and gave no word timings, so a second pass would make transcripts worse and slower.

Models tested: Cohere Transcribe 2B (as a post-meeting re-transcriber), Granite Speech 4.1-2B.

The other temptation was to keep a small model for the live preview, then cut the audio at segment boundaries and re-transcribe it with a 2B+ model that has an LLM decoder.

• The promise: A quick live preview now, and a stronger model cleaning up the transcript afterwards.
• Why it failed:
  • Dropped words: Over 18.8 hours, Cohere got 20.4% of words wrong against my pipeline’s 15.5% (and 16.4% for the same Parakeet model with none of my repair layers). It skipped 12,500 more words than mine. About 40% of those were ums and uhs it chose not to write down, but the rest were ordinary words like “the”, “you” and “know”. It also dropped nearly twice as many numbers (87 vs 49). A second pass that drops more words can’t fix anything. Under crosstalk it also got stuck in loops: one chunk came out as “Yeah.” 206 times. (See Cohere in the bake-off) *
  • Unstable at the seams: Granite decodes long chunks, not words as they arrive, so it’s no use for live preview. And its output depends on where the chunks are cut: moving the seam changed 25% of the stitched transcript, while a much longer chunk degenerated into one sentence repeated over and over. On the audio it hadn’t been trained on, it tied mine on one channel and lost on the other. (See Granite in the bake-off) *
  • No word timings: Cohere gives none and Granite only rough ones, so neither can line words up with the audio for playback or speaker attribution.

Path 3: The Usual Advice (English-only v2, NVIDIA’s own runtime)

Short version: both were worse. The English-only model skipped thousands more words, and NVIDIA’s own runtime was slightly less accurate than Apple’s Neural Engine running the same model.

Models tested: Parakeet v2 vs v3, each through NVIDIA NeMo and through FluidAudio on the Apple Neural Engine.

The common lore says English-only models beat multilingual ones on English audio, and that NVIDIA’s own runtime has to beat a CoreML port on Apple’s Neural Engine.

• The promise: Standard PyTorch/NeMo environment, easy experimentation, and focused English weights.
• Why it failed:
  • v2 is worse: Parakeet v2 was over 3 points worse than v3 through both runtimes, and the whole gap is words it skipped: 7,600 more of them. It reads cleaner because it writes down less. (See v2 vs v3 in the bake-off) *
  • “Just run it in NeMo” didn’t buy accuracy: On the same model, FluidAudio on the Apple Neural Engine beat NVIDIA’s own NeMo runtime by 0.6 points (0.7 on v2), and got more numbers right. With my repair layers on top the gap is 1.5 points. And it leaves the GPU free for whatever else is running during a call. (See runtime comparison table) *

What I Shipped: Forcing a Batch Model to Stream

Instead of accepting the accuracy hits of live-first models or the lag of post-meeting re-transcription, I took NVIDIA’s batch model (Parakeet TDT 0.6B v3) and made it run live. Text shows up about two seconds after it’s spoken, and it drops fewer words than any model I tested, live or after the call.

• How it works: Audio is decoded on the Neural Engine in overlapping ~15-second windows. A stitcher merges the overlaps and shows the newest words in grey, and they turn white once they stop changing. The last few words are held back until they’re unlikely to change, which is what keeps the preview from flickering.
• Latency: about 2 seconds behind speech, 90% within 4 seconds, across 12 recorded test sessions.† *
• Whole-system accuracy: the live stream is the saved record, so every seam fix improves the saved transcript too, not just the preview.

Every model, side by side

Lower is better in the first three number columns. “18.8 h” is 16 recorded meetings plus 11 earnings calls, every word counted. “1 meeting” is one 14-minute recorded meeting that every model ran, with ums and uhs removed; the skipped-words and sentence-end columns come from it too. “Numbers right” is from the 11 earnings calls. Full bake-off. ¹Granite’s training data includes these recordings, so its score wouldn’t be fair. *

The repair layers recover 2,108 words the model alone skips, mostly where two decoding windows meet. *

Models that score higher on the Artificial Analysis and Hugging Face Open ASR leaderboards are mostly cloud APIs, which means sending your audio off the Mac, or models several times Parakeet’s size, too heavy to run next to a video call. *

For on-device transcription, Parakeet (0.6B) is the right size. It’s accurate enough for most meetings, and chasing the last fraction of a point with cloud APIs or huge models doesn’t buy much once names, numbers and sentence ends come through. Plus your audio stays on your Mac. So boom.

This took quite a bit of seam stitching and invariant work. I wanted this running live, so while I borrowed heavily from FluidAudio, the live pipeline is built specifically for real-time meeting capture.

Amusingly, I asked an AI model for architectural advice halfway through and it gave me this:

What Should Be Stopped Immediately

1. STOP writing and maintaining custom seam-stitching and live-view invariant harnesses.
4. STOP optimizing on-device Neural Engine ASR around the constraint of zero post-meeting processing.

I’d say this was probably good advice from the model, but too late: I’ve already spent the time.

It was an absolute rabbit hole. If you look through the release notes from that stretch (from v1.0.0-rc.24 through rc.27), you can see the bizarre edge cases:

• Phrases committing twice at pauses: If someone paused mid-sentence right on an audio boundary, the stitcher would helpfully write “on Friday. on Friday.” (Which turned out to have three separate root causes).
• Numbers getting torn apart across seams: Quantities were getting sliced in half where two listening windows met, famously turning “Q3 2020” into just “Q3”.
• Words vanishing at window edges: Speech at the very start or end of a window would quietly disappear if the decoder lacked surrounding context.

So basically: you can use AI agents to make yourself feel better. Anyway, you can check out and download MimicScribe for macOS.