Technology
On-device speech, diarization, and retrieval engineering—and the public benchmarks behind every number.
Core engineering
Multi-model diarization consensus & syntactic turn repair
Acoustic clustering alone (pyannote segmentation + sentence-level WeSpeaker v2 embeddings + VBx) degrades on rapid turn-taking and far-field bleed, while end-to-end neural diarization (NVIDIA Nemotron 3) can drift on global speaker count. I intersect both: VBx clustering anchors the speaker inventory while cross-model confidence margins resolve disputed turns and overlaps. A post-clustering pass combines grammatical continuation cues with acoustic turn-taking checks to reattach split sentence fragments without absorbing real interjections.
98.9% WDER attribution across 57 public files (98.4% single-diarizer); 98.8% on held-out far-field meetings (91.7% single-diarizer), scored pre-LLM.
Boundary-aware streaming over a batch TDT encoder
Adapting a batch TDT model to ~2s live latency introduces boundary truncations and silent-head dropouts. Dual per-channel workers pair a CoreML FastConformer encoder with a direct BNNS Graph decoder/joint backend to bypass CoreML per-step dispatch overhead. Rather than hard-slicing window midpoints, the seam stitcher reconciles competing hypotheses by favoring higher-coverage lexical spans at boundaries, backed by a VAD-guided repair pass that re-decodes voiced audio the decoder left blank.
1.5% live word deletion rate (vs. 7.1% best streaming baseline); 96.6% live speaker attribution.
Margin-gated cross-meeting voice identification
Single-threshold cosine matching against saved voiceprints accumulates false positives as a speaker library grows. I score diarized speaker clusters against enrolled WeSpeaker embeddings using margin-gated similarity and acoustic quality filters, deferring ambiguous matches to conversational context or leaving speakers unnamed rather than guessing.
96.3% returning-speaker recall with 30 seconds of clear speech per recording, 0 false identifications across 779 trials. Strangers with no profile are tested separately: 0 of 2,415 on the current pipeline, 5 of 1,401 in a harder same-room test on earlier vectors.
Deterministic summary recall & small-model structured fusion
Single-pass LLM summaries drop details on hour-long calls, and complex multi-step prompts break on smaller local weights. Post-meeting enrichment runs one schema-constrained JSON pass engineered to hold up on 8B–9B local models (Ollama, LM Studio) as well as Gemini (via an open-source proxy) or enterprise OpenAI-compatible endpoints. Live notes captured during the call are injected as a structured checklist, and a deterministic reconciliation pass appends any omitted items.
Non-ducking capture & Apple Silicon AEC acceleration
macOS VoiceProcessingIO gates near-end speech and ducks call volume. I tap the raw microphone
(AVAudioEngine) and per-process system audio (CATapDescription) at 16 kHz mono, running reference-based neural echo cancellation
via LocalVQE on a custom Apple Silicon Accelerate/vDSP backend I
built and contributed upstream (~5× lower CPU than baseline
ggml).
Benchmarks
ASR bake-off
WER, deletion rate, and numerical entity accuracy across local ASR models over 18.8 hours of meetings and earnings calls.
ResultsStreaming vs. batch
Seam-stitching and blank-repair ablation deltas against full-file batch decoding, plus live speaker attribution accuracy.
ResultsSpeaker diarization
Word-level diarization error rate (WDER) across 57 files from 5 public corpora plus 80 held-out far-field meetings.
ResultsCross-meeting speaker ID
Voiceprint recall and wrong names across 779 multi-session trials on AMI, cross-checked on ICSI, plus stranger tests and a real meeting timeline.
ResultsMeeting assistant
Live briefing quality across 96 meeting scenarios, evaluated by dual LLM judges.
ResultsContext retrieval
Reference document retrieval recall during live calls across 15 document types.
ResultsMeeting search
On-device retrieval recall across 138 topic and phrase queries over 26 meetings.
ResultsVoice transform
Instruction-following pass rate (93.9%) and entity preservation (100% of numbers, dates, and URLs) for voice-driven text edits.
ResultsOpen source & upstream
Released by MimicScribe
swift-bnns-graph
Swift wrapper for Apple's BNNSGraph API to execute the RNN-T decoder and joint network on CPU without CoreML
dispatch overhead.
benchmarks
Corpora, harnesses, and reference transcripts reproducing every benchmark figure on this site.
GitHubmimicscribe-proxy
Stateless Cloudflare Worker proxy used for default cloud LLM enrichment, published for auditability.
GitHubUpstream dependencies & contributions
FluidAudio
MITCoreML runtime for Parakeet TDT, pyannote segmentation, and WeSpeaker v2 embeddings. Upstream contributions: per-chunk speaker embeddings and opt-in offline embedding skip.
