Technology

On-device speech, diarization, and retrieval engineering—and the public benchmarks behind every number.

Core engineering

Multi-model diarization consensus & syntactic turn repair

Acoustic clustering alone (pyannote segmentation + sentence-level WeSpeaker v2 embeddings + VBx) degrades on rapid turn-taking and far-field bleed, while end-to-end neural diarization (NVIDIA Nemotron 3) can drift on global speaker count. I intersect both: VBx clustering anchors the speaker inventory while cross-model confidence margins resolve disputed turns and overlaps. A post-clustering pass combines grammatical continuation cues with acoustic turn-taking checks to reattach split sentence fragments without absorbing real interjections.

98.9% WDER attribution across 57 public files (98.4% single-diarizer); 98.8% on held-out far-field meetings (91.7% single-diarizer), scored pre-LLM.


Boundary-aware streaming over a batch TDT encoder

Adapting a batch TDT model to ~2s live latency introduces boundary truncations and silent-head dropouts. Dual per-channel workers pair a CoreML FastConformer encoder with a direct BNNS Graph decoder/joint backend to bypass CoreML per-step dispatch overhead. Rather than hard-slicing window midpoints, the seam stitcher reconciles competing hypotheses by favoring higher-coverage lexical spans at boundaries, backed by a VAD-guided repair pass that re-decodes voiced audio the decoder left blank.

1.5% live word deletion rate (vs. 7.1% best streaming baseline); 96.6% live speaker attribution.


Margin-gated cross-meeting voice identification

Single-threshold cosine matching against saved voiceprints accumulates false positives as a speaker library grows. I score diarized speaker clusters against enrolled WeSpeaker embeddings using margin-gated similarity and acoustic quality filters, deferring ambiguous matches to conversational context or leaving speakers unnamed rather than guessing.

96.3% returning-speaker recall with 30 seconds of clear speech per recording, 0 false identifications across 779 trials. Strangers with no profile are tested separately: 0 of 2,415 on the current pipeline, 5 of 1,401 in a harder same-room test on earlier vectors.


Deterministic summary recall & small-model structured fusion

Single-pass LLM summaries drop details on hour-long calls, and complex multi-step prompts break on smaller local weights. Post-meeting enrichment runs one schema-constrained JSON pass engineered to hold up on 8B–9B local models (Ollama, LM Studio) as well as Gemini (via an open-source proxy) or enterprise OpenAI-compatible endpoints. Live notes captured during the call are injected as a structured checklist, and a deterministic reconciliation pass appends any omitted items.


Non-ducking capture & Apple Silicon AEC acceleration

macOS VoiceProcessingIO gates near-end speech and ducks call volume. I tap the raw microphone (AVAudioEngine) and per-process system audio (CATapDescription) at 16 kHz mono, running reference-based neural echo cancellation via LocalVQE on a custom Apple Silicon Accelerate/vDSP backend I built and contributed upstream (~5× lower CPU than baseline ggml).

Benchmarks

ASR bake-off

WER, deletion rate, and numerical entity accuracy across local ASR models over 18.8 hours of meetings and earnings calls.

Results

Streaming vs. batch

Seam-stitching and blank-repair ablation deltas against full-file batch decoding, plus live speaker attribution accuracy.

Results

Multilingual ASR

End-to-end WER across all 25 supported languages on FLEURS read speech.

Results

Speaker diarization

Word-level diarization error rate (WDER) across 57 files from 5 public corpora plus 80 held-out far-field meetings.

Results

Cross-meeting speaker ID

Voiceprint recall and wrong names across 779 multi-session trials on AMI, cross-checked on ICSI, plus stranger tests and a real meeting timeline.

Results

Meeting assistant

Live briefing quality across 96 meeting scenarios, evaluated by dual LLM judges.

Results

Context retrieval

Reference document retrieval recall during live calls across 15 document types.

Results

Meeting search

On-device retrieval recall across 138 topic and phrase queries over 26 meetings.

Results

Voice transform

Instruction-following pass rate (93.9%) and entity preservation (100% of numbers, dates, and URLs) for voice-driven text edits.

Results

Open source & upstream

Released by MimicScribe

swift-bnns-graph

Swift wrapper for Apple's BNNSGraph API to execute the RNN-T decoder and joint network on CPU without CoreML dispatch overhead.

GitHub

benchmarks

Corpora, harnesses, and reference transcripts reproducing every benchmark figure on this site.

GitHub

mimicscribe-proxy

Stateless Cloudflare Worker proxy used for default cloud LLM enrichment, published for auditability.

GitHub

Upstream dependencies & contributions

FluidAudio

MIT

CoreML runtime for Parakeet TDT, pyannote segmentation, and WeSpeaker v2 embeddings. Upstream contributions: per-chunk speaker embeddings and opt-in offline embedding skip.


LocalVQE

Apache-2.0

On-device neural acoustic echo cancellation, noise suppression, and dereverberation (DeepVQE on ggml) by localai-org. Upstream contribution: native Apple Silicon Accelerate/vDSP backend (PR #16).