Technology

Audio Capture

Microphone input is captured with a raw AVAudioEngine tap — deliberately not Apple's voice-processing I/O, so the OS never ducks or gates the meeting audio the way system-level echo cancellation does. System audio for meeting recording uses Core Audio process taps (CATapDescription + AudioHardwareCreateProcessTap), which capture at the OS level without injecting into individual applications. Both paths produce 16 kHz mono Float32 PCM.

Echo Cancellation

During meeting recording, the microphone picks up both the local speaker and remote participants' audio from the speakers. LocalVQE — a compact neural model from LocalAI for echo cancellation, noise suppression, and dereverberation — runs on-device to remove that system-audio echo from the microphone signal before it reaches the ASR model, using the system audio stream as the reference. It runs on a captured copy of the audio, so the loopback is cancelled without the OS ducking the meeting itself the way system-level echo cancellation does.

Speech Recognition

Transcription uses NVIDIA's Parakeet TDT 0.6B model, converted to CoreML format with a Token-and-Duration Transducer (TDT) architecture and FastConformer encoder. The model supports 25+ languages and all inference runs locally on Apple Silicon. During meeting recording, dual parallel ASR workers process the microphone and system audio streams independently.

For streaming transcription, a custom harness processes audio in overlapping windows. The decoder, joint network, and preprocessor run through an accelerated BNNS backend, bypassing CoreML's per-inference overhead for lower latency.

Seam Stitching

Streaming transcription decodes audio in overlapping windows, so every few seconds two windows cover the same moment and can disagree on a word at the boundary. The naive fix — trim one window's edge — quietly drops words spoken right before a pause, exactly where boundaries tend to fall. MimicScribe instead aligns the overlap, keeps every word both windows agree on, and at a genuine disagreement keeps the fuller reading rather than discarding either side. A marginal word at a window edge is never dropped to make the seam look clean. On a public benchmark of 736 real window-overlap disagreements, zero lost three or more meaningful words.

Speaker Diarization

After a meeting ends, an offline diarization pipeline runs in the background while the live transcript is already displayed. A pyannote-based segmentation model identifies speaker turn boundaries, then a WeSpeaker v2 embedding model extracts 256-dimensional voice embeddings for each segment. Clustering uses agglomerative hierarchical clustering (AHC) with centroid linkage, refined by a Variational Bayes HMM (VBx) pass over PLDA-whitened features.

Clustered embeddings are matched against stored speaker profiles using cosine similarity. When a confident match is found, the speaker is identified automatically. Ambiguous cases are resolved by an LLM disambiguation step using conversation context. Profiles accumulate samples across meetings, improving recognition over time.

Sentence-Level Embeddings

Rather than embedding voices in fixed audio windows, MimicScribe takes its speaker embeddings at the sentence level, aligning each voice measurement to where people actually pause and hand off. It's one of a number of purpose-built refinements that push accuracy well past an off-the-shelf diarizer.

Grammar-Aware Fragment Repair

Diarizers occasionally split one person's turn into tiny fragments — a half-second “again.” shorn off the end of “we can do it.” A fragment that short carries almost no voice signal, so acoustics can't place it. MimicScribe reads the grammar instead: a fragment that completes the sentence before it, or that the next turn finishes, is reattached to that neighbor — with acoustic and turn-taking guards so it never swallows a genuine short interjection.

LLM Post-Processing (optional)

After transcription, text is sent to Google's Gemini 3.1 Flash-Lite for refinement: grammar correction, filler word removal, and tone adjustment. Meeting transcripts are additionally processed for speaker attribution, summarization, and action item extraction. Static prompt content is passed as system instructions to take advantage of Gemini's implicit caching, reducing latency on repeated calls. The speaker accuracy is the deterministic pipeline's, not this pass's — the published SAA is scored before the LLM runs, which only names the speakers and cleans the text.

Live-to-Final Summary Recall

During the meeting, the live assistant keeps a running summary of the points worth remembering as they come up. When the meeting ends, that running summary is handed to the final summarization pass as an inventory of what to cover, and a deterministic coverage check appends anything the summary left out. It's a structural fix for a failure mode every LLM summarizer shares: on a long meeting, a point raised once early on gets crowded out by whatever was discussed at length. Capturing it live, then checking it off at the end, keeps it in the notes.

Speaker attribution accuracy across 57 audio files from 5 public corpora — earnings calls, panel discussions, oral arguments, and dinner-party conversation. Compares on-device diarization alone vs. the full pipeline with LLM speaker attribution.

Whether a returning speaker is recognized from a saved voice profile, across 779 trials on the AMI meeting corpus, with a second corpus (ICSI) checking the result isn't tuned to one room. 92.0% correct and zero wrong identifications. The rest are refusals: when the app is unsure it declines to pick rather than write the wrong name into a transcript.

Real-time briefing quality across 96 scenarios — sales discovery, customer success, standups, interviews, and long meetings. Evaluates action items, hallucination resistance, question detection, and interpersonal awareness using dual LLM judges.

Reference document RAG pipeline across 15 document types — CRM records, scraped webpages, SEC filings, PDFs, tracked-changes contracts, strategic plans, competitive intel, and messy meeting notes. Tests whether relevant chunks are retrieved accurately during live meetings, even from noisy real-world sources.

On-device semantic (vector) retrieval blended with SQLite full-text search, across 26 meetings with 138 queries. Measures recall and ranking for finding past meetings by topic, action item, or conversational phrase — the semantic search runs entirely on your Mac using on-device embeddings.