Five platforms, one harness: a head-to-head live translation benchmark

Have you ever used an end-to-end live translation system, or seen one in action at an event? Ever wondered how anyone decides which one is best? Here’s how we evaluate ours, and how it stacks up against four of the competitors we hear about most often: Palabra, LiveVoice Pro, OpenAI GPT-Realtime-Translate, and Google Meet.

What we score, and how

Two scores capture what listeners actually experience: how accurate the translation is, and how long they wait for it. Both line up with how a careful human listener would grade the same audio.

GEMBA-MQM v2

GEMBA-MQM v2 uses a large language model to read what the speaker said and what the system translated side by side, and returns a list of the mistakes the system made. Each mistake is tagged by type (mistranslation, omission, hallucination, and so on) and severity. Severity sets the weight each error contributes to the session’s final score:

The session score is the negated sum of those weights, so closer to zero is better. A worked example, from a Spanish science talk:

Speaker (Spanish): “…si pudiéramos vivir sin agua…”

System’s translation: “…if we could live with water…”

GEMBA returns one error: “sin agua” (“without water”) rendered as “with water”, reversing the meaning entirely. A mistranslation, severity CRITICAL, weight 25. Score: −25. A single LLM judgment is noisy, so we run the scoring 10 times and aggregate the results. For the full methodology, see our GEMBA-MQM v2 deep-dive.

Ear-Voice Span

For each phrase the speaker says, we find the matching translated phrase, then measure the time gap between when the source phrase started and when the translation started. That gap, in seconds, is the Ear-Voice Span (EVS) for that phrase.

Across a session we report the median (typical delay) and the 90th percentile (the slowest 10% of phrases). For the full methodology, see our EVS deep-dive.

Five platforms, head-to-head

We ran our evaluation on five systems using the same harness: VoiceFrom Interpret, LiveVoice Pro, Palabra, OpenAI GPT-Realtime-Translate, and Google Meet. Eight sessions of around six minutes each, across eight language pairs spanning five languages (English, Spanish, French, German, Portuguese). Same audio in, scored the same way. Google does not support translating from Spanish into Portuguese, so it has seven sessions instead of eight.

Translation accuracy

VoiceFrom leads in average accuracy by a wide margin.

Speech latency

Google and OpenAI are both faster than VoiceFrom at the median. The accuracy chart above shows the other side of the trade.

* GEMBA-MQM v2, scored 10 times per session and aggregated; closer to 0 is better.

Accuracy is where the platforms separate. The three sessions below show what that gap sounds like.

A French physics joke

The speaker is explaining cosmology in French. They reach the moment of the Big Bang and describe it as “un univers ponctuel”: a point-like universe. Then they make a joke: “ponctuel, ça veut pas dire qu’il est arrivé à l’heure.” Punctual doesn’t mean it arrived on time. The joke only works for two reasons. First, ponctuel means both point-like (the physics term) and punctual (the everyday word), so the setup needs to land both meanings in the same word. Second, the punchline has to keep the physics right: the universe is the size of a point, not that it came from a point.

Palabra and OpenAI both reach for the technical translation, point-like, and lose the wordplay outright. Google does the same with one-off, which has no double meaning to anchor the punchline. LiveVoice keeps the word punctual but stretches the punchline across three sentences. The joke wants two beats. Only VoiceFrom keeps the setup-and-punchline timing intact:

“…a punctual universe. Punctual does not mean that it arrived on time. It means that it is the size of a point.”

The same word, punctual, carries from the setup into the punchline, and the physics lands the way the speaker intended.

A Sinek contrast in German

Late in the talk, Sinek lands on his central contrast: “The goal is not to do business with everybody who needs what you have. The goal is to do business with people who believe what you believe.” The whole argument hinges on both halves landing, and an audience listening in German needs that parallel to carry across the language boundary.

LiveVoice inverts the first half, affirming the very thing the source rules out. Palabra splits the negation across three sentences and lands in the same place. Google keeps the parallel structure but truncates the second half into three short fragments. OpenAI keeps the structure and the meaning, with awkward German in the first half. Only VoiceFrom lands the parallel, the meaning, and the phrasing in one clean beat:

“Das Ziel ist es nicht, Geschäfte mit jedem zu machen, der das braucht, was sie haben. Das Ziel ist es, Geschäfte mit Menschen zu machen, die glauben, was sie glauben.”

A Spanish aside about anti-anxiety medication

A few minutes in, the speaker is asking listeners to imagine cosmic infinity, then breaks the tension with an aside: “…algo que no termina, y no termina, y no termina, y no termina, y un Rivotril, por favor.” Rivotril is a brand-name anti-anxiety medication in Spanish-speaking countries; asking for one is shorthand for I need to calm down. The line only works if the listener understands that a real drug is being named.

Every other system misses the drug. LiveVoice hears “a rivet roll, please.” Palabra hears “a river drill, please.” OpenAI hears “a river trail, please.” Google hears “arrive at real, please.” Each one turns the aside into an absurd non-sequitur. VoiceFrom lands within one letter of the real drug name, and the joke survives.

Try it at your next event

If accuracy is what your audience needs, VoiceFrom is the system to put on stage. Talk to us and we’ll set up a pilot at your next event.

For the full methodology behind the numbers above, see the metric deep-dives: GEMBA-MQM v2 for accuracy, and Ear-Voice Span for latency.

Yahya Saleh

Applied ML Engineer

Real-time speech intelligence, on-device models, from papers to production and beyond