LiveSub Guides
日本語

A flat 1-second timeout failed us: budgets for in-browser translation

Published: 2026-11-15

Works in Chrome or Edge on a computer (Firefox: through a microphone or audio input; the add-on covers YouTube and Twitch; not on phones). Japanese ⇄ English only.

Every service that runs a translation model inside the browser hits the same wall. GPUs vary from machine to machine, sentences vary in length, the model is large and slow to download, and the screen can never freeze while all of this happens.

This post is how LiveSub (a web app that turns the audio playing in a browser tab into live subtitles and translations) rebuilt its "too slow, fall back" rule for local translation from production data. LiveSub is built for live streams watched in the browser. Everything below, from never stopping the subtitles to timing decisions by the rhythm of speech, assumes a live stream. We give the shape of the formulas only. The coefficients are something you measure on your own users.

Open LiveSub and try it

1. The problem

1.1 Two engines, so subtitles never stop

Translation in LiveSub runs on two engines. The translator built into the browser (Chrome's on-device Translator API) produces subtitles first. On computers with a capable GPU, a local translation model that gives more natural output is loaded in the background. When it is ready, LiveSub switches over between sentences. The local model is PLaMo 3 Translate 0.8B by Preferred Networks, Inc. (Built with PLaMo).

After the switch, the original text still appears at once. Each sentence is sent to both engines at the same time. If the local model answers within its budget, its translation is shown; if not, the first translator's text is shown (Figure 4 in 3.1). A sentence that arrives while the local model is still busy with the previous one gets the first translator's text without waiting. On every branch, the subtitles keep moving.

1.2 Start light, then step up as the device allows

The two engines are part of a larger ladder. LiveSub starts in the form that keeps a first-time visitor waiting the least, then steps up as far as the device can handle. A live stream does not wait, so subtitles come first and quality comes after.

Step Speech recognition Translation
1. First visit Light (English about 120MB, Japanese about 206MB) Built-in browser translator
2. Balanced Balanced (English about 400MB, Japanese about 670MB) Built-in browser translator
3. If the GPU is strong enough Balanced (running on the GPU) Local model (PLaMo 3 Translate, about 570MB)

Subtitles never stop at any step. This post is about step 3: once we step up, when do we step down, and when do we step back up?

1.3 It never stops, so it fails silently

Because nothing stops, a wrong decision costs the user good translations without them noticing. The subtitles keep coming, so all they see is "the translation got a bit stiffer." Unless you count fallbacks, you will not notice either.

Here is where we started (Figure 1).

Figure 1: the funnel to local translation. With sessions that used translation as 100%: eligible 74%, started using the local model 46%, used it to the end 36%

The biggest leak is not the fallback but the step before it: sessions that closed before loading finished. That is a download speed problem, and we treat it separately. This post is about the last step: a rule that, once it fell back, never came back.

2. Why a flat timeout fails

2.1 Time grows with sentence length

The first rule was simple. If translating a sentence took longer than a fixed time (1 second), that sentence used the first translator's text, and if that kept happening, the local model was dropped for the rest of the session.

The trouble is that inference time grows with sentence length. Language model inference has two phases: reading the whole input at once (prefill) and producing the output one token at a time (decode). The time for one sentence is a fixed cost for the input plus a cost for every output token. With a flat timeout, longer sentences are more likely to be counted as "slow" (Figure 2).

Figure 2: what happens on a run of long sentences. Bars show time per sentence. The flat 1-second line cuts the peak; the proportional budget rises only for long sentences

In our measurements, a Japanese sentence of about 40 characters used more than 90% of the 1-second line even on a discrete GPU. On an integrated GPU, sentences of ordinary length went over the line. Japanese to English takes longer than English to Japanese for the same number of characters, because each source character turns into several times as many output tokens. In short, the rule was catching "long sentences," not "slow devices."

2.2 Counting consecutive misses breaks on peaks

The rule had the form "fall back when the timeout is exceeded several times in a row." Live streams have peaks of fast talk with long sentences all the time. When things get exciting, sentences get longer and pauses get shorter. If every peak flips the switch and the session never flips back, the streams with the most peaks get the fewest good translations.

Counting consecutive misses cannot tell a streak of bad luck from a truly slow device. It is the same problem people run into when designing circuit breakers in distributed systems. The rule was not too strict. It was measuring the wrong thing.

3. The new rule

We changed three things (3.1 to 3.3). All of them work inside two constraints: subtitles never stop, and switching only happens between sentences (Figure 3). We also describe the measurements behind the rule (3.4) and how names are handled (3.5).

Figure 3: the decision flow. Load the model, measure the device, compare each sentence with its budget, fall back after three over-budget sentences in a row, keep the model and switch back after successful background retries

3.1 A budget proportional to length

The time we wait for each sentence (the budget B) is set per sentence.

B = ( a + N × t ) × s (1 s ≤ B ≤ 2.5 s)

Short sentences keep the old budget, and only long sentences get extra time in proportion to their length. "Slow" no longer depends on sentence length.

Figure 4: two engines. The original appears at once, the first translator always answers, and the local model replaces only sentences that fit the budget

The 2.5-second ceiling comes from the screen. A subtitle is replaced when the next sentence arrives, so a translation that arrives later than that has nowhere to go. How long a subtitle stays on screen grows with the length of the original, so the ceiling stays within that range.

Why estimate the output length before translating? Because the budget has to be fixed when translation starts. Otherwise there is no moment to decide to show the first translator's text instead. If you only learn that a sentence was slow after it is done, its subtitle is already on screen. Estimating from the source length is rough, but it is enough to tell long sentences from short ones.

3.2 Calibrating per device

a and t are not shared across all devices. Right after loading the model, LiveSub translates a warm-up sentence and then a reference sentence, and sets a and t for this device and this session from the measured timings (input time, output tokens and output time).

The warm-up comes first because the first sentence on a GPU also pays for setup such as shader compilation. The reference sentence is an ordinary subtitle-length sentence, so the per-token time is stable.

GPU load differs from session to session, so nothing is saved; we measure every time. Measuring takes less than a second. If no measurement is available, we use a default slightly slower than a discrete GPU. Being lenient on a slow device is safer for the subtitles than being strict on a fast one.

Calibration has a side benefit. If you count the reference sentence's per-token time in speed bands (fast, medium, slow), you can see how your devices are spread across the bands. We count fallbacks by GPU vendor, so putting the two side by side shows whether fallbacks lean toward slow devices or also happen on fast ones.

3.3 Retrying: a way back

After three over-budget sentences in a row, LiveSub falls back to the first translator, but it does not throw the model away (the last step in Figure 3). After k short sentences in a row, the next sentence is also sent to the local model in the background. The subtitle still shows the first translator's text. If the background results keep fitting the budget, LiveSub switches back to the local model between sentences. After m over-budget retries, the local model is dropped for that session and released from the GPU. For speakers who rarely say short sentences, a time-based signal is also in place.

Retries run in the background so that a failed retry never makes the user wait. The subtitle is already showing the first translator's text, so a slow background result changes nothing on screen. Switching back requires several in-budget results in a row so that devices on the edge do not flip back and forth. Making it slightly harder to switch back than to fall back is hysteresis.

Short sentences are the signal because a retry on a short sentence is cheap, likely to succeed, and costs little if it fails. We retry in the valleys of speech (runs of short reactions) and not on the peaks (runs of long sentences). The rhythm of the stream itself becomes the signal.

This is a circuit breaker (closed, open, half-open) applied to a new setting. What is new is not the parts, but putting them together for live subtitles in the browser with two engines, and setting the coefficients from production measurements.

3.4 Measuring

Before changing the rule, we added measurements. For every sentence, we count input and output tokens and the time spent on each. For every session, we count whether loading started, whether model files were downloaded, whether it fell back, whether it came back, whether it stopped, and the speed band. The inference engine already returns token counts and timings, so all we added were counters.

If you move a threshold without measuring, you cannot tell whether things got better. Counting slow sentences and fallback sessions separately tells you whether the threshold is too strict or the device is too slow.

3.5 Word lists: carrying names from recognition to translation

What breaks most often in a stream are fixed names: people, game terms, product names. Recognition spells them differently every time, and translation treats them as ordinary words. LiveSub passes the words a user writes to both recognition and translation.

A name reaches the translation in three stages.

  1. Translate the sentence as it is. If the chosen spelling already appears in the translation, use it (the sentence stays natural)
  2. If not, and the local model is in use, translate again with the model's glossary field (only the name pairs found in that sentence). This translation also waits within the budget from 3.1 and is discarded if it is late
  3. If the name still does not appear, mark the name, translate, and replace the mark in the output with the chosen spelling

In our own comparison of 47 sentences, adding stage 2 kept the number of sentences where the name appeared as specified at 46. The sentences read more naturally in 9 cases and worse in 2 (judged by one reader). For example, the marker method gave "I got flanking ed again", and the glossary field gave "I got flanked again" (both are outputs of PLaMo 3 Translate).

The glossary field also runs within the proportional budget, so "never make subtitles wait" holds here too.

4. Results

Before (2026-11-08, per session, Figure 1):

After (from the version released on the evening of 2026-11-09; the five full days from November 10 to 14):

Before (Nov 8) After (Nov 10 to 14)
Eligible (first translator was Chrome's on-device translator) 74% 80%
Started using the local model 46% 40%
Used it to the end 36% 34% (32 to 38% per day)
Of sessions that started, fell back 20% 14%
Of sessions that fell back, came back No way back About a third
Slow sentences (over budget ÷ translated) Not counted 1.3% (381 / 30,081 sentences)

The first three rows are percentages of sessions that used translation, as in Figure 1. We count sessions (one page load that started subtitles), not people. "Used it to the end" is sessions that started using the local model minus those that fell back and those stopped because subtitles fell behind. To compare with the earlier figure, sessions that came back are not added in.

Fallbacks went down from 20% to 14%, and slow sentences were 1.3% of translated sentences. On the other hand, the share that used the local model to the end went from 36% to 34%, roughly flat. The new rule only acts on the last step of the funnel, and the larger leak before it (getting to the point of using the local model) is still there (Figure 1).

Fallbacks were uneven by GPU vendor.

GPU Share of sessions that started Of those, fell back
Nvidia 43% 3%
Intel (mostly integrated) 29% 32%
AMD 18% 19%
Apple 9% 0%

About two thirds of the sessions that fell back were on Intel. The proportional budget is meant to stop fast devices from falling back on long sentences, and on Nvidia, where discrete GPUs are common, only 3% fell back.

By speed band (per-token time of the reference sentence, three bands), the sessions that started using the local model were 56% fast, 22% medium and 22% slow. We count fallbacks by GPU vendor, not by band.

In production, a sentence averaged 38.3 input tokens and 9.2 output tokens and took 507 milliseconds (from the totals over 30,081 translated sentences). From November 12, when we started counting them, to November 14, sentences that arrived while the previous one was still being translated, and so got the first translator's text without waiting, were 1.4% of translated sentences (310 / 21,605).

We also changed how the input is built along the way (from the morning of November 11, the previous sentence is passed as well). Input per sentence grew from about 23 to about 43 tokens, yet slow sentences went from 0.8% on November 10 (40 / 5,227) to 1.3% on November 12 to 14 (288 / 21,605), and fallbacks from 14% to 16%. Even with nearly twice the input, over-budget sentences stayed at 1 to 2% of translated sentences.

Totals since we started counting tokens and time per sentence (from the 2026-11-09 release to November 14, a few hundred sessions in all):

Total
Sentences translated by the local model 30,346
Input tokens 1,157,385
Output tokens 280,687
Time spent translating 4 h 19 min (reading input 2 h 26 min, producing output 1 h 53 min)

Tokens are the units the model counts. We do not record characters. The source length in characters is only used to estimate the budget before translating. Having length and time as a pair for every sentence is what this rule stands on. With time alone, you cannot tell a slow device from a long sentence. With more than 30,000 sentences, there is plenty of data to see how length and time relate sentence by sentence.

Having seen these numbers, we decided to leave the thresholds and coefficients as they are (2026-11-15).

Before release, we checked three scenarios locally. Nothing was played aloud; synthesized speech was fed inside the page.

Scenario Result
Japanese sentences of about 40 characters in quick succession (discrete GPU) No over-budget sentences, no fallback
A slow device (each sentence artificially delayed) Fell back after consecutive misses, every retry was over budget, and the local model was dropped for the session. Subtitles continued with the first translator
A device that speeds up right after falling back Short-sentence retries succeeded in a row and it switched back to the local model

Note: from this version on, "slow sentence" means "over the budget" instead of "over the flat line." Read the numbers before and after that point separately.

5. Discussion

5.1 Why it works for live streams

LiveSub's biggest strength is that it is built for live streams in the browser. A recorded video can be paused, but a live stream does not wait. That is why "never stop the subtitles" is the first constraint, and why we measure how many good translations we can deliver within it.

Live stream audio rarely goes silent, voices sit on top of background music, and speech has a rhythm of peaks and valleys. Long sentences cluster on the peaks, and short reactions cluster in the valleys. The old rule broke on the peaks and never came back. The proportional budget gets through the peaks, and retries bring the model back in the valleys. Using that rhythm itself as the retry signal is what makes this fit live subtitles.

5.2 What we did not do

5.3 Beyond this app

This problem is not unique to LiveSub. Any service that runs a local model in the browser inherits device differences, input lengths, large downloads and a screen that cannot freeze. Summarization and speech recognition have the same shape: time grows with input length, and device speed differs from session to session.

The approach generalizes in this order:

  1. Measure. Know from production data what processing time is proportional to, and how fast each device is
  2. Divide by what it is proportional to. Replace a flat timeout with expected size × the device's unit cost
  3. Calibrate per device. Do not share coefficients across devices; set them per session from measurements right after loading
  4. Build a way back. Pair every rule that gives up with a rule that tries again and returns. Retry in the background, trigger on low load, and cap the number of retries

6. Summary

We moved from "cut by time" to "divide by what time is proportional to, calibrate per device, and build a way back." All of it is shaped by the conditions of live streams in the browser.

The old rule was not a configuration mistake. A flat 1 second and 3 sentences were reasonable numbers for short sentences on fast devices. The problem was that those numbers measured sentence length, not device speed. When a rule feels too strict, check what the threshold is measuring before loosening it. That was the biggest lesson from this change.

The other lesson is to pair every rule that gives up with a way back. An automatic switch that cannot switch back loses the rest of the session to a single peak. The way back took nothing more than recasting the half-open state of a circuit breaker so that subtitles never stop.

The numbers are from LiveSub in production. The coefficients (the ratio from source characters to output tokens, the factor s, k, m, the interval of the fallback signal, and the default device baseline) are something each service has to measure on its own users. If you copy the shape of this approach, you still have to measure the numbers on your own devices and users. We think that is fine. If the measurements, the shape of the formula and the way back are shared, the next person to hit the same wall does not have to start from zero.

We will add newer numbers to this post as they come in.

Built with PLaMo. Local translation in LiveSub uses PLaMo 3 Translate 0.8B by Preferred Networks, Inc. under the PLaMo Community License Agreement (Japanese text).

About this guide: written by the maker of LiveSub. Steps, button names and figures are checked against the LiveSub app as of the date above. If something is wrong, please tell us through the contact page.

Open LiveSub and try it

LiveSub Contact Privacy policy Terms of use Commerce disclosure