ja-dictation-bench v1: A Benchmark for Japanese Dictation, Including Text Refinement

Summary

This benchmark measures Japanese dictation alone: Japanese speech in, Japanese text out. It does not cover English or any other language.

We evaluated four voice input products on ja-dictation-bench v1, a public benchmark that measures whether spoken Japanese comes out as text that can be sent as is. 131 work-related items were read aloud with synthetic voices, and outputs were scored by two LLMs and a human on four layers: meaning, disfluencies and self-corrections, structure, and notation. Overall scores were Typeless 91.3, SayonaraType 89.7, Aqua Voice 57.8 (58.4 with the Whisper setting), and Wispr Flow 57.6; the difference between Typeless and SayonaraType is not statistically distinguishable. About 90% of meaning errors involved content words or terms being substituted or dropped.

The key results are also summarized on the Japanese voice input benchmark page (in Japanese). This report is the full version.

Key findings

1. Measured on 131 items with synthetic speech, default settings and no custom dictionary, the overall scores (0–100) were Typeless 91.3, SayonaraType 89.7, Aqua Voice 57.8 (58.4 when the model is set to Whisper in the settings), and Wispr Flow 57.6.
2. On the same 131 items under the same conditions, the difference between Typeless and SayonaraType (1.6 points in overall score; 11 vs. 13 items with meaning errors) is not statistically distinguishable (McNemar test p = 0.77; 95% interval of the overall-score difference −6.8 to +3.6 points).
3. On the same 131 items under the same conditions, the overall-score differences between the two products Typeless and SayonaraType on one side, and the two Aqua Voice settings and Wispr Flow on the other, were 31–34 points; in all six of these paired comparisons, the 95% interval of the difference did not include 0.
4. Of the 183 meaning-layer (Layer A) errors on the 131 items with synthetic speech (summed across five settings), 162 (about 89%) involved a content word or term being substituted or dropped (rule A1).

Background

This benchmark is about Japanese dictation, that is, turning Japanese speech into Japanese text. Everything below, including the rules and the results, concerns Japanese; it says nothing about English or other languages.

Evaluations of Japanese speech recognition have widely used metrics such as character error rate (CER), which count how accurately the spoken sounds were turned into characters. These metrics measure listening accuracy; they do not measure whether the resulting characters can be sent as text.

Meanwhile, some voice input products now refine the recognized characters with a language model before entering them (for example, Aqua Voice describes its output as "Avalon combined with language model post-processing"; Avalon is the company's own speech recognition model¹). When people use voice input for work, what they want is not a transcript but text in which hesitations and self-corrections have been removed, with the correct kanji and notation, arranged in a readable form. This stage, "turning spoken words into sendable text," cannot be measured with metrics of character accuracy.

ADLIB (a benchmark of technical terminology in Japanese speech recognition) points out that existing speech recognition evaluations tend to be a binary choice between "excessive normalization" and "exact match," which counts legitimate notational variants as errors and flattens differences in how punctuation and numbers are handled (ADLIB). This benchmark starts from the same concern and scores recognition and refinement together, asking whether the text was entered as the writer intended.

Which factors move quality is shown in a separate research note, "The biggest driver of voice input quality is the choice of LLM that refines the text" (in Japanese); the choice of the LLM that refines the text had the largest effect.

Japanese language structure and why voice input is hard

The scoring criteria of this benchmark (formatting rubric v1.0²) were not built by adding symptoms one at a time, but by deriving the issues from the structure of Japanese and from existing research. This section explains the rationale for that design, together with what happens in voice input.

Why we split the task into four layers

If errors in the refinement stage are listed as symptoms (hedges, line breaks, colloquial to written style, notation, homophones, self-corrections, and so on), a new item has to be added each time a new symptom is found, and there is no way to set priorities among items. So we redefined the task from the question "what kind of transformation is turning speech into text in voice input?" This transformation can be split into four parts.

• Layer A, preserving meaning: keep what the speaker said
• Layer B, removing disfluencies: remove elements that exist just in speech
• Layer C, structuring: give the text the structure of written language (bullet lists, line breaks, headings)
• Layer D, notation and style: put the text into written notation and style

For A and B, what to keep and what to remove is uniquely determined by the speaker's utterance (unique). For C and D, the writer has discretion, so there is a range of correct answers (acceptable range). This asymmetry determined the design of the scoring. The unique layers are scored mainly by counting deviations with automated checks; the acceptable-range layers are judged against sets of multiple correct answers and by LLM scoring. In addition, because meaning (A) errors are serious, they act as a gate, and differences among passing outputs are decided by B, C and D.

Layer A: the hierarchy of proposition and modality

Japanese sentences have been analyzed as a hierarchy in which the speaker's attitude (modality) wraps the content of the event (the proposition) from the outside (Masuoka 1991; Nihongo Kijutsu Bunpō Kenkyūkai, ed., 2003).

In Japanese, the speaker's attitudes, such as conjecture, questioning and requesting, gather at the end of the sentence. When voice input refines text, this sentence ending is easily rewritten: if 「原因だと思う」 ("I think it's the cause") becomes 「原因です」 ("It's the cause"), or 「起きてると思う？」 ("Do you think it's happening?") becomes 「起きてると思う。」 ("I think it's happening."), the meaning changes even though the words remain. Also, because subjects and objects are often left unsaid, the refinement stage sometimes supplies a subject or adds words that were not spoken.

This hierarchy gives the boundary between what the refinement stage must keep and what it may change. We treated just the outermost layer, politeness, as style (Layer D), and everything inside it as meaning (Layer A), to be preserved. A change that crosses utterance types (declarative, question, request, etc.) is an error; a difference in form within the same type (「して」 "do it" vs. 「してください」 "please do it") is accepted as style. Layer A has 12 rules: content words (A1), proper nouns (A2), numbers (A3), polarity (A4), voice and tense (A5), strength of conjecture (A6), utterance type (A7), connectives (A8), contrast (A9), question form (A10), additions (A11), and omissions (A12).

Layer B: phenomena that exist just in speech (CSJ transcription tags)

Speech has phenomena that written language does not: fillers such as 「えっと」 and 「あの」 ("um," "uh"), self-corrections such as 「来週、いや、再来週」 ("next week, no, the week after"), and repetitions of the same word. In voice input, these are either written out as they are or turned into other characters at the recognition stage (for example, 「えー」 "uh" becomes the Latin letter "A"). With self-corrections, leaving the earlier word in changes the content.

We did not decide by ourselves what counts as a phenomenon that exists just in speech; instead, we built on the transcription specification of the Corpus of Spontaneous Japanese (CSJ) (Koiso et al., CSJ). CSJ has a system of tags for fillers (F), word fragments from self-corrections or hesitations (D), self-corrections of particles, numbers and the like (D2), non-lexical sounds, and so on. We used these as candidates for "what to remove when turning speech into writing," resulting in eight rules: fillers (B1), self-corrections (B2), correction markers (B3, 「いや」 "no," 「あ、違う」 "oh, wait"), repetitions (B4), abandoned clauses (B5), self-directed speech (B6), non-lexical sounds (B7), and recognition errors (B8).

Self-corrections and contrasts must be distinguished, however. 「精度じゃなくて速度の話」 ("this is about speed, not accuracy") is a contrast, and deleting the earlier word changes the meaning (A9). 「施策の精度、じゃなくて優先度」 ("the accuracy of the initiative, I mean, the priority") is a self-correction, and just the later word is kept (B2). When it is unclear, we err on the side of keeping.

Layer C: structure (speech is a line, writing is a plane)

Speech flows in a single line along time. Writing is laid out as a plane on the page through bullet lists and line breaks. Speakers try to convey this planar structure with their voice alone, signaling enumeration with words such as 「まず」 ("first"), 「あと」 ("also") and 「〜と、〜と」 ("X and, Y and"). The refinement stage needs to pick up these cues and turn the linear utterance into a structure the reader can take in at a glance.

For this layer, our principle is "words and attitude as spoken, structure in a readable form." For enumerations of three or more items, or enumerations the speaker counted out, the expected output is a bullet list (C1; consistent with the guideline in the Agency for Cultural Affairs' "Approach to Writing Official Documents" that "three or more pieces of information go in a bullet list"). Conversely, over-formatting, such as adding a heading or a one-item bullet list to a short utterance, is penalized (C8). Structure burdens the reader both when there is too little and when there is too much.

Layer C has few properties specific to Japanese; much of it is shared across languages. That speech is linear and writing is planar is not limited to Japanese. Its weight in this benchmark is accordingly small, at 10.

Layer D: notation standards (Approach to Writing Official Documents and ADLIB)

Japanese is written with a mix of kanji, hiragana, katakana and Latin letters, and the same word can have several correct spellings (「届出」 and 「届け出」, both "notification"; 「2」 and 「二」, both "two"). At the same time, Japanese has many words with the same sound (homophones), and at the stage where voice input chooses kanji from the reading, errors occur such as 「講評」 (kōhyō, "critique/review") becoming 「公表」 (kōhyō, "public announcement"), or 「決裁」 (kessai, "managerial approval") becoming 「決済」 (kessai, "payment settlement"). Both facts hold at once: notation can have several correct answers, and some kanji are uniquely determined by context.

We did not create our own notation standard; we aligned with the Council for Cultural Affairs' "Approach to Writing Official Documents" (2022). Its principles include: Arabic numerals for numbers in horizontal writing; kanji for 兆, 億 and 万 (trillion, hundred million, ten thousand); hiragana for 「か月」 and 「か所」 (the counters for months and places); kana for auxiliary verbs, formal nouns and conjunctions; and the long-vowel mark for long vowels. Layer D is split into a side where the correct answer is unique (homophones D1, English notation D2, numbers D3, units and symbols D4) and a side where anything within the range is correct (politeness D5, request form D6, sentence-final particles D7, contractions D8, replacing hedge words D9, writing kanji words in kana D10, punctuation D11, long vowels D12, consistency of style D13, spaces around Latin letters D14, abbreviations D15).

For words that mix in English, there is no uniform norm for whether loanwords are written in katakana or in Latin letters. For this scoring we borrowed ADLIB's classification and split such words into those for which Latin letters alone are correct (exact; e.g., CSV, JSON, Notion, 1on1) and those for which both Latin letters and katakana are correct (flexible; e.g., デプロイ and deploy) (ADLIB).

Punctuation norms are also loose, and the position of commas differs from writer to writer. We therefore do not penalize the position of commas or the presence or absence of a final period; we penalize them just when the interpretation changes or the text becomes concretely hard to read (D11, D14). Whether something is a question is judged as meaning under A10, not by the presence or absence of "？".

Output length, structure and genre, and consistency within an output

The correct style changes with the length and structure of the output. For a short single sentence, it is natural to keep the spoken attitude (the form of the sentence ending, or discourse markers such as 「まあ」 "well"), while for longer, structured output, written norms take priority. However, a rule that "widens the acceptable range with length" leaks toward allowing summarization and paraphrasing in long utterances. So, taking the view that what actually changes is whether the style is consistent within the output, we made it a rule of consistency within an output (D13) plus a rule that "short sentences stay as spoken." Whether an item counts as short is not determined by counting periods in the scored output; instead, each item was flagged in advance based on its expected output (so that an output cannot change the judgment simply by adding periods). The acceptable range moves with length just for C and D; A and B do not move with length.

Because the correct style depends on where the text is entered, we split genres by output format rather than by app: chat (Slack, LINE, Teams, etc.), email, notes (memos, meeting minutes, to-dos), and instructions to AI. Aspects left to personal style, such as the amount of line breaks, heading symbols, and how large numbers are written (100万 vs. 1,000,000), are either not penalized or penalized lightly.

Summary: properties of Japanese and how the evaluation handles them

Evaluation design

This section describes how the layers and rules from Section 4 are handled in the items, the audio, the scoring and the measurement conditions.

Item set

The 131 items come from three sources. Every item has a genre, a short-sentence flag, an expected output (including lower-scored alternatives and examples of errors), and the rules that must be scored for that item (case rules).

By genre there are 60 chat, 46 notes, 13 email and 12 instructions to AI items; 111 items carry the short-sentence flag and 20 do not. Expected outputs were written on the principle of "words and attitude as spoken, structure in a readable form." Outputs of existing products were used as clues for finding issues, but never as correct answers. Mixing a particular product's habits into the correct answers would turn the benchmark into a yardstick of how similar outputs are to that product.

Why we use synthetic speech

The audio uses six voices from three engines (VOICEVOX's Shikoku Metan and Kurono Takehiro, Microsoft Azure Neural TTS's Nanami and Keita, Amazon Polly's Kazuha and Takumi), all synthetic. No real person's voice is included. We publish 786 files (131 items × 6 voices); products were measured with one voice per item. Voices were assigned with a fixed random seed (20261102) so that the six voices are evenly distributed within each combination of item source and genre (21–22 items per voice). The 131 files used for measurement total about 26 minutes, with each file 1.4–60.2 seconds long (median 12.0 seconds).

We use synthetic speech because publishing real people's voices requires speaker consent and protective measures, because anyone can re-measure with the same audio, and because the audio can be distributed under an "evaluation use (no training and no voice cloning)" license. On the other hand, synthetic speech is clear and consistent in pronunciation, so recognition of proper nouns, English terms and homophones tends to come out better than in practice, and fillers and self-corrections are scripted readings, not genuine hesitations. This mainly affects the recognition part of Layers A and B. Limitations are described in Section 8.

Scoring criteria

Scoring follows a rubric of 4 layers and 46 items. The weights are A 50, B 25, C 10 and D 15.

1. The rules to be scored are fixed before scoring. We score just the 22 common rules that any output can violate (A1–A12, B1, B4, C1, C4, C8, D3, D4, D6, D13, D14) plus the case rules for that item. With no violation a rule gets 2 points; "not applicable" cannot be chosen.
2. Each rule gets 0–2 points. If a rule applies at several places, the lowest score is taken, so that many correct fixes do not dilute a single error.
3. The layer score is the mean of the scores of the rules in that layer (0–2).
4. Gate: if any Layer A rule scores below 2, the item fails (meaning error) and scores 0.
5. The score of a passing item is 100 × Σ(layer weight × layer score / 2) / 100. The overall score is the mean over the 131 items (failed items count as 0).

Scoring was done in three stages. First, automated checks match words that must be kept (must), words that must not appear (forbid), English notation, number notation, bullet-list form and so on (mainly A1–A3, B1–B3, D1–D4, C1). Next, two LLM graders score independently, without sharing history, using the same input, the same output, the same automated results and the same grading guide (v0.6.2) (covering what automated checks cannot see, such as the strength of conjecture, utterance type and the handling of self-corrections). The graders were not shown product names. Where the two scores agreed on a rule, it was finalized as is; just the rules on which they disagreed were decided by a human (AI drafted the decisions, and the author finalized them item by item).

The published decisions (results/judgments.jsonl) include 198 rules on which the graders disagreed and a human decided (147 combinations out of 5 settings × 131 items). Of these, 145 were decided as no penalty. The most common reasons were not counting the same single fix under several rules (73 cases) and not treating notational differences as meaning errors, such as alternative spellings of a proper noun with the same reading when no dictionary entry specifies one (38 cases). Most of the disagreements came not from how good or bad the outputs were, but from how the rules were applied (the yardstick side). The decision policy is listed under "Decision policy" in the grading guide, and the per-item decisions with short reasons are also published. For the rules on which the graders disagreed, AI drafted the decisions and the author finalized them item by item.

Measurement conditions

We measured the paid version (Pro; SayonaraType version 1.3.0) of each product with default settings and no dictionary entries, in November 2026. The things we changed were the microphone (the virtual input BlackHole 2ch), the hotkey, and the setting to not use data for training. To match the state of a new user and to keep conditions the same across the 131 items, we cleared any previously learned words, writing styles and history. History that accumulated during measurement was treated as part of the product's behavior.

Audio was played into the virtual input one company at a time, item by item; recording was started and stopped with each product's hotkey, and the entered text was saved. Text was considered fully entered once it had not changed for 5 seconds, and we waited up to 2 minutes after speech ended. Products were given the audio alone; the context attached to items (the preceding exchange for the 4 "with context" items) was not given. Speed was also recorded but is not used in these results.

SayonaraType was measured three times, and the third run was taken as the official run. The third run used the public version 1.3.0 of the app, with the configuration after a change made at 17:23 on November 4, 2026 to the server's fallback path (the path used when normal processing does not return). The formatting model, instructions and dictionary (none) were the same in all three runs. The two earlier runs (with the fallback path before the change) are not included in the results, because the number of items that went through the fallback path exceeded the condition for an official run (2 items or fewer) (6 items and 11 items). The fallback path was changed after these two runs (the first run had been through automated scoring). The condition for an official run was set before looking at the results. In the adopted run, 1 item went through the fallback path. The measurement reflects the configuration at that time; server settings have changed in later updates (on November 9, 2026, the wait time before switching to the fallback path was shortened).

The two Aqua Voice settings are reported as two rows, one for each model a user can choose in the settings screen. Measured under the same conditions, the outputs of the two settings were character-for-character identical on 102 of the 131 items (ignoring differences in line breaks)³.

Data card

Items

131 items (60 v0 core, 36 formatting items, 35 new items). Genres: 60 chat, 46 notes, 13 email, 12 instructions to AI

Audio

All synthetic speech (6 voices from VOICEVOX, Microsoft Azure Neural TTS and Amazon Polly). No real person's voice is included. The 131 files used to measure products total about 26 minutes (1.4–60.2 seconds each)

Scoring

Four-layer rubric v1.0 (46 items). Automated checks plus double scoring by two LLMs; where they disagreed, a human decided (decisions and reasons are published)

License

Text CC BY 4.0, code MIT, audio for evaluation use

Use for training

Please do not use the items or expected outputs for training (every file contains a canary string)

Results

Overall score and meaning errors

Overall score (0–100)

• Typeless91.3
• SayonaraType89.7
• Aqua Voice (Avalon 1.5)57.8
• Aqua Voice (Whisper)58.4
• Wispr Flow57.6

Layer A errors (of 131 items; fewer is better)

• Typeless11
• SayonaraType13
• Aqua Voice (Avalon 1.5)54
• Aqua Voice (Whisper)53
• Wispr Flow52

The overall score ranges from 0 to 100 and is the mean over 131 items, with items that have a Layer A (meaning) error scored as 0.

Scores by layer

Layer A, meaning (0–100)

• Typeless99.2
• SayonaraType99.1
• Aqua Voice (Avalon 1.5)96.6
• Aqua Voice (Whisper)96.6
• Wispr Flow96.8

Layer B, disfluencies and self-corrections (0–100)

• Typeless99.0
• SayonaraType98.7
• Aqua Voice (Avalon 1.5)93.3
• Aqua Voice (Whisper)91.6
• Wispr Flow82.0

Layer C, structure (0–100)

• Typeless99.6
• SayonaraType99.5
• Aqua Voice (Avalon 1.5)98.1
• Aqua Voice (Whisper)98.3
• Wispr Flow98.2

Layer D, notation (0–100)

• Typeless99.7
• SayonaraType99.6
• Aqua Voice (Avalon 1.5)96.9
• Aqua Voice (Whisper)97.0
• Wispr Flow97.6

The layer scores are the mean, before the gate, of the layer scores (0–2) over the 131 items, rescaled to 0–100. Layer A scores look high at 96–99 because each item averages 12 rules; even if one rule scores 0, the item's Layer A score drops to 1.83. The gate is decided not by this mean but by whether any rule scores below 2.

Scores by item type

Scores by type were computed with the same formula as the overall score (the mean of item scores within that type, with failed items scored as 0). Numbers in parentheses are Layer A error counts. Because the rules scored differ by type, these tables are for comparing products within the same type; difficulty cannot be compared across types. For types with few items (13 email items, 12 instructions-to-AI items), a difference of one item moves the score substantially.

By item source:

By genre:

By short-sentence flag:

On the short utterances that test refinement (the 36 formatting items), the scores of all five settings clustered at 90–100, and every setting had 0–2 items with meaning errors. On the v0 core, with its embedded target words, and on the 35 new items, Layer A error counts diverged widely.

Testing differences between products

We compared products on the same set of 131 items (paired comparison). Meaning errors (Layer A errors) are counted from the final per-item decisions.

• Difference in meaning errors: McNemar test (exact two-sided binomial test on the items where just one of the two has an error)
• Difference in overall score: the mean of per-item score differences (failed items scored as 0), with a 95% interval from a bootstrap that resamples items (10,000 resamples, random seed 20261104, percentile method)

On 131 items, the difference between Typeless and SayonaraType is not statistically distinguishable. For meaning errors, the items where just one of the two erred were 5 and 7, and the 95% interval of the overall-score difference includes 0. With 131 items and one voice per item, the difference between these two products (2 items, 1.6 points) cannot be resolved. The difference between the two Aqua Voice settings likewise includes 0.

For the six pairs between Typeless and SayonaraType on one side and the two Aqua Voice settings and Wispr Flow on the other, the McNemar p was below 0.001 in every case (and stays below 0.05 even after correcting for multiple comparisons across the eight comparisons), and the 95% interval of the overall-score difference does not include 0. The top two products preserved meaning on roughly 90% of the items measured, while the other three settings had meaning errors on roughly 40% of the items.

Observations: what the meaning errors were, and how hesitations were handled

• There were 183 meaning errors in total across the five settings, of which 162 (about 89%) involved A1 (a content word or term substituted or dropped). Most were industry terms or proper nouns becoming a different word with a similar sound (Section 7). Besides A1, there were additions (A11), connectives (A8), strength of conjecture (A6) and question form (A10).
• Differences in Layer B scores came mainly from the handling of fillers (B1) and from the pre-correction word remaining (B2).
• Typeless and SayonaraType also had meaning errors on 11 and 13 items. Besides substituted industry terms, these were added words that were not spoken, dropped short preambles, and question endings turned into declaratives (Section 7).

Error analysis

Below are examples where the outputs of the five settings diverged, grouped by layer and rule. "Input sentence" is the script read aloud with synthetic speech. Each product's output is shown exactly as entered during measurement (trailing whitespace removed; line breaks shown as ↵). Decisions and scores are from the published results/judgments.jsonl. English glosses are added for readers who do not read Japanese; the outputs themselves are left untranslated.

Layer A: meaning errors

Substituted terms (A1). An industry term becomes a different word with a similar sound. Example from v0-054 (chat).

Input sentence: この件は決裁を取ってから進めます。稟議は部長承認までで足りるので、今週中に起案して、来週の経営会議には結果報告だけにしたいです。 (We'll proceed with this once we have managerial approval. Since the ringi [formal approval request] needs to go no further than the department head, I'd like to draft it this week and just report the outcome at next week's management meeting.)

When 「稟議」 (ringi, "formal approval request") becomes 「便宜」 (bengi, "convenience") or 「倫理」 (rinri, "ethics"), the sentence still looks well formed, but the recipient cannot tell what the approval is about. The automated checks detected this as the absence of the must-keep word 「稟議」.

Question endings (A10). The speaker's attitude gathers at the sentence end. This is the last sentence of v1-d35 (instructions to AI, long utterance).

Input sentence (last sentence): あ、それと、この問題って本番環境でも起きてると思う？ (Oh, and also, do you think this problem is happening in production too?)

「起きてると思う？」 ("Do you think it's happening?") is a question asking for the listener's assessment, but ending with 「思う。」 ("I think it's happening.") turns it into a statement of the speaker's own opinion. In an instruction to AI, the question the speaker wanted answered is left unanswered. Four settings turned this ending into a declarative.

Adding words that were not spoken, and dropping short words (A11, A1). Adding something that was not said and cutting something that was said are both meaning errors.

Input sentence for v1-d01 (instructions to AI): えっと、このシーエスブイを読み込んで、ユーザーIDごとに、あの、購入回数と合計金額を集計して、ジェイソンで返してください。カラム名は英語で。 (Um, load this CSV, and for each user ID, uh, total up the number of purchases and the total amount, and return it as JSON. Column names in English.)

Input sentence for ext:21 (chat): 了解です、明日送ります。 (Got it, I'll send it tomorrow.)

In v1-d01, the output that added 「お願いします」 ("please"), which was not spoken, was counted as an A11 (addition) error. In ext:21, the acknowledgment 「了解です」 ("Got it") was dropped, removing the sentence's function as a reply to the other person, so it is an A1 error. In both cases the words look tidy, which makes these errors hard for readers to notice.

Layer B: disfluencies and self-corrections

Fillers turned into other characters (B1). Example from ext:17 (chat).

Input sentence: 今回の変更は、えー、既存のユーザーには影響しないです。 (This change, uh, won't affect existing users.)

When 「えー」 ("uh") remains as the Latin letter "A," readers do not recognize it as a filler and read it as part of the content (something called "A"). Under the decision policy in the grading guide, even one remaining garbled filler gives B1 = 0.

Pre-correction words remaining (B2, B3).

Input sentence for ext:28 (chat): 見積もりは三件、あ、違う、四件届いています。 (We've received three, oh, wait, four quotes.)

Input sentence for ext:29 (chat): 担当は田中さん、じゃなくて佐藤さんにお願いします。 (Please assign it to Tanaka-san, I mean, Sato-san.)

ext:29 sits on the boundary between "self-correction and contrast" described in Section 4. The speaker started to say 「田中さん」 (Tanaka-san) and retracted it, so the expected output contains 「佐藤さん」 (Sato-san) alone. Keeping 「田中さんじゃなくて」 ("not Tanaka-san") makes it read as if Tanaka-san had been a candidate. Because meaning (Layer A) is preserved, this was not gated; it was penalized in Layer B.

Layer C: structure

Enumerations of three or more items (C1, C4). Example from v1-d08 (instructions to AI).

Input sentence: テストケースを足してほしい。正常系と、えっと、境界値と、あ、あと異常系も。異常系は空文字と、ヌル、じゃなくてアンディファインドの二つで。 (I'd like you to add test cases. Normal cases and, um, boundary values and, oh, also error cases. For error cases, two: empty string and null, I mean, undefined.)

(Gloss: in the Aqua Voice outputs, 「正常系」 "normal cases" and 「異常系」 "error cases" became the non-words 「正乗形」 and 「異乗形」, and "undefined" became the katakana 「アンリファインド」; in the Wispr Flow output, 「あと異常系も」 "also error cases" became 「後追い状況も」 "also the follow-up situation.")

The expected output is the three items as a bullet list. An output that uses a bullet list but has no blank line separating it from the preceding line (C4 = 1), and an output that lists the items as running sentences (C1 = 1), were both treated as alternative answers that pass with a lower score. Because the structure layer has an acceptable range, it is not gated as long as meaning is preserved.

Layer D: notation

Homophones (D1). Example from v0-035 (chat).

Input sentence: 防災訓練を10月の第2週に実施します。避難経路の確認と安否確認システムのテスト配信を行うので、全員参加でお願いします。終了後に講評の時間を取ります。 (We will hold a disaster drill in the second week of October. We'll check evacuation routes and send a test message from the safety-confirmation system, so please make sure everyone participates. Afterward, we'll set aside time for a review.)

「講評」 (kōhyō, "review/critique") and 「公表」 (kōhyō, "public announcement") share the same reading, and the context determines 「講評」 (the unique side, D1). 「安否」 (anpi, "safety") becoming 「暗記」 (anki, "memorization") is a substituted term, which was gated as an A1 error. Differences in the amount of line breaks (Typeless's blank lines) were not penalized.

English notation (D2), and formal sentence endings that were not spoken (D6). Example from v0-032 (chat).

Input sentence: 360度評価の結果をもとに、マネージャー層向けの研修を企画します。特に1on1の質にばらつきがあるので、フィードバックの型を揃えたいです。対象は課長以上です。 (Based on the results of the 360-degree review, we'll plan training for managers. In particular, the quality of 1on1s varies, so I'd like to standardize the feedback format. The training is for section managers and above.)

"1on1" is a word for which the Latin-letter spelling alone is correct (exact). The katakana 「ワンオンワン」 is a D2 error, and in this item it was also detected under A1 as the absence of the must-keep word "1on1." Typeless's 「揃えたいと考えています」 (a more formal "we are considering standardizing") added a formal sentence ending that was not spoken in a chat message, changing the speaker's tone of familiarity, so it was given D6 = 1. Wispr Flow's 「フィードバックの形」 (katachi, "shape," written with a different kanji from 型 kata, "format") was not penalized, as a spelling with a plausible reading that still makes sense.

Alternative notations are not counted as errors. In v0-022, Typeless's output that wrote 「ABテスト」 as 「A/B テスト」, and outputs that put spaces around Latin letters (「社内の VPN 接続」, "in-house VPN connection"), were not penalized, as alternative spellings of the same word (A1 = 2, D14). On the other hand, the output that wrote 「統計的に有意な差」 ("statistically significant difference") as 「優位な差」 (yūi, "superior difference," same reading as 有意 yūi, "significant") (Aqua Voice with the Whisper setting) is D1 = 0, an error in kanji that context uniquely determines. We score the parts with a range of correct answers separately from the parts with a single correct answer.

Limitations

Synthetic speech. All audio is synthetic. Because pronunciation is clear and consistent, recognition results tend to come out better than with real speech, and fillers and self-corrections are scripted readings. For reference, we have internal measurements comparing the natural voice of one speaker (reading a script in a quiet room) with two synthetic voices different from the six published voices, on the same 60 items and the same configurations. With automated scoring across 19 configurations, the difference in target-word accuracy (natural voice − synthetic voice) was −2.5 to +0.3 points (mean −0.7) against one synthetic voice and −0.4 to +2.8 points (mean +1.1) against the other. Within this range, the natural voice was not clearly lower than the synthetic voices, but these values come from one speaker reading aloud and cannot be generalized to spontaneous speech, individual differences between voices, noise, or differences in microphones.

Number of items. With 131 items and one voice per item, a difference of 2 items and 1.6 points cannot be resolved (Section 6). Differences due to voice are handled solely by assigning the six voices evenly and looking at the mean.

No dictionary, default settings. We did not measure quality when a dictionary is registered, or after a product has learned the user's words and writing style through continued use. The effect of dictionaries may differ by product, so results could change under dictionary conditions.

Versions at the time of measurement. Each product was measured in its version as of November 2–5, 2026. Product updates change results.

Scoring by LLMs. What automated checks cannot see was scored by two LLMs, and rules on which they disagreed were decided by a human. The graders were not shown product names, but the decisions were made on the author's side. The author is the developer of one of the products measured (SayonaraType) (conflict of interest). For this reason, all per-item decisions and the reasons for human decisions are published, and we accept challenges (Section 9).

How the configuration was chosen. The configuration of SayonaraType's public version (the LLM that refines the text and its instructions) was chosen based on scores from internal experiments that used the same 131 items as this benchmark. When choosing, we split the items into two halves, selected candidates on the scores of one half, and checked that scores did not drop on the other. The examples in the instructions do not use words or sentences from this benchmark's items. However, some of the triggers for revising the instructions (whether to turn short enumerations into bullet lists, and how to handle self-corrections) were found in outputs on this benchmark's items. Because the configuration was nonetheless chosen on scores from the same items, SayonaraType's score may be higher than it would be if measured on new items.

Properties not covered. The following are not scored in v1.

• Speaker intent that is carried in the audio but not left in the text, such as tone of voice and pauses
• Dialects and accents, noisy environments, quiet speech, spontaneous speech
• Dictation longer than the long utterances (up to 60.2 seconds), and "document" genres such as proposals and blog posts
• Adaptation to the speaker's style (the effect of dictionaries and memory)
• Behavior when the context attached to an item (the preceding exchange) is given to the product (even for the 4 "with context" items, products were given the audio alone)
• Of the 46 rubric items, the 8 that were in neither the case rules of the v1 131 items nor the common rules (B5 abandoned clauses, B7 non-lexical sounds, B8 recognition errors, C7 distinguishing enumerations from running text, C11 headings for units of meaning, D5 politeness, D12 long vowels, D15 abbreviations). Also, rules that appear solely as case rules (e.g., B2 for self-corrections, in 24 items) were not scored on other items
• Items specifically designed to test misreadings caused by word segmentation (Japanese is written without spaces between words)

How to reproduce

The items, expected outputs, rubric, grading guide, scoring code and results are on GitHub (https://github.com/sayonaratype/ja-dictation-bench ), and the audio is on Hugging Face (https://huggingface.co/datasets/sayonaratype/ja-dictation-bench ; gated access).

Checking the published results. results/judgments.jsonl contains the final decision for each product (setting) and each item (pass/fail, score, per-layer scores, per-rule scores, human decisions and reasons). Joining it with data/cases.jsonl on the item ID lets you recompute this report's overall scores, per-layer scores, per-type scores and paired comparisons.

Measuring another product under the same conditions.

1. Request access to the audio on Hugging Face, and prepare the audio for the voice assigned to each item in data/voice_assignment.csv.
2. Put the product to be measured in its default settings with no dictionary entries, and clear any previously learned words, writing styles and history. The things you may change are the microphone (virtual input), the hotkey, and the setting to not use data for training.
3. Play the audio into the virtual input one item at a time, start and stop recording with the product's hotkey, and save the entered text for each item. Consider the text fully entered once it has not changed for 5 seconds, and wait up to 2 minutes after speech ends. Replay any item for which nothing was entered.
4. Put the outputs in JSON Lines with one item per line ({"case_id": …, "text": …}) and run python -m harness score outputs.jsonl at the repository root to get the automated scores (Python 3.10 or later, standard library alone).
5. To compare with the published results, after automated scoring, run double scoring with two LLMs following the grading guide (docs/judge-guide.md, v0.6.2), and decide the rules on which they disagree according to the guide's "Decision policy." The request text and response format can be built with functions in harness (judge.request_payload, judge.SCHEMA, etc.). With automated scoring alone, Layers B–D have checks for a subset of items, so the layer scores cannot be compared with the double-scored results.

Details of the measurement procedure are in docs/method.md.

Challenges. If you disagree with a score or a human decision, please open an Issue on GitHub with the item ID, the rule and your reason. When challenges of the same type accumulate, we will spell them out as separate rules when we version up the rubric.

Future work

In the next version, we will add natural voices from third parties and re-measure the difference from synthetic speech.

From the next version onward, we will use separate items, not included in this benchmark, for choosing product configurations and revising instructions. This benchmark will be used solely to measure configurations after they have been chosen.

The rules of this benchmark were derived from the hierarchy of Japanese predicates, the transcription system of a spoken-language corpus, and an official notation standard. We believe the approach of deriving evaluation criteria from the structure of a language can be applied to other languages as well.

How to cite

If you use this benchmark or this report, please cite it as follows.

Kumagai, Keiji and SayonaraType Research Team (2026). "ja-dictation-bench v1: A Benchmark for Japanese Dictation, Including Text Refinement." CACTUS TOKYO Inc. https://github.com/sayonaratype/ja-dictation-bench

When redistributing the data (items, expected outputs, rubric, results), please include "ja-dictation-bench by CACTUS TOKYO Inc., CC BY 4.0" as the CC BY 4.0 attribution. The BibTeX entry is as follows.

@misc{kumagai2026jadictationbench,
  title        = {ja-dictation-bench v1: A Benchmark for Japanese Dictation, Including Text Refinement},
  author       = {Kumagai, Keiji and {SayonaraType Research Team}},
  year         = {2026},
  howpublished = {\url{https://github.com/sayonaratype/ja-dictation-bench}},
  note         = {Version v1. CACTUS TOKYO Inc.}
}

References

• 益岡隆志（1991）『モダリティの文法』くろしお出版 [Takashi Masuoka, The Grammar of Modality]（岡部による要約を参照：岡部「日本語のモダリティ研究の階層論」千葉大学『語文論叢』28 [summary by Okabe consulted: Okabe, "Hierarchical theories in the study of Japanese modality," Gobun Ronsō 28, Chiba University] https://opac.ll.chiba-u.jp/da/curator/900116938/gobun28_okabe.pdf）
• 日本語記述文法研究会編（2003）『現代日本語文法4 モダリティ』くろしお出版 [Group for Descriptive Japanese Grammar (ed.), Contemporary Japanese Grammar 4: Modality]（CiNii Books BA64614367）
• 小磯花絵・間淵洋子・西川賢哉・斎藤美紀・前川喜久雄「転記テキストの仕様 Version 1.0」『日本語話し言葉コーパス』国立国語研究所 [Koiso et al., "Specification of Transcribed Text, Version 1.0," Corpus of Spontaneous Japanese, National Institute for Japanese Language and Linguistics]（文書に刊行年の記載はなく、PDF の作成日は 2004 年 3 月 29 日 [no publication year is given in the document; the PDF creation date is March 29, 2004]） https://clrd.ninjal.ac.jp/csj/manu-f/transcription.pdf
• 文化審議会（2022）「公用文作成の考え方（建議）」2022 年 1 月 [Council for Cultural Affairs, "Approach to Writing Official Documents (Recommendation)," January 2022] https://www.bunka.go.jp/seisaku/bunkashingikai/kokugo/hokoku/pdf/93731901_01.pdf
• holotherapper（2026）「世界初の日本語音声認識ベンチマークを作ったよん」Zenn、2026 年 4 月 14 日（ADLIB の解説） [holotherapper, "I made the world's first Japanese speech recognition benchmark," Zenn, April 14, 2026 (an explanation of ADLIB)] https://zenn.dev/holotherapper/articles/adlib-asr-bench ／ ADLIB https://github.com/holotherapper/adlib

SayonaraType (one of the products measured): https://sayonaratype.com/ja/ (in Japanese)

Notes

1. Official description by Aqua Voice (checked 2026-11-05). It is recorded with the source URL under "Record of how Aqua Voice was measured" in docs/method.md of the public benchmark. ↩
2. Formatting rubric v1.0 (docs/rubric.md) and grading guide v0.6.2 (docs/judge-guide.md). https://github.com/sayonaratype/ja-dictation-bench ↩
3. We did not measure how much the same model varies when measured twice, so we cannot tell whether the 29 items that differed reflect the difference in models or run-to-run variation (docs/method.md). ↩

Versions

• v1　2026-11-05　Public benchmark v1 released
• v1　2026-11-09　Research report published