Benchmark
ja-dictation-bench v1: A Benchmark for Japanese Dictation, Including Text Refinement
We evaluated four voice input products on ja-dictation-bench v1, a public benchmark of Japanese dictation (Japanese speech to Japanese text) that measures whether spoken Japanese comes out as text that can be sent as is. 131 work-related items were read aloud with synthetic voices, and outputs were scored by two LLMs and a human on four layers: meaning, disfluencies and self-corrections, structure, and notation. Overall scores were Typeless 91.3, SayonaraType 89.7, Aqua Voice 57.8 (58.4 with the Whisper setting), and Wispr Flow 57.6; the difference between Typeless and SayonaraType is not statistically distinguishable. About 90% of meaning errors involved content words or terms being substituted or dropped.
Datasets
A benchmark for Japanese voice input (Japanese speech to Japanese text) that measures, with 131 work-related items and a four-layer rubric, whether spoken Japanese is entered as text that matches what the writer intended.
About
Research and evaluations on voice input and language by the makers of SayonaraType (CACTUS TOKYO Inc.).