yomidevs
yomitan
BlogDocsChangelog

Speed up tokenization for heavy workflows

#2251
Comparing
ShanaryS:tokenize-speedup
(
af1d8e5
) with
master
(
e99a827
)
CodSpeed Performance Gauge
0%
Untouched
5
Ignored
1

Benchmarks

Passed

japanese transformations-full (n=212)
benches/japanese-language-transformer.bench.js::japanese language transformer::basic tests
CodSpeed Performance Gauge
0%
22.4 ms22.4 ms
japanese transformations (n=78)
benches/japanese-language-transformer.bench.js::japanese language transformer::basic tests
CodSpeed Performance Gauge
0%
8.3 ms8.3 ms
spanish transformations (n=97)
benches/spanish-language-transformer.bench.js::spanish language transformer::basic tests
CodSpeed Performance Gauge
0%
6.4 ms6.4 ms
Translator.prototype.findKanji - (n=3)
benches/translator.bench.js::Translator
CodSpeed Performance Gauge
0%
3.8 ms3.8 ms
Translator.prototype.findTerms - (n=45)
benches/translator.bench.js::Translator
CodSpeed Performance Gauge
0%
113.8 ms114 ms

Ignored

Loop 1 through 100,000
benches/canary.bench.js::Canary
Ignored
CodSpeed Performance Gauge
0%
4.2 ms4.2 ms

Commits

Click on a commit to change the comparison range
Base
master
e99a827
-0.02%
use LRU cache for tokenize
e93838a
4 days ago
by ShanaryS
0%
fix refactor bug
6361c03
3 days ago
by ShanaryS
+0.01%
maintain separation between parse results
ae838ff
2 days ago
by ShanaryS
-0.05%
add array index to results
af1d8e5
1 day ago
by ShanaryS
© 2025 CodSpeed Technology
Home Terms Privacy Docs