pref-js - Branch - kamiazya/web-csv-toolbox

perf: JavaScript parser performance improvements

#614Merged

Comparing

pref-js

(

6926119

) with

main

(

8adf5d9

)

×4.2

Improvements: 41

Untouched: 24

Skipped: 8

Benchmarks

Skipped (8)

Improved

parseString engine:stable (50 rows)

benchmark/main.ts

×4.2

2,550.4 µs603 µs

Engine comparison: mainThread (500 rows)

benchmark/main.ts

×3.6

30.9 ms8.6 ms

parseString.toArraySync(50 cols, 50 rows, quoted)

benchmark/main.ts

×3.6

18.6 ms5.2 ms

Line ending: CRLF (1000 rows)

benchmark/main.ts

×3.4

18.5 ms5.5 ms

parseString.toArraySync(100 cols, 50 rows, quoted)

benchmark/main.ts

×3.4

37.2 ms11.1 ms

parseString.toArraySync(10 cols, 50 rows, quoted)

benchmark/main.ts

×3.2

3.7 ms1.2 ms

Low-level: CSVLexer only (1000 rows)

benchmark/main.ts

×3

17.6 ms5.8 ms

parseString.toArraySync(100 cols, 50 rows, unquoted)

benchmark/main.ts

×2.9

20.1 ms6.8 ms

parseString.toArraySync(50 cols, 50 rows, unquoted)

benchmark/main.ts

×2.9

10.1 ms3.5 ms

parseString.toArraySync(10 cols, 50 rows, unquoted)

benchmark/main.ts

×2.6

2,133 µs822.9 µs

Data transformation: type conversion (100 rows)

benchmark/main.ts

×2.4

3.2 ms1.4 ms

parseStringStream: medium (1000 rows)

benchmark/main.ts

×2.1

171.4 ms81.4 ms

Stream approach: parseBinaryStream (100KB)

benchmark/main.ts

×2.1

346.1 ms164.7 ms

parseString.toArraySync(50 rows)

benchmark/main.ts

×2

2.7 ms1.3 ms

Data transformation: no conversion (100 rows)

benchmark/main.ts

+98%

2.6 ms1.3 ms

parseBinary.toArraySync(50 rows)

benchmark/main.ts

+98%

1,060.2 µs536.8 µs

Queuing: medium (1000 rows) - default HWM

benchmark/main.ts

+92%

256.5 ms133.4 ms

Scaling: 100 rows

benchmark/main.ts

+92%

2.9 ms1.5 ms

Binary approach: parseBinary (1MB)

benchmark/main.ts

+90%

369.9 ms195.2 ms

Scaling: 50 rows

benchmark/main.ts

+88%

1,522 µs810.1 µs

Low-level: CSVLexer + CSVRecordAssembler (1000 rows)

benchmark/main.ts

+87%

28.2 ms15.1 ms

parseString engine:stable (1000 rows)

benchmark/main.ts

+86%

19.9 ms10.7 ms

Memory: toIterableIterator (streaming)

benchmark/main.ts

+86%

28.1 ms15.2 ms

Memory: toArraySync (allocate all at once)

benchmark/main.ts

+85%

29.3 ms15.9 ms

Scaling: 1000 rows

benchmark/main.ts

+85%

29.5 ms16 ms

Stream approach: parseBinaryStream (1MB)

benchmark/main.ts

+83%

3.1 s1.7 s

Scaling: 5000 rows

benchmark/main.ts

+82%

159.9 ms87.7 ms

Line ending: LF (1000 rows)

benchmark/main.ts

+81%

9.9 ms5.5 ms

Queuing: large (10000 rows) - default HWM

benchmark/main.ts

+78%

2.3 s1.3 s

Scaling: 500 rows

benchmark/main.ts

+75%

14.8 ms8.5 ms

Custom delimiter: PSV (100 rows)

benchmark/main.ts

+74%

1,108.6 µs637.6 µs

Field length: 10 chars (1000 rows)

benchmark/main.ts

+73%

12.4 ms7.2 ms

Custom delimiter: TSV (100 rows)

benchmark/main.ts

+70%

1,103.9 µs648.5 µs

Engine comparison: mainThread (5000 rows)

benchmark/main.ts

+67%

152.3 ms90.9 ms

parseString engine:fast (1000 rows)

benchmark/main.ts

+66%

30.1 ms18.2 ms

Quote ratio: 25% (1000 rows)

benchmark/main.ts

+64%

13.2 ms8 ms

parseString.toArraySync(10000 cols, 10 rows, quoted)

benchmark/main.ts

+64%

452.8 ms276.9 ms

Worker perf: large (10000 rows) - mainThread

benchmark/main.ts

+61%

196.2 ms121.7 ms

Binary approach: parseBinary (1KB)

benchmark/main.ts

+60%

463 µs288.7 µs

Quote ratio: 50% (1000 rows)

benchmark/main.ts

+58%

12.7 ms8 ms

Quote ratio: 0% (1000 rows)

benchmark/main.ts

+56%

12.2 ms7.8 ms

Passed

parseString engine:fast (50 rows)

benchmark/main.ts

+48%

1,233 µs834.3 µs

Queuing: small (100 rows) - default HWM

benchmark/main.ts

+44%

23 ms15.9 ms

Concurrent: Parallel mainThread

benchmark/main.ts

+43%

35.1 ms24.5 ms

Field length: 100 chars (1000 rows)

benchmark/main.ts

+41%

13.4 ms9.5 ms

Concurrent: Sequential mainThread

benchmark/main.ts

+41%

35.3 ms25.1 ms

Quote ratio: 100% (1000 rows)

benchmark/main.ts

+33%

13.2 ms9.9 ms

parseBinary.toStream(50 rows)

benchmark/main.ts

+28%

2 ms1.6 ms

Field length: 1KB (1000 rows)

benchmark/main.ts

+25%

27.3 ms21.9 ms

Engine comparison: wasm (5000 rows)

benchmark/main.ts

+23%

109.8 ms89.4 ms

parseString.toArraySync(10000 cols, 10 rows, unquoted)

benchmark/main.ts

+17%

385.5 ms329.1 ms

parseStringToArraySyncWASM(50 rows)

benchmark/main.ts

+17%

1.5 ms1.3 ms

Field length: 10KB (100 rows)

benchmark/main.ts

+13%

16.7 ms14.8 ms

parseString.toIterableIterator(50 rows)

benchmark/main.ts

+8%

1.5 ms1.4 ms

parseStringStream: small (100 rows)

benchmark/main.ts

-2%

19 ms19.4 ms

Engine comparison: wasm (500 rows)

benchmark/main.ts

-7%

9.2 ms9.9 ms

Binary approach: parseBinary (100KB)

benchmark/main.ts

-15%

36.1 ms42.3 ms

Binary approach: parseBinary (10KB)

benchmark/main.ts

-16%

3.7 ms4.4 ms

Stream approach: parseBinaryStream (1KB)

benchmark/main.ts

-18%

4.5 ms5.4 ms

Worker perf: small (100 rows) - mainThread

benchmark/main.ts

-20%

2.4 ms3 ms

Worker perf: medium (1000 rows) - mainThread

benchmark/main.ts

-21%

22.4 ms28.4 ms

Stream approach: parseBinaryStream (10KB)

benchmark/main.ts

-27%

37.6 ms51.3 ms

Worker perf: tiny (10 rows) - mainThread

benchmark/main.ts

-28%

380.4 µs526 µs

parseBinary.toIterableIterator(50 rows)

benchmark/main.ts

-39%

1.1 ms1.8 ms

parseString.toStream(50 rows)

benchmark/main.ts

-47%

2.3 ms4.3 ms

Commits

Click on a commit to change the comparison range

Base

main

8adf5d9

×5.1

style: format code

5dc9bbe

4 days ago

by kamiazya

fix: update test expectations for new parser behavior Missing CSV fields now return empty string instead of undefined. Updated test expectations in: - StringCSVParserStream.test.ts - BinaryCSVParserStream.test.ts - parseBinaryToIterableIterator.test.ts - parse.spec.ts

09bfc64

4 days ago

by kamiazya

-36.03%

fix: change toStrictEqual to toEqual in browser tests Object.create(null) records don't match toStrictEqual due to prototype differences. Changed to toEqual to properly compare object values. Updated test files: - parseResponse.browser.spec.ts - parseBinaryStream.browser.spec.ts - parseBinary.browser.spec.ts

a2c5520

4 days ago

by kamiazya

+36.04%

style: format test files Applied biome formatting to: - src/parser/stream/StringCSVParserStream.test.ts - src/parser/stream/BinaryCSVParserStream.test.ts

5e6e690

4 days ago

by kamiazya

×4

feat(parser): enhance FlexibleStringCSVLexer with reusable array pooling - Implemented ReusableArrayPool to optimize memory usage for string segments in FlexibleStringCSVLexer. - Updated FlexibleStringCSVLexer to utilize the new array pool for segment management, improving performance in parsing. - Adjusted token handling to ensure correct delimiter usage, replacing EOF with Record where appropriate. - Modified tests in CSVRecordAssemblerTransformer and StringCSVLexerTransformer to reflect changes in token structure and delimiter handling. - Cleaned up test cases for better readability and consistency.

58e5d70

3 days ago

by kamiazya

Add record-view assembler and tighten column strategies

125c345

3 days ago

by kamiazya

-4.06%

feat: apply const type parameters consistently across all parser classes プロジェクト全体でconst type parametersを一貫して適用しました。変更内容： - FlexibleCSVArrayRecordAssembler - FlexibleCSVObjectRecordAssembler - FlexibleStringArrayCSVParser - FlexibleStringObjectCSVParser - FlexibleBinaryArrayCSVParser - FlexibleBinaryObjectCSVParser すべてのクラスで`Header extends ReadonlyArray<string>`を `const Header extends ReadonlyArray<string>`に変更。これにより、ユーザーは`as const`を書かなくてもリテラル型が正しく推論されるようになりました： Before: header: ['name', 'age'] as const After: header: ['name', 'age'] テスト：1341個すべて成功 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

e300438

2 days ago

by kamiazya

×5.2

fix: address code review comments レビューコメントに基づいて以下を修正： 1. parseResponse.spec.ts - describeブロック名を修正（parseRequest → parseResponse） 2. column-count-strategy-rename.md - breaking changeをminorに変更し、移行ガイドを追加 3. performance-improvements.md - ベンチマーク結果の記述を修正（46% → 83% faster） 4. vitest.setup.ts - endOnFailureのコメントを正確に修正（デフォルトはfalse） 5. createStringCSVLexer.ts - 戻り値の型にTrackLocationジェネリックを追加すべてのテストが成功（1341個） 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

48f5d4f

2 days ago

by kamiazya

style: improve formatting of expected tokens in StringCSVLexerTransformer tests

19c01bd

2 days ago

by kamiazya

+37.97%

test: consolidate test files and improve coverage config - Consolidate FlexibleCSVRecordAssembler test files (6 → 2 files) - Merged array-output, field-count-limit, object-output, and prototype-safety tests into .test.ts - All 77 tests passing - Consolidate FlexibleStringCSVLexer test files (4 → 2 files) - Merged buffer-overflow and undefined-check tests into .test.ts - All 33 tests passing - Improve coverage configuration in vite.config.ts - Add explicit exclude patterns for test files - Add reporter configuration - Set reportsDirectory - Add @vitest/coverage-v8 to devDependencies for future migration Total: 110 test cases passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

4395ef1

23 hours ago

by kamiazya

×4.8

refactor: enhance comments and improve header handling in FlexibleCSVRecordAssembler tests

6926119

22 hours ago

by kamiazya

Home Terms Privacy Docs