pref-js - Branch - kamiazya/web-csv-toolbox

perf: JavaScript parser performance improvements

#614Merged

Comparing

pref-js

(

6926119

) with

main

(

8adf5d9

)

×4.2

Improvements: 41

Untouched: 24

Skipped: 8

Benchmarks

Skipped (8)

These benchmarks were skipped, so their baseline results are used instead. If they were deleted in your codebase, archive them to remove them from the performance reports.
Learn more about archiving benchmarks

parseString engine:wasm (1000 rows)

benchmark/main.ts

Skipped

13.5 ms*

parseString engine:wasm (50 rows)

benchmark/main.ts

Skipped

1.2 ms*

parseString engine:mainThread (1000 rows)

benchmark/main.ts

Skipped

20.2 ms*

parseString engine:mainThread (50 rows)

benchmark/main.ts

Skipped

1.2 ms*

Stream approach: parseUint8ArrayStream (100KB)

benchmark/main.ts

Skipped

306.7 ms*

Stream approach: parseUint8ArrayStream (1MB)

benchmark/main.ts

Skipped

3.3 s*

Stream approach: parseUint8ArrayStream (1KB)

benchmark/main.ts

Skipped

4.4 ms*

Stream approach: parseUint8ArrayStream (10KB)

benchmark/main.ts

Skipped

32.6 ms*

Improved

parseString engine:stable (50 rows)

benchmark/main.ts

×4.2

2,550.4 µs603 µs

Engine comparison: mainThread (500 rows)

benchmark/main.ts

×3.6

30.9 ms8.6 ms

parseString.toArraySync(50 cols, 50 rows, quoted)

benchmark/main.ts

×3.6

18.6 ms5.2 ms

Line ending: CRLF (1000 rows)

benchmark/main.ts

×3.4

18.5 ms5.5 ms

parseString.toArraySync(100 cols, 50 rows, quoted)

benchmark/main.ts

×3.4

37.2 ms11.1 ms

parseString.toArraySync(10 cols, 50 rows, quoted)

benchmark/main.ts

×3.2

3.7 ms1.2 ms

Low-level: CSVLexer only (1000 rows)

benchmark/main.ts

×3

17.6 ms5.8 ms

parseString.toArraySync(100 cols, 50 rows, unquoted)

benchmark/main.ts

×2.9

20.1 ms6.8 ms

parseString.toArraySync(50 cols, 50 rows, unquoted)

benchmark/main.ts

×2.9

10.1 ms3.5 ms

parseString.toArraySync(10 cols, 50 rows, unquoted)

benchmark/main.ts

×2.6

2,133 µs822.9 µs

Data transformation: type conversion (100 rows)

benchmark/main.ts

×2.4

3.2 ms1.4 ms

parseStringStream: medium (1000 rows)

benchmark/main.ts

×2.1

171.4 ms81.4 ms

Stream approach: parseBinaryStream (100KB)

benchmark/main.ts

×2.1

346.1 ms164.7 ms

parseString.toArraySync(50 rows)

benchmark/main.ts

×2

2.7 ms1.3 ms

Data transformation: no conversion (100 rows)

benchmark/main.ts

+98%

2.6 ms1.3 ms

parseBinary.toArraySync(50 rows)

benchmark/main.ts

+98%

1,060.2 µs536.8 µs

Queuing: medium (1000 rows) - default HWM

benchmark/main.ts

+92%

256.5 ms133.4 ms

Scaling: 100 rows

benchmark/main.ts

+92%

2.9 ms1.5 ms

Binary approach: parseBinary (1MB)

benchmark/main.ts

+90%

369.9 ms195.2 ms

Scaling: 50 rows

benchmark/main.ts

+88%

1,522 µs810.1 µs

Low-level: CSVLexer + CSVRecordAssembler (1000 rows)

benchmark/main.ts

+87%

28.2 ms15.1 ms

parseString engine:stable (1000 rows)

benchmark/main.ts

+86%

19.9 ms10.7 ms

Memory: toIterableIterator (streaming)

benchmark/main.ts

+86%

28.1 ms15.2 ms

Memory: toArraySync (allocate all at once)

benchmark/main.ts

+85%

29.3 ms15.9 ms

Scaling: 1000 rows

benchmark/main.ts

+85%

29.5 ms16 ms

Stream approach: parseBinaryStream (1MB)

benchmark/main.ts

+83%

3.1 s1.7 s

Scaling: 5000 rows

benchmark/main.ts

+82%

159.9 ms87.7 ms

Line ending: LF (1000 rows)

benchmark/main.ts

+81%

9.9 ms5.5 ms

Queuing: large (10000 rows) - default HWM

benchmark/main.ts

+78%

2.3 s1.3 s

Scaling: 500 rows

benchmark/main.ts

+75%

14.8 ms8.5 ms

Custom delimiter: PSV (100 rows)

benchmark/main.ts

+74%

1,108.6 µs637.6 µs

Field length: 10 chars (1000 rows)

benchmark/main.ts

+73%

12.4 ms7.2 ms

Custom delimiter: TSV (100 rows)

benchmark/main.ts

+70%

1,103.9 µs648.5 µs

Engine comparison: mainThread (5000 rows)

benchmark/main.ts

+67%

152.3 ms90.9 ms

parseString engine:fast (1000 rows)

benchmark/main.ts

+66%

30.1 ms18.2 ms

Quote ratio: 25% (1000 rows)

benchmark/main.ts

+64%

13.2 ms8 ms

parseString.toArraySync(10000 cols, 10 rows, quoted)

benchmark/main.ts

+64%

452.8 ms276.9 ms

Worker perf: large (10000 rows) - mainThread

benchmark/main.ts

+61%

196.2 ms121.7 ms

Binary approach: parseBinary (1KB)

benchmark/main.ts

+60%

463 µs288.7 µs

Quote ratio: 50% (1000 rows)

benchmark/main.ts

+58%

12.7 ms8 ms

Quote ratio: 0% (1000 rows)

benchmark/main.ts

+56%

12.2 ms7.8 ms

Passed

parseString engine:fast (50 rows)

benchmark/main.ts

+48%

1,233 µs834.3 µs

Queuing: small (100 rows) - default HWM

benchmark/main.ts

+44%

23 ms15.9 ms

Concurrent: Parallel mainThread

benchmark/main.ts

+43%

35.1 ms24.5 ms

Field length: 100 chars (1000 rows)

benchmark/main.ts

+41%

13.4 ms9.5 ms

Concurrent: Sequential mainThread

benchmark/main.ts

+41%

35.3 ms25.1 ms

Quote ratio: 100% (1000 rows)

benchmark/main.ts

+33%

13.2 ms9.9 ms

parseBinary.toStream(50 rows)

benchmark/main.ts

+28%

2 ms1.6 ms

Field length: 1KB (1000 rows)

benchmark/main.ts

+25%

27.3 ms21.9 ms

Engine comparison: wasm (5000 rows)

benchmark/main.ts

+23%

109.8 ms89.4 ms

parseString.toArraySync(10000 cols, 10 rows, unquoted)

benchmark/main.ts

+17%

385.5 ms329.1 ms

parseStringToArraySyncWASM(50 rows)

benchmark/main.ts

+17%

1.5 ms1.3 ms

Field length: 10KB (100 rows)

benchmark/main.ts

+13%

16.7 ms14.8 ms

parseString.toIterableIterator(50 rows)

benchmark/main.ts

+8%

1.5 ms1.4 ms

parseStringStream: small (100 rows)

benchmark/main.ts

-2%

19 ms19.4 ms

Engine comparison: wasm (500 rows)

benchmark/main.ts

-7%

9.2 ms9.9 ms

Binary approach: parseBinary (100KB)

benchmark/main.ts

-15%

36.1 ms42.3 ms

Binary approach: parseBinary (10KB)

benchmark/main.ts

-16%

3.7 ms4.4 ms

Stream approach: parseBinaryStream (1KB)

benchmark/main.ts

-18%

4.5 ms5.4 ms

Worker perf: small (100 rows) - mainThread

benchmark/main.ts

-20%

2.4 ms3 ms

Worker perf: medium (1000 rows) - mainThread

benchmark/main.ts

-21%

22.4 ms28.4 ms

Stream approach: parseBinaryStream (10KB)

benchmark/main.ts

-27%

37.6 ms51.3 ms

Worker perf: tiny (10 rows) - mainThread

benchmark/main.ts

-28%

380.4 µs526 µs

parseBinary.toIterableIterator(50 rows)

benchmark/main.ts

-39%

1.1 ms1.8 ms

parseString.toStream(50 rows)

benchmark/main.ts

-47%

2.3 ms4.3 ms

Commits

Click on a commit to change the comparison range

Base

main

8adf5d9

×5.1

style: format code

5dc9bbe

4 days ago

by kamiazya

fix: update test expectations for new parser behavior Missing CSV fields now return empty string instead of undefined. Updated test expectations in: - StringCSVParserStream.test.ts - BinaryCSVParserStream.test.ts - parseBinaryToIterableIterator.test.ts - parse.spec.ts

09bfc64

4 days ago

by kamiazya

-36.03%

fix: change toStrictEqual to toEqual in browser tests Object.create(null) records don't match toStrictEqual due to prototype differences. Changed to toEqual to properly compare object values. Updated test files: - parseResponse.browser.spec.ts - parseBinaryStream.browser.spec.ts - parseBinary.browser.spec.ts

a2c5520

4 days ago

by kamiazya

+36.04%

style: format test files Applied biome formatting to: - src/parser/stream/StringCSVParserStream.test.ts - src/parser/stream/BinaryCSVParserStream.test.ts

5e6e690

4 days ago

by kamiazya

×4

feat(parser): enhance FlexibleStringCSVLexer with reusable array pooling - Implemented ReusableArrayPool to optimize memory usage for string segments in FlexibleStringCSVLexer. - Updated FlexibleStringCSVLexer to utilize the new array pool for segment management, improving performance in parsing. - Adjusted token handling to ensure correct delimiter usage, replacing EOF with Record where appropriate. - Modified tests in CSVRecordAssemblerTransformer and StringCSVLexerTransformer to reflect changes in token structure and delimiter handling. - Cleaned up test cases for better readability and consistency.

58e5d70

3 days ago

by kamiazya

Add record-view assembler and tighten column strategies

125c345

3 days ago

by kamiazya

-4.06%

feat: apply const type parameters consistently across all parser classes プロジェクト全体でconst type parametersを一貫して適用しました。変更内容： - FlexibleCSVArrayRecordAssembler - FlexibleCSVObjectRecordAssembler - FlexibleStringArrayCSVParser - FlexibleStringObjectCSVParser - FlexibleBinaryArrayCSVParser - FlexibleBinaryObjectCSVParser すべてのクラスで`Header extends ReadonlyArray<string>`を `const Header extends ReadonlyArray<string>`に変更。これにより、ユーザーは`as const`を書かなくてもリテラル型が正しく推論されるようになりました： Before: header: ['name', 'age'] as const After: header: ['name', 'age'] テスト：1341個すべて成功 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

e300438

2 days ago

by kamiazya

×5.2

fix: address code review comments レビューコメントに基づいて以下を修正： 1. parseResponse.spec.ts - describeブロック名を修正（parseRequest → parseResponse） 2. column-count-strategy-rename.md - breaking changeをminorに変更し、移行ガイドを追加 3. performance-improvements.md - ベンチマーク結果の記述を修正（46% → 83% faster） 4. vitest.setup.ts - endOnFailureのコメントを正確に修正（デフォルトはfalse） 5. createStringCSVLexer.ts - 戻り値の型にTrackLocationジェネリックを追加すべてのテストが成功（1341個） 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

48f5d4f

2 days ago

by kamiazya

style: improve formatting of expected tokens in StringCSVLexerTransformer tests

19c01bd

1 day ago

by kamiazya

+37.97%

test: consolidate test files and improve coverage config - Consolidate FlexibleCSVRecordAssembler test files (6 → 2 files) - Merged array-output, field-count-limit, object-output, and prototype-safety tests into .test.ts - All 77 tests passing - Consolidate FlexibleStringCSVLexer test files (4 → 2 files) - Merged buffer-overflow and undefined-check tests into .test.ts - All 33 tests passing - Improve coverage configuration in vite.config.ts - Add explicit exclude patterns for test files - Add reporter configuration - Set reportsDirectory - Add @vitest/coverage-v8 to devDependencies for future migration Total: 110 test cases passing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>

4395ef1

22 hours ago

by kamiazya

×4.8

refactor: enhance comments and improve header handling in FlexibleCSVRecordAssembler tests

6926119

21 hours ago

by kamiazya

Home Terms Privacy Docs