Latest Results
perf(lexer): skip single space in `read_next_token` (#15513)
It's very common for tokens to be separated by a single space. e.g. `const x = 1`, `x === y`.
Previously a single space resulted in calling the `SPS` byte handler, which consumes the space, and then going round the loop again in `Lexer::read_next_token`.
Instead, branchlessly consume a single space (if there is one) before calling the byte handler.
Gives between 2% and 7% perf improvement on parser benchmarks.
---
This also enables a further optimization (not yet implemented).
Now the handler for whitespace (`SPS`) no longer has a hot path for single spaces - it's now only called for a tab, or a 2nd space in a row. In both those cases, it's quite likely there'll be more whitespace following it, so it can now be optimized for that case, and continue consuming bytes until it finds one that *isn't* whitespace.
If handlers for whitespace, line breaks, and comments all continue consuming bytes until they find a "real" token, then we can get rid of `Kind::Skip`, and remove the loop from `read_next_token`. This would remove another unpredictable branch. Active Branches
#155250%
#15521+7%
#155230%
© 2025 CodSpeed Technology