fix(tests): fix flaky "generatorHandler > nonexistent executable"(#18372)

Merged
Merging
fix/generator-helper-flaky-test
into
main
-76%
IMPROVEMENTS
0
REGRESSIONS
1
UNTOUCHED
2
NEW
0
DROPPED
0

Benchmarks

Failing

typescript compilation ~50 Models
Regression
packages/client/src/__tests__/benchmarks/huge-schema/huge-schema.bench.ts::typescript::typescript compilation ~50 Models
-76%
61.8 ms
258.1 ms

Passing

client generation ~50 Models
packages/client/src/__tests__/benchmarks/huge-schema/huge-schema.bench.ts::typescript::client generation ~50 Models
+2%
6 s
5.9 s
client generation 100 models with relations
packages/client/src/__tests__/benchmarks/lots-of-relations/lots-of-relations.bench.ts::typescript::client generation 100 models with relations
-5%
56.6 s
59.7 s

Commits

Click on a commit to change the comparison range
refactor: remove some promisify to fs.promises (#18990)
base
cec59d3
7 months ago by Jolg42
+1%
fix(tests): fix flaky "generatorHandler > nonexistent executable" The test sometimes fails on CI with: ``` Error: expect(received).rejects.toThrow() Received promise resolved instead of rejected Resolved to value: undefined ``` There are two sources of flakiness in this test: 1. The `setTimeout` timer that waits for `initWaitTime` after the process spawns to see if it will exit with an error shortly after starting must only be set after the process actually starts (`child.on('spawn')` event), otherwise there's a chance that the timer fires before the process even finishes spawning. 2. We don't spawn the executable directly, but wrapped in a shell. On macOS CI (or locally under big load with `detect_flake -c 'pnpm --filter @prisma/generator-helper test generatorHandler.test.ts -- -t nonexistent' -t 100 -r 2 -p -e`), sometimes it takes more than 200 ms for the shell to initialize, parse the command and try executing the non-existing binary, so the command doesn't fail in `initWaitTime` and `GeneratorProcess` considers it successfully started. This commit changes `initWaitTime` in this test to 2 seconds. With these changes, there are no errors for me locally when spawning 200 parallel instances of the test with 5 runs per thread, but without them at least 3% of the runs fail with just 100 threads. It is still possible that it would fail sometimes as any tests that rely on timers are inherently flaky under load, but the probability of this should be now much lower. This commit also fixes some TypeScript errors along the way (we don't type-check the old tests on CI, so they accumulate type errors eventually).
47c0d03
9 months ago by aqrln
-77%
Simplify control flow by moving the logic to exit handler
7267e01
7 months ago by aqrln
ResourcesHomePricingDocsBlogGitHub
Copyright © 2023 CodSpeed Technology SAS. All rights reserved.