Latest Results
refactor(search): split runSearchCommand so tests run its real route
The BM25 tests read file scores, which the output hides. Their helper
bm25NodeSearch copied the strict branch of runSearchCommand step by
step: withSkippedNodes, planCorpora, withIndexWindow, the
searchExecution literal, withBM25Command, searchNodes with strictNode,
and rescoreStrictResults. It also skipped the cache root and cache
budget checks. When the command changed, the copy did not follow it.
runSearchCommand now plans the query and calls searchGraph with
writeSearchedGraph as its show function:
- searchGraph runs every step up to the display: the cache root check,
the node graph, the build window, the model start, the cache budget
check, the name-only count, the BM25 command, and the strict or
model route. Then it gives the searched graph to show. Its defers
keep their order: it closes the model and enforces the cache budget
after show returns, so the output does not wait for them. On an
error it does not call show.
- writeSearchedGraph warns for each file operand that the index keeps
by name only and that has no match, and writes the results. With no
result it returns noMatchOrInterrupted.
- searchedGraph carries the results before the display orders them and
sets the headers.
bm25NodeSearch now calls searchGraph with the lexical-only policy and
reads the scores and the BM25 command (from the context, as the
searches do) in its show function. Four comments that named
runSearchCommand for steps that moved now name searchGraph.
Checks: the step order, the context values and the defer order match
the old function (line-by-line review). Without rescoreStrictResults,
TestBM25RescoreIsExact, TestBM25FileAtomTermInAnyLayout and
TestBM25LayoutIndependent fail, so the helper still tests the rescore.
TestBM25* passes with -race; TestRun*, TestE2E*, TestCLI*, TestSearch*,
TestRoot*, TestNameOnly* and the unindexed-warning and cache budget
tests pass; make test-static is clean. Add read-semaphore + windowed shard indexing; fix deadlock + perf
PROBLEM
Folder corpora exceeding `maxInFlightBytes` (600 MiB) wedged
indexing in a three-way deadlock: workers blocked on
`readSemaphore.Acquire`, producers blocked on channel send, consumer
blocked waiting to close the channel — `builder.Finish()` never ran
so the aggregate `Release` never fired. Reproduced live at PID
51141 with `seek "lol" ~/Downloads ~/Development/git/.../monorepo
~/Documents` (4.1 GiB folder hung at 530 MB RSS, 0% CPU).
FIX — windowed shard rotation
`indexDocuments` now rotates a `shardWindow` builder every
`indexWindowBytes` (300 MiB default, half of `maxInFlightBytes`).
Each window releases its accumulated weight after `builder.Finish()`
returns, bounding peak in-flight bytes to
`indexWindowBytes + 2 × maxIndexedDocumentBytes` (compile-time
invariant in `caps.go`). Five-layer architecture:
LAYER A: indexDocuments — windowed rotation (indexer.go)
LAYER B: readUncommittedCandidates — bounded drain returning
errUncommittedPayloadTooLarge (indexer.go)
LAYER C: indexFolderDocumentsDelta — content-bytes guard
returning errFolderDeltaPayloadTooLarge (folder_indexer.go)
LAYER D/E: indexUncommitted + indexFolderDocuments catch the
sentinels via errors.Is and fall back to LAYER A's
windowed full rebuild.
PERFORMANCE
Beyond the deadlock fix, multiple optimizations land:
1. Pre-flight lstat-sum short-circuit in readUncommittedCandidates
and changedFolderDocumentsFromManifest: O(N) loop summing
candidate.size returns the window sentinel BEFORE any read
syscall. Bench shows 24x speedup vs the slow drain path,
0 allocs vs 268 KiB on the overflow path
(BenchmarkReadUncommittedCandidates_PreFlight).
2. Streaming JSON encoder for writeFolderManifest +
writeUncommittedManifest: json.NewEncoder + bufio.Writer
replaces json.Marshal + os.WriteFile. Cuts peak transient
allocation from ~2x payload to bufio working set
(~32 KiB). On 100k entries: 22 MB peak vs 44 MB pre-fix
(BenchmarkWriteFolderManifest/100k).
3. Parallel per-shard search in executeParsedSearchScoped:
fan-out across runtime.NumCPU goroutines bounded by a
semaphore channel. Single-shard fast path preserved. Wall
time becomes max-of-shards instead of sum-of-shards;
5-10x speedup at 100+ shards (BenchmarkShardSearch_NShards).
4. acquireSearchLock stale-shard fallback: when LOCK_SH is
contended AND shards exist in indexDir, returns nil with
slog.Warn instead of polling 60s. Mirrors acquireLock's
equivalent fallback. Bench: 117 us fast path
(BenchmarkAcquireSearchLock_StaleFallback).
5. FD ceiling startup warn: warnLowFileDescriptorLimit logs a
hint when soft RLIMIT_NOFILE < 4096 (multi-corpus rotation
can hold 3 corpora x ~420 shards = ~1260 fds during search).
6. Per-plan context.WithCancel in main.go wrapped in func +
defer cancelPlan: panic-safe cleanup ensures no cross-plan
semaphore leak via the process-global readSemaphore.
7. Stale-shard fallback in acquireLock now logs slog.Warn
instead of silent: hung indexer was previously masked as
graceful degradation forever.
8. GC sweeps orphan .folder-manifest-v1.tmp +
.uncommitted-manifest-v1.tmp files older than 1 hour.
INSTRUMENTATION
shardWindowFinishes / shardWindowMetaRewrites atomic.Int64
counters surface the rotation cost in benchmarks
(BenchmarkFolderCorpus_ColdIndex_LargeRotation reports
windows/op + metaRewrites/op). Quadratic IsDelta=true meta
rewrite cost is now CI-visible: 15 rewrites/op for 6 windows
matches Sum(N=0..5)=15 formula exactly.
folderDeltaPreflightTrips / folderDeltaContentTrips
atomic.Int64 counters distinguish which guard branch fired so
tests can verify the pre-flight optimization specifically
(not just the post-read content-bytes guard).
TESTS
18 new test files covering: pressure (4 MiB swap budget +
12 MiB payload), cancellation with goleak, multi-corpus
sequential drain, property/fuzz over file-size distributions,
cross-window early-beacon survival, pre-flight short-circuit
isolation, acquireSearchLock both branches, soak (4 GiB
sparse for 5 min, gated behind //go:build soak + SEEK_SOAK=1).
Helpers: planSynthCorpus, writeRandomFolder, writeSparseFile,
goroutineLeakGuard, withReadSemLock, swapReadSemaphoreForTest
(also swaps indexWindowBytes proportionally),
swapIndexWindowBytesForTest, waitUntilSemaphoreBelow,
mustHoldTestReadSemMu (TryLock panic guard against future
t.Parallel misuse).
5 new benchmarks isolating each perf optimization with
b.ReportMetric for shard count, window count, allocation
peak, and meta-rewrite cost. Validates every claim above.
BREAKING CHANGE
streamFiles signature now requires context.Context.
acquireSearchLock signature now requires the indexDir string.
fileContent embeds a weight field that the consumer must
Release exactly once: per shardWindow.finish() for
indexDocuments, per call for indexDeltaDocuments (see
read_semaphore.go and indexer.go::fileContent docstrings).
All in-tree callers updated.
VERIFICATION
go test -race -timeout=600s: PASS (373s)
golangci-lint run: 0 issues
Repro: seek "lol" \$HOME/Downloads ... completes instead of
wedging.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Latest Branches
N/A
N/A
codspeed-wizard-1774529268605 © 2026 CodSpeed Technology