Latest Results
Stop the filesystem lease storage from revoking live leases
Fixes #22935.
The filesystem concurrency lease storage keeps one file per lease plus a
single shared `expirations.json`. `_update_expiration_index` reads the
whole index, edits one key and writes the whole index back, with an
`await` between the read and the write. Two renewals that interleave
there take the same snapshot, and the second write drops the first one's
update. `_atomic_write_json` makes each write atomic, which prevents a
torn file but does nothing about a lost update.
That would be recoverable if the reaper cross-checked, but
`read_expired_lease_ids` trusts the index alone. So the losing lease
keeps a stale expiration, the repossessor revokes it, and its holder --
alive, and still renewing on schedule -- gets a 410 on the next renewal.
It does not need multiple processes: `_load_expiration_index` awaits, so
two concurrent renewal requests interleave inside one server process.
At the default renewal cadence of 0.75 x TTL a single lost write is
always fatal. A renewal at `t` leaves `t + 0.25 x TTL` in the index
instead of `t + TTL`, the repossessor revokes at `t + 0.25 x TTL`, and
the next renewal is only due at `t + 0.75 x TTL`.
Two changes:
- `read_expired_lease_ids` confirms every index candidate against its
lease file before reporting it. The lease file is the source of truth
-- `renew_lease` writes it atomically, and writes it first -- and the
index is only an accelerator for the scan. A lease whose file is still
in the future is left alone and its index entry repaired, so the
active-lease reads agree again. An entry with no lease file behind it
is still reported, so the revocation path clears the orphan.
- An `asyncio.Lock` serializes the index read-modify-write. It is
class-level because `get_concurrency_lease_storage()` builds a new
storage instance per call, so a per-instance lock would never contend.
The lock covers a single process; the file check covers the rest, and is
what makes a stale index entry non-fatal in any deployment.
Three of the new tests fail on the current implementation
(`test_concurrent_renewals_both_reach_the_index`,
`test_stale_index_entry_does_not_expire_a_live_lease`,
`test_stale_index_entry_is_repaired`); the other two guard against the
file check turning into "never expire anything".anthrax63:fix/lease-storage-index-race Serialize the lease index lock across threads and processes with a file lock
A per-event-loop asyncio.Lock registry fixed the RuntimeError from an
embedded server restart binding a stale lock, but it never actually
serialized concurrent access: two loops running at the same time (two
worker threads, each with its own loop) each get a different lock from
the registry, so their read-modify-write cycles on the expiration index
still race and one write clobbers the other.
Replace the registry with the same FileLock used elsewhere in this
codebase (GitRepository.pull_code, the server startup migration lock).
It's not bound to any event loop or thread, so a single lock file next
to the index serializes renewals across loops, threads, and even
separate processes sharing the same storage_path, which this backend
already needs to tolerate given it stores everything on disk. Its
aacquire() polls instead of blocking the event loop while waiting.
Added a regression test that runs two updates concurrently on separate
threads, each with its own asyncio.run() loop, and confirms both index
entries survive instead of one clobbering the other.afonsojanu:fix/lease-expiration-index-race-22935 Latest Branches
+1%
alekseevpavel04:fix/ui-api-url-from-serving-server 0%
devin1/oss-8185-revoke_expired_lease-revokes-a-lease-that-was-renewed-after 0%
afonsojanu:fix/lease-expiration-index-race-22935 © 2026 CodSpeed Technology