Latest Results
Build popitem()'s result after removing the pair (#1625)
<!-- Thank you for your contribution! -->
## What do these changes do?
`popitem()` built its result while it still pointed at the popped entry,
and read the entry's value afterwards. Building the result can run
Python code that mutates the multidict: a `str` subclass key's
`__str__()` when a `CIMultiDict` makes its `istr`, and on Python 3.10
and 3.11 a garbage collection the tuple allocation triggers, whose
finalizers run there. If that code replaced the table, the value came
from freed memory (a wrong value, or a crash). The C implementation now
takes its own references, removes the pair, and only then builds the
result, the order the pure-Python implementation already used.
The pure-Python `popitem()` decremented its size after building the key,
so it miscounted `len()` in the same case, and kept the old entry list
alive in a local until it returned; both are fixed.
## Are there changes in behavior for the user?
Yes: the use-after-free is gone. If building the key raises, the pair is
removed before the exception propagates, as the pure-Python
implementation always did; the C one used to leave it in place.
## Is it a substantial burden for the maintainers to support this?
No.
## Related issue number
Found by the finalizer audit in #1631; same class as #1619.
## Checklist
- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt` (N/A)
- [x] Add a new news fragment into the `CHANGES/` folder
- [x] `make doc-spelling` passes and any new technical words are added
to `docs/spelling_wordlist.txt`
<details>
<summary>Agent run details (optional, for reviewers)</summary>
New tests: `test_popitem_key_str_mutates` (fails on both backends
against master: C returns the value `'1'` read from the freed table,
pure Python miscounts `len`), `test_popitem_key_str_raises`, and
`test_popitem_collection_mutates`, which arms a collection at each of 12
allocation offsets; on 3.10 against master it fails 6 C cases with wrong
values.
Tests, one venv and tree copy per leg, full suite: GIL 3.14.7 (2951
passed), FT 3.14.7t (2939), 3.10.17 (2951), `MULTIDICT_DEBUG_BUILD=1`
GIL (2951) and FT (2939), pure Python with `MULTIDICT_NO_EXTENSIONS=1
--no-c-extensions` (1384), and ASan+UBSan on a debug GIL build with
`PYTHONMALLOC=malloc` (2939, `test_leaks` deselected) and again with
`MULTIDICT_NO_FREELIST=1` (2939). The Cython harness was not built, so
its cases skipped.
Callgrind, `PYTHONHASHSEED=0`, `benchmarks/callgrind_driver.py
--include-multidict-only`, base master 9802f1b, Ir per op: rows that
moved by 0.5% or more:
| row | GIL MultiDict | GIL CIMultiDict | FT MultiDict | FT CIMultiDict
|
|---|---|---|---|---|
| popitem | 502.6 -> 489.4 (-2.6%) | 1507.1 -> 1495.9 (-0.7%) | 646.6 ->
564.4 (-12.7%) | 1698.0 -> 1613.8 (-5.0%) |
| delitem | 237.5 -> 234.5 (-1.3%) | 290.2 -> 287.2 (-1.0%) | 392.2 ->
396.2 (+1.0%) | 443.9 -> 447.9 (+0.9%) |
| setitem_replace | 327.5 -> 321.5 (-1.8%) | | | |
| setitem_insert | 402.5 -> 395.7 (-1.7%) | | | |
| iter_items | 129.7 -> 127.7 (-1.5%) | 133.7 -> 131.7 (-1.5%) | 188.7
-> 186.7 (-1.1%) | 189.7 -> 187.7 (-1.1%) |
| iter_keys | 60.1 -> 59.1 (-1.7%) | 64.1 -> 63.1 (-1.6%) | | |
| getall | 812.9 -> 807.9 (-0.6%) | | | |
`popitem()` takes over the entry's references (`_md_unlink_at()`, split
out of `_md_del_at()`) instead of adding its own; a first cut that did
`Py_NewRef()` three times cost +6.3% on GIL `popitem`.
`md_next()` is `ALWAYS_INLINE` here too (the same hunk as #1628): after
the rebase onto 9802f1b, GCC 15 pushed it out of the FT items iterator,
and `tools/check_inlining.py` failed its #1601 rule; pinned, all 8 rules
pass on GIL and FT with GCC 13 and 15. Lint: pre-commit clean; `make
doc-spelling` passes (adds `finalizers`).
</details>
Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov. Drop ASSERT_CONSISTENT()'s update flag (#1629)
<!-- Thank you for your contribution! -->
## What do these changes do?
`ASSERT_CONSISTENT(md, update)` took a flag that relaxed the debug-only
consistency check for the half-deleted entries (`key == NULL`, identity
kept) that `update()` used to leave in the table mid-batch. #1628
removed half-deletion and made the check strict on every call, leaving
the flag ignored; this drops it from the macro,
`_md_check_consistency()` and every call site.
## Are there changes in behavior for the user?
No. The macro compiles to nothing in release builds.
## Is it a substantial burden for the maintainers to support this?
No, it removes a parameter.
## Related issue number
Follows #1628.
## Checklist
- [x] I think the code is well written
- [ ] Unit tests for the changes exist (N/A, no behavior change; the
debug suites exercise every call site)
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt` (N/A)
- [x] Add a new news fragment into the `CHANGES/` folder
- [x] `make doc-spelling` passes and any new technical words are added
to `docs/spelling_wordlist.txt`
<details>
<summary>Agent run details (optional, for reviewers)</summary>
`tools/codegen_diff.py --ref upstream/master` (9684a61, #1628 merged): 0
of 256 functions differ on 3.14.7, 0 of 265 on 3.14.7t (release flags).
Full suite with `MULTIDICT_DEBUG_BUILD=1`, where every
`ASSERT_CONSISTENT()` runs: GIL 3.14.7 3028 passed, FT 3.14.7t 3016
passed.
Lint: pre-commit clean.
</details>
Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov. Give the re-init stress test's readers time to drain (#1633)
<!-- Thank you for your contribution! -->
## What do these changes do?
`test_reinit_finalizer_vs_lock_free_readers` (#1621) timed out on the
Windows 3.10, 3.11 and 3.12 jobs, at `f.result(timeout=60)` for its
readers. The faulthandler dump from a 183 s run shows every reader still
inside `Value.__del__()` -> `probe()` after the writer finished, each
blocked in `coverage/collector.py: lock_data`. Under coverage's C tracer
(3.10 to 3.12) every traced line takes a lock the threads share, and the
readers are draining the finalizers of generations their `copy()`
snapshots outlived. That work is bounded (3216 finalizers, nesting depth
1), just slow: 0.4 s without coverage, 12 to 20 s with it on a Linux
3.11 box, and over three minutes on the Windows runners. The failure
reports also show every reader future finished by the time pytest
rendered them.
The writer now gets 300 s and the readers a shared 300 s deadline,
inside the job's 15-minute limit.
## Are there changes in behavior for the user?
No, tests only.
## Is it a substantial burden for the maintainers to support this?
No.
## Related issue number
Follows #1621. Failed on #1625, #1628 (twice) and #1630; re-runs passed.
## Checklist
- [x] I think the code is well written
- [ ] Unit tests for the changes exist (N/A, a test timeout)
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt` (N/A)
- [x] Add a new news fragment into the `CHANGES/` folder
- [x] `make doc-spelling` passes and any new technical words are added
to `docs/spelling_wordlist.txt`
<details>
<summary>Agent run details (optional, for reviewers)</summary>
`tests/test_free_threading.py`: GIL 3.14.7 10 passed, FT 3.14.7t 10
passed (without coverage); 3.11.13 with coverage, the two cases took
12.4 s and 19.9 s.
Lint: pre-commit clean.
</details>
Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov. Keep update()'s doomed entries whole until the batch ends (#1628)
<!-- Thank you for your contribution! -->
## What do these changes do?
`update()` half-deleted the later matches of a key as it went (key and
value set to `NULL`, identity kept) until `md_post_update()` finished
them off, and kept its marks keyed by entry index, dropping them
whenever anything else touched the table. Python code that runs between
the items of one `update()` (the argument's own iterator, a key's
`lower()`, a finalizer of a dropped item) could therefore:
- crash reading a half-deleted entry through `get()`, `getall()`,
`items()`, `copy()` or `popall()`;
- make a later item for the same key overwrite a value the call had just
written, since any mutation dropped the marks;
- have `merge()` skip a key a pending deletion was about to remove.
Doomed entries now stay whole until `md_post_update()` removes them,
marked only in the batch's bitmap. Each is recorded with the value it
held when doomed, referenced so the address cannot be reused, and is
removed only if it still holds it: a nested `update()` or `md[key] =
value` run between items that writes to it wins. (Only the value is
compared, since reading a `CIMultiDict` key swaps the stored `str` for
its `istr`.) The marks stay valid because nothing reuses an entry index
while a batch is in flight: `md` counts its batches in `is_ci`'s padding
(the object stays 64 bytes), and while there are any a resize keeps
every entry at its index, `popitem()` leaves its trailing holes, and a
clear or re-init starts the new table with a hole per old entry. A
generation bumped with every new table tells a batch to widen its
bitmaps. The remap, the sweep and the half-deletion all go.
The pure-Python implementation had the same shape with `HASH_MARK` and
`key=None` in the table, and returned `None` for doomed pairs; it now
keeps its marks in dicts keyed by entry id, and holds replaced pairs
until the call ends, as C does.
## Are there changes in behavior for the user?
Yes: no crash, and `update()`/`merge()` keep what they wrote when the
argument's iterator or a finalizer mutates the multidict. Watchers now
see a doomed pair's `DELETED` at the end of the batch, and a pair an
earlier item doomed and a later one reuses as `REPLACED` rather than
`DELETED` plus `ADDED`.
## Is it a substantial burden for the maintainers to support this?
No: the invariant is "no index is reused while a batch runs", checked in
four places, and it removes more code than it adds.
## Related issue number
Found by the finalizer audit in #1631.
## Checklist
- [x] I think the code is well written
- [x] Unit tests for the changes exist
- [x] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
`CONTRIBUTORS.txt` (N/A)
- [x] Add a new news fragment into the `CHANGES/` folder
- [x] `make doc-spelling` passes and any new technical words are added
to `docs/spelling_wordlist.txt`
<details>
<summary>Agent run details (optional, for reviewers)</summary>
New tests in `tests/test_update.py`: readers and mutators run from the
argument's iterator between two items of `update()` and `merge()`, marks
outgrowing the table, and
`test_update_keeps_what_code_between_items_wrote` for Greptile's
nested-write case on #1629 (a nested `update()`, `md[key] = value`,
`merge()` and a plain read between items; the nested `update()` cases
failed on both backends before the fix, the read case on C with a first
cut that also compared keys). Against master the C reader cases
segfault, 20 C mutator cases fail and 12 pure-Python cases fail.
Tests, one venv and tree copy per leg, full suite, on the merge with
2087809: GIL 3.14.7 (3028 passed), FT 3.14.7t (3016), 3.10.17 (3028),
`MULTIDICT_DEBUG_BUILD=1` GIL (3028) and FT (3016), pure Python with
`MULTIDICT_NO_EXTENSIONS=1 --no-c-extensions` (1425), and ASan+UBSan on
a debug GIL build with `PYTHONMALLOC=malloc` (3016, `test_leaks`
deselected) and again with `MULTIDICT_NO_FREELIST=1` (3016). The Cython
harness was not built, so its cases skipped.
Callgrind, `PYTHONHASHSEED=0`, `benchmarks/callgrind_driver.py
--include-multidict-only`, base master 2087809, Ir per op; rows that
moved by 0.5% or more:
| row | GIL MultiDict | GIL CIMultiDict | FT MultiDict | FT CIMultiDict
|
|---|---|---|---|---|
| update | 36996 -> 37476 (+1.3%) | 42636 -> 43316 (+1.6%) | 46411 ->
42898 (-7.6%) | 52311 -> 48798 (-6.7%) |
| ctor_items | 44842 -> 41660 (-7.1%) | 55923 -> 52741 (-5.7%) | | |
| ctor_small | 4228 -> 3927 (-7.1%) | 5349 -> 5048 (-5.6%) | | |
| reinit_items | 45468 -> 42285 (-7.0%) | 56413 -> 53230 (-5.6%) | | |
| reinit_items_small | 4854 -> 4552 (-6.2%) | 5839 -> 5537 (-5.2%) | | |
The construction gains are GCC re-planning its inlining after the
half-delete path went; nothing here targets them. `update()` with
duplicate keys, the CodSpeed `update_*_with_duplicates` shape (a
150-item copy updated with 100 items, 75 of them dooming an entry), per
call on GIL: 84107 -> 89769 (`MultiDict`, +6.7%) and 87530 -> 93192
(`CIMultiDict`, +6.5%), about 75 Ir per doomed entry for its record and
reference; that is the price of keeping a nested write.
Pure Python, callgrind Ir per `copy().update()`: plain update of 200
into 200 (`MultiDict`) 6044942 -> 5233224 (-13.4%) and `CIMultiDict`
-12.8%, since there is no full-table mark sweep unless something was
doomed; with duplicate keys (the CodSpeed `update_*_with_duplicates`
shape) 3779495 -> 3967045 (+5.0%) and +4.7%, down from the +17% CodSpeed
measured on the first push, which walked a hash chain per doomed entry.
A registration-list design that grew the object to 72 bytes cost ctor ~8
Ir and was dropped for the counters in the padding.
`tools/check_inlining.py` failed `multidict_items_iter_tp_iternext ->
md_next` on FT with GCC 15; `md_next()` is now `ALWAYS_INLINE` (the same
hunk as #1625 and #1627), and all 8 rules pass on GIL and FT with GCC 13
and 15. Lint: pre-commit clean; `make doc-spelling` passes.
</details>
Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov. Latest Branches
0%
asvetlov:to-dict-gc-finalizer 0%
asvetlov:popitem-key-reentrancy 0%
asvetlov:cover-pure-update-match © 2026 CodSpeed Technology