Latest Results
Fix _UnpicklingFuture.add_done_callback swallowing deserialization errors (#21610)
When cloudpickle.loads() fails inside _UnpicklingFuture.add_done_callback,
the exception propagates into concurrent.futures.Future._invoke_callbacks,
which silently catches and logs it. The callback `fn` (which resolves the
_ChainedFuture via PrefectWrappedFuture.call_with_self) never executes,
so the future chain is never resolved and the flow run hangs indefinitely
as a zombie — consuming cluster resources until manually killed.
This can happen when task code changes between the time a task is submitted
and when its result is deserialized (e.g., module-level renames, removed
classes, or image version mismatches in Kubernetes deployments).
Closes #21610
Changes:
- Add super().__init__() to _UnpicklingFuture so it can track its own state
- Wrap cloudpickle.loads() in try/except within add_done_callback, storing
any deserialization error in _deserialization_error
- Always call fn(self) — not fn(result) — so the callback chain completes
regardless of deserialization success (also fixes a latent bug: fn expects
a future per the add_done_callback contract, not a deserialized value)
- Override result() and exception() to surface _deserialization_error when set
Testing:
- Added TestUnpicklingFuture with two unit tests:
- test_add_done_callback_propagates_deserialization_error: feeds invalid
bytes to the inner future, asserts callback fires and exception() returns
the deserialization error. Verified this test FAILS on the unfixed code
(callback never invoked, 5s timeout hit) and PASSES after the fix.
- test_add_done_callback_passes_result_on_success: feeds valid cloudpickle
bytes, asserts callback fires with self and result() returns correctly.
- Tests are pure unit tests against _UnpicklingFuture directly, no server
or database fixtures required.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>saschwartz:fix/21610-unpickling-future-error-propagation Latest Branches
0%
saschwartz:fix/21610-unpickling-future-error-propagation 0%
devin/oss-7887-1776669201 -3%
devin1/oss-7874-fix-dns-rebinding-toctou-bypass-in-validate_restricted_url © 2026 CodSpeed Technology