Latest Results
fix: relinquish evicted flow runs to infrastructure retry
Closes #22620.
With a Kubernetes work pool at backoffLimit > 0, a SIGTERM (eviction) made
the in-pod engine finalize the run Crashed, but Kubernetes then retried the
Job and re-ran the flow, so the run left a terminal state and executed twice.
Teach `prefect flow-run execute` a termination behavior via
PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR (`crash`, `reschedule` or
`relinquish`) and deliver it to the engine as an intent over the control
channel that already connects the two. On SIGTERM the supervisor signals the
intent, proposes AwaitingRetry only for `reschedule`, kills the engine, and
exits non-zero for `relinquish` so Kubernetes counts the attempt and retries.
Once it has done so it stops supervising, so the executor cannot start another
process or propose a state of its own.
The engine now dispatches on intent alone and proposes no state when someone
else owns it, so it no longer reads Kubernetes-specific configuration. Any
intent also exits the engine cleanly, matching how cancellation already works,
so the supervisor does not have to suppress a crash from the exit code.
Ordinary cancellations carry no intent and still crash the run.
The engine child now starts in its own process group on POSIX, matching the
existing Windows behavior, so an external SIGTERM cannot reach it before the
supervisor has signalled its intent.
The kubernetes worker sets `relinquish` whenever backoffLimit is not 0,
keeping Prefect's own reschedule and Kubernetes Job retries mutually exclusive.chuqCTC:fix/22620-abandon-sigterm-backofflimit fix: relinquish evicted flow runs to infrastructure retry
Closes #22620.
With a Kubernetes work pool at backoffLimit > 0, a SIGTERM (eviction) made
the in-pod engine finalize the run Crashed, but Kubernetes then retried the
Job and re-ran the flow, so the run left a terminal state and executed twice.
Teach `prefect flow-run execute` a termination behavior via
PREFECT_FLOW_RUN_EXECUTE_SIGTERM_BEHAVIOR (`crash`, `reschedule` or
`relinquish`) and deliver it to the engine as an intent over the control
channel that already connects the two. On SIGTERM the supervisor signals the
intent, proposes AwaitingRetry only for `reschedule`, kills the engine, and
exits non-zero for `relinquish` so Kubernetes counts the attempt and retries.
The engine now dispatches on intent alone and proposes no state when someone
else owns it, so it no longer reads Kubernetes-specific configuration. Any
intent also exits the engine cleanly, matching how cancellation already works,
so the supervisor does not have to suppress a crash from the exit code.
Ordinary cancellations carry no intent and still crash the run.
The engine child now starts in its own process group on POSIX, matching the
existing Windows behavior, so an external SIGTERM cannot reach it before the
supervisor has signalled its intent.
The kubernetes worker sets `relinquish` whenever backoffLimit is not 0,
keeping Prefect's own reschedule and Kubernetes Job retries mutually exclusive.chuqCTC:fix/22620-abandon-sigterm-backofflimit Latest Branches
0%
qdivan:fix/process-worker-sigterm-drain-22528 0%
devin/1785249110-suspension-event-confirmation 0%
g-ringeisen:fix-for-issue-22639 © 2026 CodSpeed Technology