fix: mark followers as seen and clean up lost follower Redis keys
Bug 1: In preceding_event_confirmed, followers processed by leaders via
handler(waiter) were never marked as seen because they bypassed
event_is_processing. Downstream events following those followers would
park and become orphaned for PRECEDING_EVENT_LOOKBACK (15 min).
Fix: Call record_event_as_seen(waiter) after handler(waiter) succeeds.
Bug 2: In record_lost_follower_task_run_events, forget_follower was never
called after processing, leaving event:X and followers:Y keys permanently
in Redis with no TTL.
Fix: Call ordering.forget_follower(event) after record_task_run_event.
Closes #20954
Co-authored-by: Alex Streed <alex.s@prefect.io>
Co-Authored-By: alex.s <ajstreed1@gmail.com>
Separate raise_on_lease_renewal_failure from strict in concurrency()
The `strict` parameter currently controls two independent behaviors:
1. Whether to raise if the concurrency limit doesn't exist (acquisition)
2. Whether to terminate execution on lease renewal failure (renewal)
For long-running tasks (hours), users want strict acquisition (fail fast
if the limit doesn't exist) but resilient renewal (log a warning instead
of killing a 10-hour job on a transient HTTP blip).
This adds `raise_on_lease_renewal_failure` as a separate parameter to
the public `concurrency()` context manager:
- `None` (default): follows `strict` for backward compatibility
- `False`: continue execution on renewal failure (log warning)
- `True`: terminate execution on renewal failure
Also adds `exc_info` to lease renewal failure log messages so users and
the Prefect team can diagnose why renewals are failing (#19068).
Fix: use break instead of return in _clone_repo retry loop
The return statement was short-circuiting out of the entire method,
skipping the post-clone steps (sparse-checkout set, commit fetch/checkout).
Using break exits only the retry loop, allowing those steps to execute.
Co-authored-by: bot_apk <apk@cognition.ai>