Branches performance
Pull requests
last run
2 hours ago
Bump packaging upper bound from <25.1 to <26.1
2 hours ago
4325162
purificant:bump-packaging-upper-bound
0%
last run
2 hours ago
fix: resolve deployment concurrency slot lease release race condition
Fixes critical race condition where deployment concurrency lease_id is
read from the wrong state object after propose_state() overwrites the
flow run's state, causing slots to remain occupied until TTL expires.
Also prevents flows from being incorrectly marked as Crashed when
BaseExceptions occur during post-execution state transitions.
## Root Cause
### Race Condition (Issue #17415)
In FlowRunEngine.set_state() (both sync and async):
1. Flow in RUNNING state with lease_id in state_details
2. propose_state(Completed()) returns new COMPLETED state
3. propose_state() OVERWRITES self.flow_run.state with new state
4. Code reads lease_id from self.flow_run.state (now COMPLETED with lease_id=null)
5. release_concurrency_slots_with_lease() never called
6. Slot remains occupied until 300s TTL expiration
7. Other flows stuck in AwaitingConcurrencySlot state
### False Crash Detection (Issue #19068)
When flows complete successfully but encounter infrastructure issues:
1. User code completes successfully
2. set_state(Completed()) makes API call
3. Lease renewal or API timeout raises BaseException
4. Existing code: "BaseException + not final state = crash!"
5. Flow incorrectly marked as CRASHED despite successful execution
## Changes
### Client-Side Fix (flow_engine.py)
**Lease Release Timing Fix:**
- Capture lease_id from current state BEFORE calling propose_state()
- Use saved lease_id value for release API call
- Applied to both sync (lines 507-545) and async (lines 1145-1180)
**False Crash Prevention:**
- Add `_flow_executed` flag to track when user code finishes
- Set flag in handle_success() after user code completes
- Update BaseException handler to check flag before marking as crashed
- Prevents infrastructure issues from masking successful executions
### Server-Side Fixes (core_policy.py)
- Fix SecureFlowConcurrencySlots.cleanup() to only decrement if lease exists
- Restore ReleaseFlowConcurrencySlots.after_transition() with version checking
### Backwards Compatibility (dependencies.py + core_policy.py)
- Added MIN_CLIENT_VERSION_FOR_CLIENT_SIDE_LEASE_RELEASE = Version("3.6.23")
- Old clients (< 3.6.23 or None): Server auto-releases (prevents orphans)
- New clients (>= 3.6.23): Client-side release only (avoids race)
- Defaults to old behavior when version unknown (safe default)
## Testing
- All 70 existing TestFlowConcurrencyLimits tests pass
- Parameterized 4 key tests for old/new client behaviors
- Added lease renewal failure resilience tests
Closes #17415
Closes #19068
Related: #18942
2 hours ago
2dd4274
ubiquitousbyte:fix/deployment-concurrency-lease-release
0%
last run
4 hours ago
Add end_time filter support for flow & task runs
4 hours ago
b99a29f
jellyfish0316:feature/end-time-filter
0%
last run
7 minutes ago
test(worker): update assertions and add tests for create_pool_if_not_found flag
Fix the 5 failing "Runner, Worker, Settings, Input, and CLI Tests" CI checks.
The new create_pool_if_not_found kwarg added in the previous commit is now
passed to worker_cls(...), but 6 existing mock-based tests asserted the call
signature without it, causing strict assert_called_once_with() failures.
Changes:
- Add create_pool_if_not_found=True to all 6 existing mock_worker.assert_called_once_with() calls
- Add test_start_worker_create_pool_if_not_found_default: verifies the flag defaults to True
- Add test_start_worker_no_create_pool_if_not_found: verifies --no-create-pool-if-not-found passes False
11 minutes ago
f0262df
VittoriaLanzo:fix/20980-no-create-pool-if-not-found
0%