Latest Results
perf(server): stop fragmenting task run recorder bulk upserts
`task-run-recorder` batched rows for bulk insert by
`(frozenset(task_run_dict.keys()), conflict_target)`, merging only contiguous
runs of an identical key over a list sorted by `_task_run_upsert_key`. Both
halves of that key vary independently of the sort: the column set varies because
the client emitter dumps with `exclude_none=True`, and the conflict target is
resolved per row against what is already in the database. Equal keys are
therefore rarely adjacent, so a 300-task-run flush executed 300 single-row
INSERT statements and the service became bound by round trips rather than CPU.
Grouping by signature across the whole flush is not available. #22112 made
grouping contiguous precisely so that a sorted input like `A, B, A` cannot
execute as `A, A, B`, which would weaken the row-level lock ordering concurrent
recorders depend on.
Extend a batch for as long as it is safe to instead, in one left-to-right pass
over the sorted rows. A row joins the open batch unless it upserts on a
different conflict target, would need a NULL in a column where NULL is not the
same as absent, repeats a conflict target already in the batch, or would push
the statement past the driver's bind-parameter limit. Absent keys are NULL-filled
and restored in the SET clause with `coalesce(excluded.col, task_run.col)`.
Batches remain contiguous blocks of the sorted list emitted in order, so the
#22112 guarantee is preserved. On a 300-task-run flush this gives one statement
on PostgreSQL and eight on SQLite.
The repeated-conflict-target condition also closes a pre-existing data-loss path
that the fragmentation was masking by accident: two distinct conflict groups can
resolve to the same canonical id, which PostgreSQL rejects with
CardinalityViolationError (not an IntegrityError, so never retried) and SQLite
applies as silent last-write-wins.
The `coalesce` applies to exactly the columns that were NULL-filled, so a payload
asking to write a real null still writes one and the batch splits instead. Stored
values are unchanged from before this commit in every case.
Also fixes a `KeyError` in the same function, which has to be fixed for the
repeated-conflict-target case above to be reachable at all. `conflict_target`
rewrites `task_run.id` in its third branch, which changed the row's upsert key
mid-flush and left the canonical-id fix-up looking up a key that no longer
existed. `KeyError` is not an `IntegrityError`, so it was never retried: the
consumer's `flush()` requeued the batch and dropped every event in it after
`max_persist_retries`. Each row's upsert key is now pinned before the rewrite
can happen.chuqCTC:fix/task-run-recorder-batch-fragmentation perf(server): stop fragmenting task run recorder bulk upserts
`task-run-recorder` batched rows for bulk insert by
`(frozenset(task_run_dict.keys()), conflict_target)`, merging only contiguous
runs of an identical key over a list sorted by `_task_run_upsert_key`. Both
halves of that key vary independently of the sort: the column set varies because
the client emitter dumps with `exclude_none=True`, and the conflict target is
resolved per row against what is already in the database. Equal keys are
therefore rarely adjacent, so a 300-task-run flush executed 300 single-row
INSERT statements and the service became bound by round trips rather than CPU.
Grouping by signature across the whole flush is not available. #22112 made
grouping contiguous precisely so that a sorted input like `A, B, A` cannot
execute as `A, A, B`, which would weaken the row-level lock ordering concurrent
recorders depend on.
Extend a batch for as long as it is safe to instead, in one left-to-right pass
over the sorted rows. A row joins the open batch unless it upserts on a
different conflict target, would need a NULL in a column where NULL is not the
same as absent, repeats a conflict target already in the batch, or would push
the statement past the driver's bind-parameter limit. Absent keys are NULL-filled
and restored in the SET clause with `coalesce(excluded.col, task_run.col)`.
Batches remain contiguous blocks of the sorted list emitted in order, so the
#22112 guarantee is preserved. On a 300-task-run flush this gives one statement
on PostgreSQL and eight on SQLite.
The repeated-conflict-target condition also closes a pre-existing data-loss path
that the fragmentation was masking by accident: two distinct conflict groups can
resolve to the same canonical id, which PostgreSQL rejects with
CardinalityViolationError (not an IntegrityError, so never retried) and SQLite
applies as silent last-write-wins.
The `coalesce` applies to exactly the columns that were NULL-filled, so a payload
asking to write a real null still writes one and the batch splits instead. Stored
values are unchanged from before this commit in every case.
Also fixes a `KeyError` in the same function, which has to be fixed for the
repeated-conflict-target case above to be reachable at all. `conflict_target`
rewrites `task_run.id` in its third branch, which changed the row's upsert key
mid-flush and left the canonical-id fix-up looking up a key that no longer
existed. `KeyError` is not an `IntegrityError`, so it was never retried: the
consumer's `flush()` requeued the batch and dropped every event in it after
`max_persist_retries`. Each row's upsert key is now pinned before the rewrite
can happen.chuqCTC:fix/task-run-recorder-batch-fragmentation perf(server): stop fragmenting task run recorder bulk upserts
`task-run-recorder` batched rows for bulk insert by
`(frozenset(task_run_dict.keys()), conflict_target)`, merging only contiguous
runs of an identical key over a list sorted by `_task_run_upsert_key`. Both
halves of that key vary independently of the sort: the column set varies because
the client emitter dumps with `exclude_none=True`, and the conflict target is
resolved per row against what is already in the database. Equal keys are
therefore rarely adjacent, so a 300-task-run flush executed 300 single-row
INSERT statements and the service became bound by round trips rather than CPU.
Grouping by signature across the whole flush is not available. #22112 made
grouping contiguous precisely so that a sorted input like `A, B, A` cannot
execute as `A, A, B`, which would weaken the row-level lock ordering concurrent
recorders depend on.
Extend a batch for as long as it is safe to instead, in one left-to-right pass
over the sorted rows. A row joins the open batch unless it upserts on a
different conflict target, would need a NULL in a column where NULL is not the
same as absent, repeats a conflict target already in the batch, or would push
the statement past the driver's bind-parameter limit. Absent keys are NULL-filled
and restored in the SET clause with `coalesce(excluded.col, task_run.col)`.
Batches remain contiguous blocks of the sorted list emitted in order, so the
#22112 guarantee is preserved. On a 300-task-run flush this gives one statement
on PostgreSQL and eight on SQLite.
The repeated-conflict-target condition also closes a pre-existing data-loss path
that the fragmentation was masking by accident: two distinct conflict groups can
resolve to the same canonical id, which PostgreSQL rejects with
CardinalityViolationError (not an IntegrityError, so never retried) and SQLite
applies as silent last-write-wins.
The `coalesce` applies to exactly the columns that were NULL-filled, so a payload
asking to write a real null still writes one and the batch splits instead. Stored
values are unchanged from before this commit in every case.
Also fixes a `KeyError` in the same function, which has to be fixed for the
repeated-conflict-target case above to be reachable at all. `conflict_target`
rewrites `task_run.id` in its third branch, which changed the row's upsert key
mid-flush and left the canonical-id fix-up looking up a key that no longer
existed. `KeyError` is not an `IntegrityError`, so it was never retried: the
consumer's `flush()` requeued the batch and dropped every event in it after
`max_persist_retries`. Each row's upsert key is now pinned before the rewrite
can happen.chuqCTC:fix/task-run-recorder-batch-fragmentation perf(server): stop fragmenting task run recorder bulk upserts
`task-run-recorder` batched rows for bulk insert by
`(frozenset(task_run_dict.keys()), conflict_target)`, merging only contiguous
runs of an identical key over a list sorted by `_task_run_upsert_key`. Both
halves of that key vary independently of the sort: the column set varies because
the client emitter dumps with `exclude_none=True`, and the conflict target is
resolved per row against what is already in the database. Equal keys are
therefore rarely adjacent, so a 300-task-run flush executed 300 single-row
INSERT statements and the service became bound by round trips rather than CPU.
Grouping by signature across the whole flush is not available. #22112 made
grouping contiguous precisely so that a sorted input like `A, B, A` cannot
execute as `A, A, B`, which would weaken the row-level lock ordering concurrent
recorders depend on.
Extend a batch for as long as it is safe to instead, in one left-to-right pass
over the sorted rows. A row joins the open batch unless it upserts on a
different conflict target, would need a NULL in a column where NULL is not the
same as absent, repeats a conflict target already in the batch, or would push
the statement past the driver's bind-parameter limit. Absent keys are NULL-filled
and restored in the SET clause with `coalesce(excluded.col, task_run.col)`.
Batches remain contiguous blocks of the sorted list emitted in order, so the
#22112 guarantee is preserved. On a 300-task-run flush this gives one statement
on PostgreSQL and eight on SQLite.
The repeated-conflict-target condition also closes a pre-existing data-loss path
that the fragmentation was masking by accident: two distinct conflict groups can
resolve to the same canonical id, which PostgreSQL rejects with
CardinalityViolationError (not an IntegrityError, so never retried) and SQLite
applies as silent last-write-wins.
Behaviour change: a payload that explicitly sets a fillable column to null no
longer clears it. `flow_run_id` is deliberately excluded, so an event that drops
its `flow-run` related resource still detaches the task run.
Also fixes a `KeyError` in the same function, which has to be fixed for the
repeated-conflict-target case above to be reachable at all. `conflict_target`
rewrites `task_run.id` in its third branch, which changed the row's upsert key
mid-flush and left the canonical-id fix-up looking up a key that no longer
existed. `KeyError` is not an `IntegrityError`, so it was never retried: the
consumer's `flush()` requeued the batch and dropped every event in it after
`max_persist_retries`. Each row's upsert key is now pinned before the rewrite
can happen.chuqCTC:fix/task-run-recorder-batch-fragmentation perf(server): stop fragmenting task run recorder bulk upserts
`task-run-recorder` batched rows for bulk insert by
`(frozenset(task_run_dict.keys()), conflict_target)`, merging only contiguous
runs of an identical key over a list sorted by `_task_run_upsert_key`. Both
halves of that key vary independently of the sort: the column set varies because
the client emitter dumps with `exclude_none=True`, and the conflict target is
resolved per row against what is already in the database. Equal keys are
therefore rarely adjacent, so a 300-task-run flush executed 300 single-row
INSERT statements and the service became bound by round trips rather than CPU.
Grouping by signature across the whole flush is not available. #22112 made
grouping contiguous precisely so that a sorted input like `A, B, A` cannot
execute as `A, A, B`, which would weaken the row-level lock ordering concurrent
recorders depend on.
Extend a batch for as long as it is safe to instead, in one left-to-right pass
over the sorted rows. A row joins the open batch unless it upserts on a
different conflict target, would need a NULL in a column where NULL is not the
same as absent, repeats a conflict target already in the batch, or would push
the statement past the driver's bind-parameter limit. Absent keys are NULL-filled
and restored in the SET clause with `coalesce(excluded.col, task_run.col)`.
Batches remain contiguous blocks of the sorted list emitted in order, so the
#22112 guarantee is preserved. On a 300-task-run flush this gives one statement
on PostgreSQL and eight on SQLite.
The repeated-conflict-target condition also closes a pre-existing data-loss path
that the fragmentation was masking by accident: two distinct conflict groups can
resolve to the same canonical id, which PostgreSQL rejects with
CardinalityViolationError (not an IntegrityError, so never retried) and SQLite
applies as silent last-write-wins.
Behaviour change: a payload that explicitly sets a fillable column to null no
longer clears it. `flow_run_id` is deliberately excluded, so an event that drops
its `flow-run` related resource still detaches the task run.
Also fixes a `KeyError` in the same function, which has to be fixed for the
repeated-conflict-target case above to be reachable at all. `conflict_target`
rewrites `task_run.id` in its third branch, which changed the row's upsert key
mid-flush and left the canonical-id fix-up looking up a key that no longer
existed. `KeyError` is not an `IntegrityError`, so it was never retried: the
consumer's `flush()` requeued the batch and dropped every event in it after
`max_persist_retries`. Each row's upsert key is now pinned before the rewrite
can happen.chuqCTC:fix/task-run-recorder-batch-fragmentation Latest Branches
0%
chuqCTC:fix/task-run-recorder-batch-fragmentation 0%
tianrking:fix/process-worker-crash-hooks 0%
devin/1785929288-process-worker-auto-uv © 2026 CodSpeed Technology