Latest Results
fix(asof): make join_asof output partition-invariant on (by, on) ties
join_asof's result depended on partition count and execution order when
several right rows shared the same (by, on) key: the operator kept whichever
row it saw first during the concurrent probe/merge, which varies with how
right batches split across workers and with shuffle order. The same query
could return different right-side values at different partition counts.
Resolve equal-on-key ties by value instead of arrival order: keep the greatest
right row for backward/nearest and the least for forward, compared
lexicographically over the on-key and the right output columns. Being a
per-key max/min over a total order, the choice is independent of execution
order, so the output is partition-invariant. This matches pandas merge_asof
(backward keeps the last row, forward the first).
- daft-local-execution: the tie-break folds into the existing per-batch-pair
comparator (nearest breaks only exact (distance, on-key) ties). It compares
the pruned output columns, which are already retained for output, so it pins
no extra memory.
- daft-distributed: the no-by carryover keeps every row tied at the boundary
on-key (whole-frame window + filter), mirroring the with-by path, so the
local operator stays the single place ties are broken.
Regression test test_no_by_duplicate_keys_partition_invariant asserts the
result is identical at 1/2/4/8 partitions across all three strategies.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>chuqCTC:asof-deterministic-tie-break Latest Branches
0%
chuqCTC:asof-deterministic-tie-break 0%
chuqCTC:fix/escape-sql-string-literals 0%
dwallener:codex/add-to-map-function © 2026 CodSpeed Technology