Eventual-Inc/Daft - CodSpeed

Daft

Blog Docs Changelog

Performance History

Latest Results

perf(parquet): order pipelined predicate groups by stats-derived benefit Refine the LM-pipelined group ordering so the first-decoded group narrows the RowSelection the most per unit of decode work. Rank is (selectivity - 1) / cost -- i.e. benefit (1 - selectivity) / cost sorted descending -- with both axes read from row-group Parquet metadata: - selectivity: extend numeric min/max interpolation with (a) Utf8 lexicographic min/max interpolation for string ranges (64-byte guard for possibly-truncated stats) and (b) equality via the column distinct_count (NDV) when the writer provides it, else a min/max containment test, else a constant. - cost: summed uncompressed column-chunk bytes replace the crude column-count proxy, so a narrow selective group orders ahead of a wide one at equal selectivity. Kill-switches DAFT_PARQUET_LM_STR_STATS and DAFT_PARQUET_LM_COST_BYTES (default on) join DAFT_PARQUET_LM_PIPELINE so ordering can be A/B'd and rolled back. Ordering still affects decode work only, never the result set: any estimation miss degrades to a neutral order. Adds unit tests for NDV-driven equality selectivity and byte-cost tie-breaking, migrating the ordering tests onto a shared SelCtx. Measured on a 3M-row file: string-stats ordering 5.7x faster than the mis-ordered baseline; byte-cost ordering 1.66x at an exact selectivity tie.

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

1 hour ago

feat(io): add Iceberg COW table changelog (CDC) reads Adds daft.read_iceberg_changes(), reading a copy-on-write Iceberg snapshot range as an INSERT/DELETE changelog stream with carryover rows removed. MOR (position/equality deletes) and multi-schema tables are explicitly rejected rather than silently approximated. - src/daft-parquet: new read_parquet_arrow_schema footer-only binding that returns the raw pre-Daft-downcast Arrow schema, preserving nested struct/list/map field metadata (needed to verify Iceberg field IDs at every nesting level, which Daft's own List/Map types can't represent). - daft/io/iceberg/_changelog_planning.py: snapshot range resolution, delete-manifest safety guard, and ADDED/DELETED file task planning. - daft/io/iceberg/_changelog_schema.py: two-layer schema safety gate -- a cheap single-schema fast-reject plus a recursive per-file Parquet footer verification against the baseline schema at every nesting level. - daft/io/iceberg/iceberg_changes_scan.py: IcebergChangesDataSource, built on the DataSource API, attaching _change_type/_change_ordinal/ _commit_snapshot_id as constant columns via DataSourceTask.parquet()'s partition_values mechanism, with pickle-safe lazy planning/validation caching shared across get_tasks() calls. - daft/io/iceberg/_changelog_postprocess.py: remove_carryovers, cancelling same-commit DELETE+INSERT pairs of identical rows via window functions.

kyo-tom:iceberg-cdc-datasource

3 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

3 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

3 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

4 hours ago

style(core): split var/stddev doc summaries from their detail paragraphs

jonasdedden:fix/var-stddev-chan-merge

12 hours ago

docs(expression): trim to_datetime comments and docstring

jonasdedden:fix-7469-to-datetime

13 hours ago

style(core): use mul_add in var partial and trim var/stddev docs

jonasdedden:fix/var-stddev-chan-merge

13 hours ago

Latest Branches

0%

perf(parquet): LM-pipelined predicate evaluation with stats-based group ordering#7488

2 hours ago

ed4ad1f

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

0%

feat(io): add Iceberg COW table changelog (CDC) reads#7401

1 month ago

12aa597

kyo-tom:iceberg-cdc-datasource

0%

build: bump ray to include 2.58 and uv to 0.11.33#7487

4 hours ago

4f3a9a6

NellyWhads:ray-258

© 2026 CodSpeed Technology

Home Terms Privacy Docs