Eventual-Inc
Daft
Blog
Docs
Changelog
Blog
Docs
Changelog
Overview
Branches
Benchmarks
Runs
Performance History
Latest Results
perf(parquet): order pipelined predicate groups by stats-derived benefit Refine the LM-pipelined group ordering so the first-decoded group narrows the RowSelection the most per unit of decode work. Rank is (selectivity - 1) / cost -- i.e. benefit (1 - selectivity) / cost sorted descending -- with both axes read from row-group Parquet metadata: - selectivity: extend numeric min/max interpolation with (a) Utf8 lexicographic min/max interpolation for string ranges (64-byte guard for possibly-truncated stats) and (b) equality via the column distinct_count (NDV) when the writer provides it, else a min/max containment test, else a constant. - cost: summed uncompressed column-chunk bytes replace the crude column-count proxy, so a narrow selective group orders ahead of a wide one at equal selectivity. Kill-switches DAFT_PARQUET_LM_STR_STATS and DAFT_PARQUET_LM_COST_BYTES (default on) join DAFT_PARQUET_LM_PIPELINE so ordering can be A/B'd and rolled back. Ordering still affects decode work only, never the result set: any estimation miss degrades to a neutral order. Adds unit tests for NDV-driven equality selectivity and byte-cost tie-breaking, migrating the ordering tests onto a shared SelCtx. Measured on a 3M-row file: string-stats ordering 5.7x faster than the mis-ordered baseline; byte-cost ordering 1.66x at an exact selectivity tie.
hello-peter-tang:perf/6340-lm-pipelined-predicate-eval
1 hour ago
feat(io): add Iceberg COW table changelog (CDC) reads Adds daft.read_iceberg_changes(), reading a copy-on-write Iceberg snapshot range as an INSERT/DELETE changelog stream with carryover rows removed. MOR (position/equality deletes) and multi-schema tables are explicitly rejected rather than silently approximated. - src/daft-parquet: new read_parquet_arrow_schema footer-only binding that returns the raw pre-Daft-downcast Arrow schema, preserving nested struct/list/map field metadata (needed to verify Iceberg field IDs at every nesting level, which Daft's own List/Map types can't represent). - daft/io/iceberg/_changelog_planning.py: snapshot range resolution, delete-manifest safety guard, and ADDED/DELETED file task planning. - daft/io/iceberg/_changelog_schema.py: two-layer schema safety gate -- a cheap single-schema fast-reject plus a recursive per-file Parquet footer verification against the baseline schema at every nesting level. - daft/io/iceberg/iceberg_changes_scan.py: IcebergChangesDataSource, built on the DataSource API, attaching _change_type/_change_ordinal/ _commit_snapshot_id as constant columns via DataSourceTask.parquet()'s partition_values mechanism, with pickle-safe lazy planning/validation caching shared across get_tasks() calls. - daft/io/iceberg/_changelog_postprocess.py: remove_carryovers, cancelling same-commit DELETE+INSERT pairs of identical rows via window functions.
kyo-tom:iceberg-cdc-datasource
3 hours ago
build: bump ray to include and dev against 2.58 - throw in cheap uv update too
NellyWhads:ray-258
3 hours ago
build: bump ray to include and dev against 2.58 - throw in cheap uv update too
NellyWhads:ray-258
3 hours ago
build: bump ray to include and dev against 2.58 - throw in cheap uv update too
NellyWhads:ray-258
4 hours ago
style(core): split var/stddev doc summaries from their detail paragraphs
jonasdedden:fix/var-stddev-chan-merge
12 hours ago
docs(expression): trim to_datetime comments and docstring
jonasdedden:fix-7469-to-datetime
13 hours ago
style(core): use mul_add in var partial and trim var/stddev docs
jonasdedden:fix/var-stddev-chan-merge
13 hours ago
Latest Branches
CodSpeed Performance Gauge
0%
perf(parquet): LM-pipelined predicate evaluation with stats-based group ordering
#7488
2 hours ago
ed4ad1f
hello-peter-tang:perf/6340-lm-pipelined-predicate-eval
CodSpeed Performance Gauge
0%
feat(io): add Iceberg COW table changelog (CDC) reads
#7401
1 month ago
12aa597
kyo-tom:iceberg-cdc-datasource
CodSpeed Performance Gauge
0%
build: bump ray to include 2.58 and uv to 0.11.33
#7487
4 hours ago
4f3a9a6
NellyWhads:ray-258
© 2026 CodSpeed Technology
Home
Terms
Privacy
Docs