Eventual-Inc/Daft - CodSpeed

Daft

Blog Docs Changelog

Performance History

Latest Results

ci(integration): build Iceberg test image from python:3.10-bookworm The `integration-test-catalogs` job fails on every PR at the "Spin up services" step. The Iceberg docker-compose image builds FROM python:3.10-bullseye and installs openjdk-11-jdk in one RUN. bullseye's apt index/pool is now out of sync (and its bullseye-security Release file has expired), so `apt-get install` resolves package versions that 404 on download: E: Failed to fetch .../openjdk-11-jdk_11.0.32.1+1-1~deb11u1_amd64.deb 404 Not Found ERROR: process "/bin/sh -c apt-get ... install ..." exit code: 100 The build failure aborts `docker compose up`, so the `rest` catalog (port 8181) never starts and the dependent catalog tests cascade-fail with connection-refused. Fix: move the base image to python:3.10-bookworm, whose Debian 12 security suite is actively synced. bookworm ships openjdk-17 (not openjdk-11), and Spark 3.4.2 fully supports Java 17, so switch the JDK package to openjdk-17-jdk. No other change is needed: - check-license uses $JAVA_HOME/bin/java when set and otherwise falls back to `java` on PATH (Debian alternatives point it at 17); there is no hardcoded java-11 path. - The Gravitino compose stack uses prebuilt images (apache/gravitino, mysql, minio) with no Dockerfile, so it is unaffected by bullseye. Fixes #7476

hello-peter-tang:fix/7476-iceberg-dockerfile-bookworm

5 hours ago

perf(parquet): order pipelined predicate groups by stats-derived benefit Refine the LM-pipelined group ordering so the first-decoded group narrows the RowSelection the most per unit of decode work. Rank is (selectivity - 1) / cost -- i.e. benefit (1 - selectivity) / cost sorted descending -- with both axes read from row-group Parquet metadata: - selectivity: extend numeric min/max interpolation with (a) Utf8 lexicographic min/max interpolation for string ranges (64-byte guard for possibly-truncated stats) and (b) equality via the column distinct_count (NDV) when the writer provides it, else a min/max containment test, else a constant. - cost: summed uncompressed column-chunk bytes replace the crude column-count proxy, so a narrow selective group orders ahead of a wide one at equal selectivity. Kill-switches DAFT_PARQUET_LM_STR_STATS and DAFT_PARQUET_LM_COST_BYTES (default on) join DAFT_PARQUET_LM_PIPELINE so ordering can be A/B'd and rolled back. Ordering still affects decode work only, never the result set: any estimation miss degrades to a neutral order. Adds unit tests for NDV-driven equality selectivity and byte-cost tie-breaking, migrating the ordering tests onto a shared SelCtx. Measured on a 3M-row file: string-stats ordering 5.7x faster than the mis-ordered baseline; byte-cost ordering 1.66x at an exact selectivity tie.

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

8 hours ago

feat(io): add Iceberg COW table changelog (CDC) reads Adds daft.read_iceberg_changes(), reading a copy-on-write Iceberg snapshot range as an INSERT/DELETE changelog stream with carryover rows removed. MOR (position/equality deletes) and multi-schema tables are explicitly rejected rather than silently approximated. - src/daft-parquet: new read_parquet_arrow_schema footer-only binding that returns the raw pre-Daft-downcast Arrow schema, preserving nested struct/list/map field metadata (needed to verify Iceberg field IDs at every nesting level, which Daft's own List/Map types can't represent). - daft/io/iceberg/_changelog_planning.py: snapshot range resolution, delete-manifest safety guard, and ADDED/DELETED file task planning. - daft/io/iceberg/_changelog_schema.py: two-layer schema safety gate -- a cheap single-schema fast-reject plus a recursive per-file Parquet footer verification against the baseline schema at every nesting level. - daft/io/iceberg/iceberg_changes_scan.py: IcebergChangesDataSource, built on the DataSource API, attaching _change_type/_change_ordinal/ _commit_snapshot_id as constant columns via DataSourceTask.parquet()'s partition_values mechanism, with pickle-safe lazy planning/validation caching shared across get_tasks() calls. - daft/io/iceberg/_changelog_postprocess.py: remove_carryovers, cancelling same-commit DELETE+INSERT pairs of identical rows via window functions.

kyo-tom:iceberg-cdc-datasource

10 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

10 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

10 hours ago

build: bump ray to include and dev against 2.58 - throw in cheap uv update too

NellyWhads:ray-258

11 hours ago

style(core): split var/stddev doc summaries from their detail paragraphs

jonasdedden:fix/var-stddev-chan-merge

19 hours ago

docs(expression): trim to_datetime comments and docstring

jonasdedden:fix-7469-to-datetime

20 hours ago

Latest Branches

0%

ci(integration): build Iceberg test image from python:3.10-bookworm#7489

6 hours ago

58313ab

hello-peter-tang:fix/7476-iceberg-dockerfile-bookworm

0%

perf(parquet): LM-pipelined predicate evaluation with stats-based group ordering#7488

9 hours ago

ed4ad1f

hello-peter-tang:perf/6340-lm-pipelined-predicate-eval

0%

feat(io): add Iceberg COW table changelog (CDC) reads#7401

1 month ago

12aa597

kyo-tom:iceberg-cdc-datasource

© 2026 CodSpeed Technology

Home Terms Privacy Docs