Eventual-Inc/Daft

Daft

Performance History

Latest Results

feat(iceberg): add overwrite_filter for static partition overwrite write_iceberg(mode="overwrite") always replaced the whole table, leaving no way to replace a single partition. overwrite_filter scopes the delete to the rows matching a predicate, with the delete and the append still landing in one Iceberg transaction: df.write_iceberg(table, mode="overwrite", overwrite_filter="dt = '2024-01-01'") The filter accepts an Iceberg predicate string, a Daft expression, or a PyIceberg BooleanExpression, and is bound against the table schema before the write runs so an invalid predicate fails before any data files are staged. By default the rows being written must match the filter as well, so an overwrite cannot leave behind rows its delete did not cover. The check follows Iceberg's OverwriteFiles.validateAddedFilesMatchOverwriteFilter, accepting a file when its partition value proves every row matches or when its column statistics do. Both tests are conservative and statistics only ever widen, so it may refuse a write it cannot prove but never accepts an unmatched row. Pass validate_overwrite_filter=False to skip it. Also fixes the partitioning column of the returned DataFrame, which reported null for every partition on pyiceberg >= 0.9: data_file.partition is a positional Record that no longer exposes fields as attributes, so the getattr lookup always missed. Values are now read by index against the spec the file was written with, and keyed by Iceberg partition field name instead of source column name, which also stops two partition fields over one source column from collapsing into a single key.

atovk:feat/iceberg-overwrite-filter

6 minutes ago

fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs PyIceberg installs a LegacyOAuth2AuthManager even when no credential is configured, and its `auth_header()` formats the unset token unconditionally -- so `_open_iceberg_table_via_rest` sent a literal `Authorization: Bearer None`. REST catalogs that require no auth reject that with 401 rather than ignoring it, making every unauthenticated Gravitino Iceberg REST catalog unreadable. Declare `auth={"type": "noop"}` when none of `credential`/`token`/`auth` are present in the catalog properties, so no Authorization header is sent. Catalogs that do configure bearer/OAuth2 auth keep their existing behavior, which the added parametrized test covers. Verified against a live Gravitino Iceberg REST service on pyiceberg 0.7.0, 0.8.1 and 0.11.1: only 0.11.1 sends `Bearer None` and 401s; on the older versions the extra property is inert, so the change is a no-op there.

kyo-tom:fix/gravitino-iceberg-rest-noop-auth

6 hours ago

fix(shuffle): stop into_partitions coalesce from panicking under flight shuffle

BABTUNA:fix/into-partitions-flight-coalesce

7 hours ago

ci: retrigger to clear flaky dead-plan-reuse error masking (#7169)

BABTUNA:refactor/random-shuffle-shared-read-tasks

8 hours ago

Merge branch 'main' into fix/coalesce-unreachable-validation

DogerW666:fix/coalesce-unreachable-validation

10 hours ago

fix(functions): warn that single-argument coalesce is a no-op Per review feedback: coalesce(<expr>) == <expr> is correct but can be confusing, so emit a UserWarning instead of silently passing through.

DogerW666:fix/coalesce-unreachable-validation

11 hours ago

fix: match already-exists errors by type in SQL path

Lucas61000:issue-7310

11 hours ago

refactor(shuffle): migrate random shuffle reads onto shared read-task emission

BABTUNA:refactor/random-shuffle-shared-read-tasks

11 hours ago

Latest Branches

feat(iceberg): add overwrite_filter for static partition overwrite#7419

44 minutes ago

bf49d92

atovk:feat/iceberg-overwrite-filter

fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs#7417

9 hours ago

f45d663

kyo-tom:fix/gravitino-iceberg-rest-noop-auth

fix(shuffle): stop into_partitions coalesce from panicking under flight shuffle#7414

7 hours ago

5fea4b5

BABTUNA:fix/into-partitions-flight-coalesce

Home Terms Privacy Docs