Eventual-Inc/Daft

Daft

Performance History

Latest Results

fix(dashboard): normalize and unconditionally honor DAFT_RAY_DASHBOARD_URL Addresses review feedback on the override: - Normalize the override the same way as the auto-detected URL: prepend `http://` when no scheme is given, so a bare hostname does not produce a broken link. - Check the override before the `RAY_DISABLE_DASHBOARD` guard. The two env vars serve different purposes: one turns off Ray's built-in dashboard, the other points at an externally hosted one, so disabling the former should no longer silently discard the latter. - Append the Ray job route (`/#/jobs/<job_id>`) to the override too, unless it already contains a `#` fragment, so overridden links deep-link to the job like auto-detected ones do. The resolution logic moves out of `run_iter` into `_resolve_ray_dashboard_url` so it is unit-testable, and is documented in the dashboard docs.

claude/ray-ui-url-investigation-Q9HBW

2 minutes ago

fix(catalog): declare noop auth for Gravitino Iceberg REST catalogs (#7417) ## Changes Made `_open_iceberg_table_via_rest` passed no auth configuration to PyIceberg's `RestCatalog`. On pyiceberg 0.11.1 that isn't the same as "no credentials": `RestCatalog` installs a `LegacyOAuth2AuthManager` whose `auth_header()` formats the unset token unconditionally, so Daft sent a literal `Authorization: Bearer None`. REST catalogs that require no auth reject that with 401, which makes them unreadable through Gravitino. Declare `auth={"type": "noop"}` when the catalog properties carry none of `credential` / `token` / `auth`, so no `Authorization` header is sent. Catalogs that do configure auth keep their current behavior. Tested against a live Gravitino Iceberg REST service on pyiceberg 0.7.0, 0.8.1 and 0.11.1 — only 0.11.1 was broken; on the older versions the extra property is inert, so this is a no-op there. AI disclosure: analysis and patch written with Claude; verified by hand against the live deployment and with `tests/catalog/test_gravitino.py`. ## Related Issues Closes #7416

main

2 hours ago

Merge branch 'main' into codex/agg-sum-literal-rewrite

codex/agg-sum-literal-rewrite

4 hours ago

fix(functions): make coalesce() argument validation reachable (#7404) The zero-argument ValueError in coalesce() was placed after an unconditional return statement, making it dead code. Calling coalesce() with no arguments silently constructed an invalid expression and only failed later at plan-building time with an opaque DaftCoreException. - Move the validation before the return so coalesce() raises ValueError eagerly - Remove the unreachable single-argument shortcut (the native kernel already handles single-argument coalesce) - Fix the docstring: single-argument coalesce is supported and covered by existing tests ## Changes Made The zero-argument `ValueError` in `coalesce()` was placed after an unconditional `return` statement, making it dead code. Calling `coalesce()` with no arguments silently constructed an invalid expression and only failed later at plan-building time with an opaque `DaftCoreException`. - Move the validation before the `return` so `coalesce()` raises `ValueError` eagerly at expression construction time - Remove the unreachable single-argument shortcut (the native kernel already handles single-argument coalesce, and existing tests cover it) - Fix the docstring: single-argument coalesce is supported ("Two or more" → "One or more") - Add a regression test `test_coalesce_no_arguments_raises` ## Related Issues Closes #7403 --------- Co-authored-by: Daoke <wangdaoshuai.wds@alibaba-inc.com>

main

4 hours ago

fix ci

slade/azure-tests

4 hours ago

fix(iceberg): apply partition predicates to count pushdown count_rows() on an identity-partitioned Iceberg table returned the whole table's row count whenever the only predicate was on the partition column. PushDownFilter moves a pure identity-transform partition predicate out of `filters` and into `partition_filters`, and the count-pushdown guard in push_down_aggregation.rs only inspects `filters`, so it reads that as "no filter" and pushes the aggregation into the source. `_create_count_tasks` then summed record counts across every data file without applying either filter, unlike `_create_regular_tasks`, which prunes each file against `partition_filters`. Prune the same way in the count path, and fall back to a regular scan when `filters` is set, since a row-level predicate cannot be answered from record counts. Partition pruning is exact for the predicates that reach here, so the count stays metadata-only rather than falling back to reading data. Also documents the pushdown contract on DataSource.supports_count_pushdown: a source that absorbs a count must honor partition_filters, which can be set while filters is None. Closes #7420

atovk:fix/iceberg-count-pushdown-partition-filters

8 hours ago

fix(iceberg): report partition fields from older specs in overwrite results A deleted file written under an earlier partition spec carries that spec's fields. The result struct was keyed by the current spec alone, so values decoded from an older spec were dropped and reported as null. Key the struct by the union of the current spec's fields and the fields present on the files being reported, which also merges the duplicated ADD and DELETE reporting loops.

atovk:feat/iceberg-overwrite-filter

12 hours ago

feat(iceberg): add overwrite_filter for static partition overwrite write_iceberg(mode="overwrite") always replaced the whole table, leaving no way to replace a single partition. overwrite_filter scopes the delete to the rows matching a predicate, with the delete and the append still landing in one Iceberg transaction: df.write_iceberg(table, mode="overwrite", overwrite_filter="dt = '2024-01-01'") The filter accepts an Iceberg predicate string, a Daft expression, or a PyIceberg BooleanExpression, and is bound against the table schema before the write runs so an invalid predicate fails before any data files are staged. By default the rows being written must match the filter as well, so an overwrite cannot leave behind rows its delete did not cover. The check follows Iceberg's OverwriteFiles.validateAddedFilesMatchOverwriteFilter, accepting a file when its partition value proves every row matches or when its column statistics do. Both tests are conservative and statistics only ever widen, so it may refuse a write it cannot prove but never accepts an unmatched row. Pass validate_overwrite_filter=False to skip it. Also fixes the partitioning column of the returned DataFrame, which reported null for every partition on pyiceberg >= 0.9: data_file.partition is a positional Record that no longer exposes fields as attributes, so the getattr lookup always missed. Values are now read by index against the spec the file was written with, and keyed by Iceberg partition field name instead of source column name, which also stops two partition fields over one source column from collapsing into a single key.

atovk:feat/iceberg-overwrite-filter

13 hours ago

Latest Branches

feat(dashboard): add DAFT_RAY_DASHBOARD_URL env var override#6652

15 minutes ago

72335d6

claude/ray-ui-url-investigation-Q9HBW

perf(agg): rewrite sum over integer literals#6582

4 hours ago

1547eb6

codex/agg-sum-literal-rewrite

test: Add Azurite integration tests#7410

4 hours ago

59802af

slade/azure-tests

Home Terms Privacy Docs