Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

fix(lance): narrow merge target URI

FANNG1:feat/daft-lance-0.5-integration

50 minutes ago

chore(lance)!: remove deprecated daft.io.lance module (#7514) ## Changes Made Removes the deprecated `daft.io.lance` helpers and defers the Lance write logic to the `daft-lance` community extension, per the v0.8.0 cleanup. - **`daft.io.lance`**: dropped `compact_files`, `create_scalar_index`, `merge_columns`, and `merge_columns_df`. The submodule now only exposes `read_lance`, kept as-is since it's a lightweight wrapper over `daft_lance.read_lance` with a convenient docstring. Deleted the seven pure forwarding shims (`lance_compaction`, `lance_data_sink`, `lance_merge_column`, `lance_scalar_index`, `lance_scan`, `point_lookup`, `utils`). - **`DataFrame.write_lance`**: non-merge modes now delegate straight to `daft_lance.write_lance`; merge mode keeps the Daft-side decision logic (create vs. append vs. per-fragment merge) but calls `daft_lance.merge_columns_df` instead of the removed local path. This also fixes a `TypeError` from the old `merge_columns_from_df` signature, which no longer matches daft-lance 0.5.0. - **Docs**: `docs/connectors/custom.md` links now point at the `daft-engine/daft-lance` repo instead of the deleted in-tree files. ## Related Issues Closes #7406

main

2 hours ago

fix: eliminate TOCTOU race in CREATE TABLE IF NOT EXISTS (#7387) ## Changes Made | File | Change | |---|---| | `src/daft-catalog/src/impls/memory.rs` | Check-then-insert is now atomic under a single write lock | | `src/daft-sql/src/exec.rs` | Creates first; treats "already exists" as success when `IF NOT EXISTS` is set | | `daft/catalog/__init__.py` | Same create-first pattern; falls back to `get_table` on `ValueError` | | `tests/catalog/test_catalog.py` | Covers idempotency and concurrent creation | - `CREATE TABLE IF NOT EXISTS` had a TOCTOU race: both the SQL executor and the Python API checked `has_table` before creating, so two concurrent callers could both pass the check and the loser failed with "already exists". Both paths now create first and catch that error instead. - The root cause sat in `MemoryCatalog::create_table`: the read-lock check and the write-lock insert were two separate steps, so the second concurrent creator silently overwrote the first. They are now one atomic step under a single write lock. - One subtlety: the table is built *before* taking the lock, because `MemoryTable::new` can call back into the Python runner and release the GIL — holding the lock across that callback would deadlock with a concurrent creator. - Error detection is dual-path: typed `CatalogError::ObjectAlreadyExists` for Rust catalogs, and the "already exists" message for Python-backed catalogs where the error is wrapped. Same idea as Spark's `CreateTableExec`. ## Related Issues Closes #7310

main

3 hours ago

fix: validate variadic inputs and accept all-null columns in @daft.func Cover *args/**kwargs in input-type validation, accept Null dtype (all-null columns) to match native validation, and hoist inline import to file scope. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Jorge-Polanco-Roque:fix/daf-func-signature-validation

3 hours ago

test(lance): verify writes with native Lance

FANNG1:feat/daft-lance-0.5-integration

4 hours ago

Merge branch 'main' into issue-2423

Lucas61000:issue-2423

4 hours ago

Merge branch 'main' into issue-7310

Lucas61000:issue-7310

4 hours ago

Merge remote-tracking branch 'origin/main' into issue-7406 # Conflicts: # daft/dataframe/dataframe.py # daft/io/lance/utils.py # pyproject.toml # tests/io/lancedb/test_lance_batching.py # uv.lock

Lucas61000:issue-7406

6 hours ago

Latest Branches

feat(lance): add lance namespace and insert overwrites mode for write lance#7518

1 hour ago

851df4c

FANNG1:feat/daft-lance-0.5-integration

fix: validate @daft.func input types at planning time#7508

3 hours ago

1c4abd2

Jorge-Polanco-Roque:fix/daf-func-signature-validation

feat(optimizer): add common subplan elimination optimization rule#6788

5 hours ago

c698733

Lucas61000:issue-2423

Home Terms Privacy Docs