Latest Results
fix(dataframe): preserve explain() plan for all write sinks (#6580)
PR #6564 fixed `explain()` for `write_parquet`/`write_csv`/`write_json`
by using `_get_current_builder()` to preserve the original logical plan
after execution. However, all other write sinks still collapsed the plan
to an in-memory source.
The root cause: methods like `write_iceberg`, `write_deltalake`, and
`write_sink` construct a new DataFrame via `from_pydict()` or
`_from_micropartitions()` with a different schema than the write plan
output (e.g. Iceberg returns `{operation, rows, file_size, file_name}`
while the logical plan outputs `{data_file}`). The
`_get_current_builder()` swap doesn't work here because of the schema
mismatch.
To fix this, we add a `_source_builder` field to `DataFrame` that stores
the original write plan builder. When set, `explain()` displays the full
source-to-sink plan instead of the in-memory scan. This field is set in:
- `write_iceberg` and `write_deltalake` (post-commit summary DataFrames)
- `write_sink` and all its delegates (`write_sql`, `write_paimon`,
`write_turbopuffer`, `write_clickhouse`, `write_huggingface`,
`write_bigtable`)
- `write_lance` merge and REST paths (which bypass Daft's planner, so we
show the source plan without a Sink node)
The field does not propagate through downstream operations (e.g.
`result.select(...)` creates a fresh DataFrame), which is the correct
behavior since the source plan is only meaningful on the direct write
result.
Closes #3827
---------
Co-authored-by: Srinivas Lade <srinulade1@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> test(iceberg): cover read_iceberg against a local catalog (#7357)
## Changes Made
`read_iceberg`'s read path has no unit-level coverage today:
- `tests/io/iceberg/` covers writes, expressions, `io_config`, and
metadata — but not reads.
- The read tests in `tests/integration/iceberg/` are gated behind
`--integration` and a docker-compose REST catalog, so they are excluded
by the default `addopts = "-m 'not (integration or benchmark or
hypothesis)'"` and don't run on ordinary PRs.
Net effect: snapshot/branch/tag resolution, per-snapshot schema
handling, and partition-column reconstruction are only exercised in a
suite most contributors never run locally.
Adds `tests/io/iceberg/test_iceberg_reads.py` using a temporary
SQLite-backed `SqlCatalog` (the same fixture pattern as
`test_iceberg_writes.py` and `test_ignore_corrupt_files.py`), covering:
- reads from a table object and from a metadata path, asserted equal
- empty table — zero rows with the schema preserved, not an error
- `snapshot_id` / `branch` / `tag` all resolving to the same historical
snapshot, parametrized, plus the default latest-snapshot read
- selector error paths: mutually exclusive combinations, unknown ref
names, and passing a tag name as `branch` (and vice versa), which
reports the actual ref type — that last case had no test anywhere
- schema evolution: a snapshot predating an added column reads with the
schema it was written under (`["x"]`, not `["x", "y"]`)
- projection / filter / limit / count pushdowns
- identity-partition column reconstruction: Iceberg strips these columns
from the data files, so `select("part")` has to rebuild them from the
manifest (the area of #2129)
No production code changes.
## Testing
- `DAFT_RUNNER=native pytest tests/io/iceberg/`: 197 passed, 21 skipped
(15 new).
- `DAFT_RUNNER=ray pytest tests/io/iceberg/test_iceberg_reads.py`: 15
passed.
- Skips cleanly without pyiceberg via the existing `importorskip`
pattern.
## Related Issues
None. The partition-projection test touches the area of #2129 but does
not fix it — that case passes today for identity transforms and the test
locks the behavior in. Latest Branches
0%
desmond/explain-all-sinks 0%
0%
© 2026 CodSpeed Technology