Latest Results
docs(checkpoint): add user guide and connector cross-references (#6985)
## Changes Made
User-facing documentation for the checkpoint feature shipped in #6932
(`df.write_iceberg(checkpoint=...)`,
`df.write_deltalake(checkpoint=...)`, plus `CheckpointStore` /
`CheckpointConfig` / `IdempotentCommit`).
**New surfaces:**
- `docs/use-case/checkpointing.md` — canonical user guide. Covers the
two guarantees (always-on skip; conditional idempotent commit), mental
model (with a diagram), scope of guarantees, limitations, and
chronological recovery walkthrough.
- `docs/img/checkpointing.png` — mental-model diagram.
- `docs/connectors/iceberg.md` — new `## Checkpointing` section with
end-to-end example, marker inspection, sink-specific constraints,
idempotence-key contract, recovery semantics, and Iceberg-specific
notes.
- `docs/connectors/delta_lake.md` — mirror with Delta-specific details
(`custom_metadata` marker, `table.history()` inspection, fresh-table
creation, `CommitFailedError` retry).
- `docs/ai-functions/overview.md` — See Also link to the user guide.
- `daft/dataframe/dataframe.py` — `write_iceberg` / `write_deltalake`
docstrings forward-reference the user guide.
## Related Issues
Documents the feature added in #6932.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Everett Kleven <ekleven@vangelis.tech> feat(writers): add single_file kwarg to write_parquet (#7004)
## Changes Made
Add `single_file: bool = False` to `DataFrame.write_parquet`. When
`True`, the data is coalesced into one Parquet file written at the exact
path given, instead of a directory of UUID-named files.
```py
df.write_parquet("out.parquet", single_file=True)
```
**How it works.** The flag is threaded from the Python API through the
logical plan into the writer pipeline. A new non-rotating writer wrapper
bypasses the usual target-file-size rollover so all input partitions
stream into one writer. The filename helper uses the user's path
verbatim, skipping partition-dir and UUID generation. Parent directories
are auto-created. Row-group batching is preserved. For empty DataFrames,
the writer is skipped and an empty Parquet file is emitted via the
commit path at the exact destination.
**Modes.** Both `append` (default) and `overwrite` work — Parquet has no
append semantic, so each write truncates the target via `File::create`.
`overwrite`'s pre-clean is a safe no-op for single-file paths.
**Validation.** Raises a clear error when `single_file=True` is combined
with `partition_cols`, `write_mode="overwrite-partitions"`,
`write_success_file=True`, the Ray runner, or schemas that fall back to
the PyArrow writer.
**Scope.** Parquet on the native runner only. CSV/JSON, the PyArrow
fallback, and a shuffle-to-one for the Ray runner are left as follow-ups
— those paths currently error out instead of silently falling back to
UUID-dir behavior.
Tests cover the happy path, parent-dir auto-creation, large data that
would otherwise split across multiple files, empty DataFrames,
`overwrite` mode, and rejection of `partition_cols` /
`write_success_file`.
## Related Issues
Closes #2359 Latest Branches
0%
BABTUNA:feat/functions-try-arithmetic 0%
rohit/df-2049-checkpoint-user-guide 0%
© 2026 CodSpeed Technology