Eventual-Inc/Daft

Daft

Performance History

Latest Results

feat: expand Source/Sink JSON serialization for lineage tracing Add source_type, file_paths, glob_paths, sink_type, root_dir, file_format to repr_json output so that df.lineage() can extract input/output metadata from the logical plan.

Lucas61000:issue-6496

3 hours ago

fix: change decimals to Option<i32> for negative scale support and clean up Float16 tests

cmirandavega:feat/rint-bround

6 hours ago

fix: sort imports in _mcap.py to satisfy ruff I001 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

everettVT/mcap-extra-check

7 hours ago

fix: normalize HF bucket tree paths (#7218) ## Summary Normalize `hf://buckets` paths copied from Hugging Face `/tree/` pages before globbing, and harden bucket listing/globbing/errors based on live-API behavior. Fixes #7217. - Strip the reserved `tree` segment from copied browser URLs (`hf://buckets/org/repo/tree/{path}`, including the bucket-root `.../tree`), and canonicalize bucket glob paths before globbing. - Fix bucket listings: the buckets tree API returns entries without the git `oid` field, so every non-empty bucket `ls` failed to deserialize — breaking all wildcard globs on buckets. `Item.oid` is now optional. - Follow `Link: rel="next"` pagination on tree listings (1000 entries/page); previously large-bucket globs silently truncated. Also benefits dataset listings. - Actionable error for bucket 401s: Hugging Face returns 401 both for private buckets and buckets that don't exist — including when a user swaps `datasets`→`buckets` in a URI (datasets are **not** automatically exposed as buckets). Previously this surfaced as a bare "check your credentials" error; it now explains buckets vs datasets and points at `hf://datasets/...`. - Skip the datasets-only parquet-conversion API for buckets (previously produced the same misleading 401 on bucket-root reads). ## Tests - `cargo test -p daft-io huggingface` — parse-level tests for naive copied-URL patterns (file/dir/root tree URLs, globs under `tree/`, tree-like object keys left untouched, canonicalization idempotence) plus live-API tests for bucket `ls`, tree-URL wildcard glob, and the dataset-as-bucket error hint. - End-to-end against a local build: - original `/tree/` bucket parquet MRE from #7217 (350 rows) - wildcard glob through a copied tree URL (previously broken) - recursive `**/*.parquet` glob from bucket root - `hf://buckets/wikimedia/wikipedia/...` (dataset-as-bucket) surfaces the buckets-vs-datasets hint

main

7 hours ago

docs: de-vibe comments

AnayGarodia:fix/5539-oss-storage-options

7 hours ago

docs: de-vibe comments

AnayGarodia:fix/5172-video-frames-mkv-webm

7 hours ago

style: apply rustfmt

everettVT/hf-bucket-errors

7 hours ago

fix(io/av): decode video stream with a single generator read_video_frames recreated the PyAV decode generator on every loop iteration (next(container.decode(stream))). Recreating the generator per frame is unsupported by PyAV: it drops buffered frames and, for some containers, raises EOFError on the recreated generator immediately, which the loop treated as end-of-stream and silently returned an empty DataFrame. mp4 happened to survive this, but .mkv/.webm did not. Decode the stream with a single generator instead, iterating it to completion. Update the video test mocks to model PyAV's real contract (decode() returns one generator) and add a regression test asserting the stream is decoded with a single decode() call. Fixes #5172

AnayGarodia:fix/5172-video-frames-mkv-webm

8 hours ago

Latest Branches

feat: add df.lineage() introspection API with OpenLineage SDK integration#7231

4 hours ago

967eaab

Lucas61000:issue-6496

feat: add rint and bround math functions#7222

1 day ago

cbb0640

cmirandavega:feat/rint-bround

feat: add mcap pip extra and dependency checks for read_mcap#7220

14 hours ago

f17c14c

everettVT/mcap-extra-check

Home Terms Privacy Docs