Latest Results
fix(io): decode video with a single PyAV generator so .mkv/.webm yield frames (#7229)
Fixes #5172.
## What
`read_video_frames` returned an empty DataFrame for `.mkv`/`.webm` while
`.mp4` worked.
## Root cause
The decode loop recreated the PyAV decode generator on every iteration:
```python
while True:
frame = next(container.decode(stream)) # new generator each time
```
Recreating `container.decode(stream)` per frame is unsupported by PyAV —
it drops buffered frames and, depending on the container, raises
`EOFError` on the freshly-created generator, which the loop treated as a
clean end-of-stream. `.mp4` happened to survive this; `.mkv`/`.webm` did
not.
## How
Create the decode generator once, before the loop, and iterate it to
completion.
## Testing
- Verified locally by decoding generated `.mp4`/`.mkv`/`.webm` fixtures
with PyAV: the old per-iteration pattern lost frames, the
single-generator fix decodes the full set from all three containers.
- Updated the test mocks to model PyAV's real contract —
`container.decode(stream)` returns **one** generator iterated to
completion (the previous mocks returned a fresh one-frame iterator per
call, which only matched the buggy code path), and `EOFError` is now
raised while iterating rather than from the `decode()` call. Existing
behavior (graceful empty result on EOF) is preserved.
- Added `test_list_frames_uses_single_decode_generator` (all frames
returned, `decode()` called exactly once).
**Verification limits (honest):** I could not run the full
`read_video_frames` path end-to-end (needs a native maturin build), and
could not reproduce a *literally* empty result on synthetic fixtures —
the exact zero-frame symptom is codec/container specific — but the
per-iteration generator recreation is unambiguously incorrect PyAV usage
and the direct cause of container-dependent frame loss. I did **not**
add a hard error on zero decoded frames, to preserve the
graceful-empty-on-EOF behavior from #5343.
## AI disclosure
Produced with LLM assistance; I reviewed the change and verified it as
described above.
---------
Co-authored-by: Anay Garodia <ps3561@columbia.edu> feat(io): add HDFS support via OpenDAL services-hdfs (#7202)
## Changes Made
**`src/common/io-config/src/hdfs.rs`** — New `HdfsConfig`:
- Fields: `name_node`, `root`
- `to_opendal_config(url)`: explicit config > URL authority fallback
**`src/common/io-config/src/config.rs` / `python.rs` / `lib.rs`** —
IOConfig integration:
- `StorageConfig` HDFS variant, exposed via
`IOConfig(hdfs=HdfsConfig(...))`
**`src/daft-io/src/lib.rs`** + **`src/daft-io/src/opendal_source.rs`** —
HDFS scheme routing:
- `"hdfs"` → `services-hdfs` in `available_schemes()`
- `#[cfg(not(feature = "hdfs"))]` gate with `log::warn!` hint
- `get_size()`: add `is_dir()` guard (general fix for all OpenDAL
backends)
**`src/daft-io/Cargo.toml`** — Add `hdfs = ["opendal/services-hdfs"]`
feature.
**`Cargo.toml`** — Forward `hdfs` feature for `maturin develop
--features hdfs`.
**`tests/io/test_opendal.py`** — 6 tests: HDFS config, IOConfig wiring,
pickle, URL routing.
## Testing
### 1. Without hdfs feature
```bash
$ DAFT_RUNNER=native python -c "
import daft
daft.read_parquet('hdfs://localhost:9000/data.parquet').collect()
"
DaftError::External Failed to load Credentials for store: opendal(hdfs)
Details: "... Available OpenDAL schemes: [oss, cos, obs, tos, goosefs, memory, fs, github].
scheme: hdfs => scheme is not registered"
# "HDFS support is not compiled in. Rebuild with `--features hdfs` to enable it."
$ DAFT_RUNNER=native python -c "
import daft
daft.read_parquet('oss://bucket/data.parquet').collect()
"
# Normal OSS error (no hdfs warn output)
```
### 2. With hdfs feature (macOS ARM64, brew hadoop)
Build:
```bash
$ export HDFS_LIB_DIR=~/hadoop-3.3.3/lib/native
$ export JAVA_HOME=/Library/Java/JavaVirtualMachines/zulu-8.jdk/Contents/Home
$ export DYLD_LIBRARY_PATH="$JAVA_HOME/jre/lib/server:$HDFS_LIB_DIR"
$ export CLASSPATH=$(hadoop classpath --glob)
$ maturin develop --features "python,hdfs"
Compiling daft v0.3.0-dev0
Finished `dev` profile in 37.14s
🛠 Installed daft-0.3.0.dev0
```
### 4. End-to-end on running HDFS cluster
```python
>>> from daft.daft import HdfsConfig, IOConfig
>>> import daft
>>> # 1. read_parquet
>>> df = daft.read_parquet('hdfs://localhost:9000/user/test/test_hdfs.parquet')
>>> df.show()
Daft Query ID: mighty-flame-02ef94
╭───────┬────────╮
│ a ┆ b │
│ --- ┆ --- │
│ Int64 ┆ String │
╞═══════╪════════╡
│ 1 ┆ x │
├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 2 ┆ y │
├╌╌╌╌╌╌╌┼╌╌╌╌╌╌╌╌┤
│ 3 ┆ z │
╰───────┴────────╯
(Showing first 3 of 3 rows)
>>> # 2. filter + select
>>> df.where(df['a'] > 1).select('b').show()
Daft Query ID: fast-wren-444d99
╭────────╮
│ b │
│ --- │
│ String │
╞════════╡
│ y │
├╌╌╌╌╌╌╌╌┤
│ z │
╰────────╯
(Showing first 2 of 2 rows)
>>> # 3. write_parquet → read back
>>> df.write_parquet('hdfs://localhost:9000/tmp/test_daft_write/')
Daft Query ID: quick-raven-93273f
>>> daft.read_parquet('hdfs://localhost:9000/tmp/test_daft_write/').show()
Daft Query ID: cool-fox-3b7295
╭───────┬────────╮
│ a ┆ b │
│ --- ┆ --- │
│ Int64 ┆ String │
╞═══════╪════════╡
│ 1 ┆ x │
│ 2 ┆ y │
│ 3 ┆ z │
╰───────┴────────╯
(Showing first 3 of 3 rows)
```
## Related Issues
<!-- e.g., Closes #123 -->
Closes #2786 fix: Gravitino catalog S3 credential key mismatch and _has_table error handling (#7279)
## Changes Made
Two fixes for the Gravitino catalog integration that affect users
reading Iceberg tables stored on S3 via Gravitino.
**S3 credential dual-key format**
(`daft/catalog/__gravitino/_client.py`): Gravitino's Fileset Catalog and
Iceberg REST Catalog use different property key conventions for S3
credentials — hyphen format (`s3-access-key-id`) vs. dot format
(`s3.access-key-id`). The previous implementation only checked the
hyphen format, which meant Iceberg-on-S3 users got no credentials and
couldn't read data. Now checks both formats for all S3-related
properties (access key, secret key, endpoint, session token, region),
with hyphen taking precedence when both are present.
**`_has_table` error handling**
(`daft/catalog/__gravitino/_catalog.py`): The original `except
Exception: return False` swallowed network timeouts, authentication
failures, and actual "table not found" errors — all returning `False`
silently. The fix catches only the specific case (exact `Exception` type
with "not found" in the message), and re-raises everything else.
GravitinoClient's `load_table` wraps 404s in a plain `Exception` rather
than a typed error; a TODO note is left to switch to a typed
`TableNotFoundError` once the upstream Gravitino Python client supports
it.
## Testing
Added `tests/catalog/test_gravitino_client.py` covering: hyphen and dot
key formats, hyphen-precedence when both present, no-credentials returns
None, s3a scheme, _has_table returns False on not-found, and _has_table
re-raises ConnectionError and HTTPError. All existing `tests/catalog/`
tests pass.
## Related Issues
Closes #7200
---------
Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com> Latest Branches
0%
Lucas61000:issue-6496-plan-json-lineage 0%
satya323:satyendra/mongodb-data-source-support 0%
hfutatzhanghb:agent/opendal-hdfs-community © 2026 CodSpeed Technology