Latest Results
feat: add Avro file format read/write support (#7009)
Add native Rust-based Apache Avro file format support, enabling
read_avro() and write_avro() on DataFrame.
## Changes Made
**New crate: src/daft-avro/**
- Based on arrow-avro (57.x) for both reading and writing, providing
Arrow-native OCF handling without an intermediate Avro library.
- Schema conversion: bidirectional arrow_type_to_daft_type() handling
all primitive types, nullables, lists, structs, and maps. Reader schema
is extracted directly from the OCF header.
- Reader: single-file OCF reader with column projection pushdown and
max_records limiting via ReaderBuilder.
- Writer: serialize RecordBatch to OCF via WriterBuilder with
compression support (null, deflate). Unsigned integer types
(UInt8/16/32/64) are automatically cast to signed equivalents
(Int32/Int64) with overflow validation for UInt64.
- Python bindings: PyO3 bindings exposing read_avro and
write_record_batch_to_avro.
**Writer integration (src/daft-writers/)**
- NativeAvroWriter integrated into the PhysicalWriterFactory pipeline
(same as Parquet/CSV/JSON).
- Hive-style partitioned writes with partition_values union.
- UUID-based file naming, create_dir_all for partition directories.
**Scan integration (src/daft-scan/, src/daft-local-execution/)**
- Avro scan task reader with async I/O, globbing/wildcard support,
column projection pushdown.
**Public API**
- daft.read_avro(): schema inference/hints, column projection, hive
partitioning, file path column, checkpoint support.
- DataFrame.write_avro(): append/overwrite, compression, partitioning,
success file.
**Tests**
- 15 Rust tests and 26 Python tests.
## Related Issues
Closes #6901
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Srinivas Lade <srinulade1@gmail.com> Latest Branches
0%
0%
0%
© 2026 CodSpeed Technology