Eventual-Inc
Daft
Blog
Docs
Changelog
Blog
Docs
Changelog
Overview
Branches
Benchmarks
Runs
Performance History
Latest Results
refactor: move download import to module level in _blob.py Address review feedback: the deferred import had no import-cycle reason. Import from daft.functions.url directly.
hello-tang-rui:read-blob
52 minutes ago
fix: use floor division semantics for pre-epoch timestamps Address review feedback: unit downcasts previously truncated toward zero, so unix_seconds(-1.5s) returned -1 instead of -2. Spark uses floorDiv for these functions, so convert via explicit floor division on the physical epoch values instead of a unit cast. Adds regression tests for negative fractional epochs.
hello-tang-rui:feat/unix-epoch-functions
57 minutes ago
Merge remote-tracking branch 'origin/main' into codex/mcap-native-reader # Conflicts: # src/daft-scan/src/python.rs
codex/mcap-native-reader
2 hours ago
feat: add unix_seconds, unix_millis, unix_micros temporal functions Implements 3 temporal functions for PySpark parity (issue #3798): - unix_seconds(timestamp): seconds since 1970-01-01 00:00:00 UTC - unix_millis(timestamp): milliseconds since epoch - unix_micros(timestamp): microseconds since epoch All three return Int64, preserve timezone-aware inputs correctly, and are exposed as Rust UDFs, SQL functions, and Python API functions with tests.
hello-tang-rui:feat/unix-epoch-functions
3 hours ago
refactor: migrate IcebergScanOperator to DataSource (#7311) ## Changes Made * Migrated `IcebergScanOperator(ScanOperator)` to `IcebergDataSource(DataSource)`: `to_scan_tasks()` (sync iterator of `ScanTask`) becomes `async def get_tasks()` (async generator of `DataSourceTask`); `display_name()` / `schema()` / `partitioning_keys()` map to the corresponding `DataSource` interface * Data files are read via `DataSourceTask.parquet()`, extended with an optional `iceberg_delete_files` parameter (default `None`) so positional delete files still reach the native Parquet reader for merge-on-read tables * Kept the metadata-based count pushdown: added an optional `DataSource.supports_count_pushdown()` (default `False`, so other sources are unaffected) and forwarded it in `PyDataSourceWrapper` so the existing optimizer rule still applies; count results are produced by a small `_IcebergCountTask` * Partition pruning now happens inside `get_tasks` using `pushdowns.partition_filters`, since the DataSource path no longer goes through `ScanTask.catalog_scan_task` * Updated the SQL scan path (`scan_builder.rs`) to construct `IcebergDataSource` via `ScanOperatorHandle::from_data_source`, same pattern as `delta_scan` * Updated existing unit/integration tests and connector docs to the new API; behavior is intended to be a 1-to-1 migration with no user-facing changes Tested locally: Iceberg unit tests, and end-to-end reads against a local catalog (regular reads, partition-pruned reads, count pushdown, and the positional delete file path via a hand-built delete file, since PyIceberg 0.11.1 falls back to copy-on-write for deletes). ## Related Issues Closes Eventual-Inc/Daft#7296
main
4 hours ago
refactor(io): migrate Hudi scans to DataSource (#7324) ## Changes Made - Replace `HudiScanOperator` with `HudiDataSource`, migrating the Hudi reader from the older `ScanOperator` interface to the `DataSource` interface. - Route `read_hudi` through the DataSource bridge via `ScanOperatorHandle.from_data_source`, keeping the manual builder path so the `checkpoint` parameter continues to work. - Move partition pruning from the Rust-side `catalog_scan_task` into `HudiDataSource.get_tasks`, following the DataSource model where pruning is the source's responsibility. - Preserve existing behavior 1-to-1: full-scan warning, limit-based file pruning, per-file column min/max statistics, and partition values keyed by `_hoodie_partition_path`. ### Validation - `DAFT_RUNNER=native pytest tests/io/hudi/` passes, and `count_rows`/limit/select/filter give identical results to the previous implementation on the same test tables. ## Related Issues Closes #7295
main
4 hours ago
test(io): prove MCAP predicate pushdown prunes remote I/O Adds end-to-end coverage that .where() filters on topic/log_time reach the summary index (bounded range requests on a remote file), that unfiltered reads of physically out-of-order files return log-time order, and that predicate and kwargs constraints produce identical results. Documents the ordering contract and pushdown behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
codex/mcap-native-reader
5 hours ago
fix: truncate default IOConfig fields in explain output Sparse/diff-from-default display for IOConfig backends so df.explain() no longer dumps every S3/Azure/GCS/... default. When scan URIs have a known scheme, only that backend section is shown. Closes #7308 Co-authored-by: Cursor <cursoragent@cursor.com>
shreynath:fix/truncate-ioconfig-explain-7308
6 hours ago
Latest Branches
CodSpeed Performance Gauge
0%
feat: add read_blob function for reading files as raw bytes
#7318
1 hour ago
deed35c
hello-tang-rui:read-blob
CodSpeed Performance Gauge
0%
feat: add unix_seconds, unix_millis, unix_micros temporal functions
#7328
1 hour ago
3aa7ec4
hello-tang-rui:feat/unix-epoch-functions
CodSpeed Performance Gauge
0%
feat(io): add native MCAP reader
#7319
3 hours ago
d7318eb
codex/mcap-native-reader
ยฉ 2026 CodSpeed Technology
Home
Terms
Privacy
Docs