Eventual-Inc/Daft

Daft

Performance History

Latest Results

refactor: move download import to module level in _blob.py Address review feedback: the deferred import had no import-cycle reason. Import from daft.functions.url directly.

hello-tang-rui:read-blob

52 minutes ago

fix: use floor division semantics for pre-epoch timestamps Address review feedback: unit downcasts previously truncated toward zero, so unix_seconds(-1.5s) returned -1 instead of -2. Spark uses floorDiv for these functions, so convert via explicit floor division on the physical epoch values instead of a unit cast. Adds regression tests for negative fractional epochs.

hello-tang-rui:feat/unix-epoch-functions

57 minutes ago

Merge remote-tracking branch 'origin/main' into codex/mcap-native-reader # Conflicts: # src/daft-scan/src/python.rs

codex/mcap-native-reader

2 hours ago

feat: add unix_seconds, unix_millis, unix_micros temporal functions Implements 3 temporal functions for PySpark parity (issue #3798): - unix_seconds(timestamp): seconds since 1970-01-01 00:00:00 UTC - unix_millis(timestamp): milliseconds since epoch - unix_micros(timestamp): microseconds since epoch All three return Int64, preserve timezone-aware inputs correctly, and are exposed as Rust UDFs, SQL functions, and Python API functions with tests.

hello-tang-rui:feat/unix-epoch-functions

3 hours ago

refactor: migrate IcebergScanOperator to DataSource (#7311) ## Changes Made * Migrated `IcebergScanOperator(ScanOperator)` to `IcebergDataSource(DataSource)`: `to_scan_tasks()` (sync iterator of `ScanTask`) becomes `async def get_tasks()` (async generator of `DataSourceTask`); `display_name()` / `schema()` / `partitioning_keys()` map to the corresponding `DataSource` interface * Data files are read via `DataSourceTask.parquet()`, extended with an optional `iceberg_delete_files` parameter (default `None`) so positional delete files still reach the native Parquet reader for merge-on-read tables * Kept the metadata-based count pushdown: added an optional `DataSource.supports_count_pushdown()` (default `False`, so other sources are unaffected) and forwarded it in `PyDataSourceWrapper` so the existing optimizer rule still applies; count results are produced by a small `_IcebergCountTask` * Partition pruning now happens inside `get_tasks` using `pushdowns.partition_filters`, since the DataSource path no longer goes through `ScanTask.catalog_scan_task` * Updated the SQL scan path (`scan_builder.rs`) to construct `IcebergDataSource` via `ScanOperatorHandle::from_data_source`, same pattern as `delta_scan` * Updated existing unit/integration tests and connector docs to the new API; behavior is intended to be a 1-to-1 migration with no user-facing changes Tested locally: Iceberg unit tests, and end-to-end reads against a local catalog (regular reads, partition-pruned reads, count pushdown, and the positional delete file path via a hand-built delete file, since PyIceberg 0.11.1 falls back to copy-on-write for deletes). ## Related Issues Closes Eventual-Inc/Daft#7296

main

4 hours ago

refactor(io): migrate Hudi scans to DataSource (#7324) ## Changes Made - Replace `HudiScanOperator` with `HudiDataSource`, migrating the Hudi reader from the older `ScanOperator` interface to the `DataSource` interface. - Route `read_hudi` through the DataSource bridge via `ScanOperatorHandle.from_data_source`, keeping the manual builder path so the `checkpoint` parameter continues to work. - Move partition pruning from the Rust-side `catalog_scan_task` into `HudiDataSource.get_tasks`, following the DataSource model where pruning is the source's responsibility. - Preserve existing behavior 1-to-1: full-scan warning, limit-based file pruning, per-file column min/max statistics, and partition values keyed by `_hoodie_partition_path`. ### Validation - `DAFT_RUNNER=native pytest tests/io/hudi/` passes, and `count_rows`/limit/select/filter give identical results to the previous implementation on the same test tables. ## Related Issues Closes #7295

main

4 hours ago

test(io): prove MCAP predicate pushdown prunes remote I/O Adds end-to-end coverage that .where() filters on topic/log_time reach the summary index (bounded range requests on a remote file), that unfiltered reads of physically out-of-order files return log-time order, and that predicate and kwargs constraints produce identical results. Documents the ordering contract and pushdown behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

codex/mcap-native-reader

5 hours ago

fix: truncate default IOConfig fields in explain output Sparse/diff-from-default display for IOConfig backends so df.explain() no longer dumps every S3/Azure/GCS/... default. When scan URIs have a known scheme, only that backend section is shown. Closes #7308 Co-authored-by: Cursor <cursoragent@cursor.com>

shreynath:fix/truncate-ioconfig-explain-7308

6 hours ago

Latest Branches

feat: add read_blob function for reading files as raw bytes#7318

1 hour ago

deed35c

hello-tang-rui:read-blob

feat: add unix_seconds, unix_millis, unix_micros temporal functions#7328

1 hour ago

3aa7ec4

hello-tang-rui:feat/unix-epoch-functions

feat(io): add native MCAP reader#7319

3 hours ago

d7318eb

codex/mcap-native-reader

Home Terms Privacy Docs