Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

fix: drop OnceLock cache for simpler per-call connector creation

BABTUNA:fix/s3-proxy-env-vars

2 hours ago

fix: address PR review feedback for temporal batch 3 - Fix make_timestamp timezone bug: interpret inputs as local time when a timezone is provided (use tz.from_local_datetime) - Make next_day day_of_week argument required (remove #[arg(optional)]) - Add non-UTC timezone test for make_timestamp - Use short-form day names in next_day test

BABTUNA:feat/temporal-expressions-batch-3

2 hours ago

ci: instrument integration-test-tpch for codecov Same pattern as the integration-test-ray change: swap the pre-built release wheel for an in-place maturin develop under -C instrument-coverage, then emit an lcov artifact that publish-coverage-reports merges into the codecov upload. TPC-H is the higher-yield target for daft-distributed coverage because the ray block runs the full query suite (joins, sorts, aggregations, shuffles) through FlotillaRunner, so it should actually exercise pipeline_node/* and plan/runner.rs. The ray-only integration test (autoscaling) moved coverage barely 1pp because it only tests Ray cluster lifecycle, not query execution. Timeout bumped 45 to 75 minutes to absorb the slower instrumented build and two test suites (native + ray) running back-to-back under coverage. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

desmond/coverage-integration-ray

3 hours ago

fix(scheduler): include dispatched tasks in autoscaling ratio (#6388) After the Ray autoscaler ramp-up rework in #6653, one bug from the original autoscaler underscaling investigation is still live. `DefaultScheduler::needs_autoscaling()` computes the ratio as `pending_tasks / total_capacity`. However, `schedule_tasks()` runs immediately before this check and drains pending tasks onto available workers, so the ratio only reflects residual demand. When the cluster is saturated and most demand has just been dispatched, the ratio collapses below the threshold and `try_autoscale()` is never called. To fix this, track `last_scheduled_count` in the scheduler and include it in the numerator so the ratio reflects total demand (pending + just-dispatched). Reset the counter inside `get_autoscaling_request()` to prevent double-counting when it is called multiple times between `schedule_tasks()` rounds. Also strips zero-valued GPU/memory keys from Ray resource bundles in `try_autoscale()` so the autoscaler doesn't interpret them as demand for zero-resource bundles on specialized nodes. **Scope changes from the original PR:** - Dropped the `try_autoscale()` available-vs-total capacity fix: the code it modified was replaced entirely by #6653's ramp-up algorithm. A residual off-by-one edge case still exists there but requires a different patch. - Dropped the `active_memory_bytes()` trait addition: it existed only to support the capacity fix above. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

main

3 hours ago

refactor(distributed): Remove node_origin_id self-ref in distributed (#6738) ## Changes Made Make NodeInfo.node_origin_id an Option<NodeID> so distributed plan nodes (which are themselves the origin) carry None instead of self-referencing via node.node_id(). Local-execution nodes still set Some(distributed_id) when attributing stats back to a distributed node.

main

4 hours ago

Update comments and log

chris/distributed-origin-node-id

4 hours ago

test(pycapsule): use PyArrow 15.0.0-compatible APIs CI runs unit tests against pyarrow==15.0.0 (matrix lower bound). Earlier tests used APIs only available in newer pyarrow: - pa.chunked_array(obj) as PyCapsule consumer (added post-15) - pa.record_batch(dict, schema=...) dict-form constructor Replaced with pa.RecordBatch.from_pydict / from_arrays and exercised requested_schema cast via pa.RecordBatchReader.from_stream(obj, schema=) which works across all supported pyarrow versions.

aaron-ang:arrow-pycapsule

5 hours ago

test(list_chunk): move groupby regression into tests/dataframe

veinkr-bot:fix-chunk-return-type

7 hours ago

Latest Branches

fix: respect proxy env vars (HTTP_PROXY, HTTPS_PROXY, etc.) in S3 client#6679

9 days ago

fd621d4

BABTUNA:fix/s3-proxy-env-vars

feat: implement batch 3 temporal functions (make_date, make_timestamp, make_timestamp_ltz, last_day, next_day)#6672

2 hours ago

62083d2

BABTUNA:feat/temporal-expressions-batch-3

ci: instrument integration-test-ray for codecov#6744

3 hours ago

ce8ae21

desmond/coverage-integration-ray

Home Terms Privacy Docs