Eventual-Inc/Daft - CodSpeed

Daft

Blog Docs Changelog

Performance History

Latest Results

fix: align hash implementations with equality

3em0:fix/6954-hash-collisions

4 hours ago

refactor(parquet): rename arrowrs_v2 → reader, restore Error enum, trim comments - Rename module: `src/daft-parquet/src/arrowrs_v2/` → `src/daft-parquet/src/reader/` (single-file reader engine; sibling to `read.rs` orchestration). - Rename entry points: `parquet_stream_v2` → `stream_parquet`, `parquet_read_v2` → `read_parquet`, `parquet_stream_v2_from_source` → `stream_parquet_from_source`. - Restore the parquet `Error` enum to its original 17 variants (and the matching `From<Error> for DaftError` arm); the prune of 10 unused variants was out of scope for this PR. - Trim verbose / redundant comments across the reader module (~125 → ~30 comment lines), and drop "v2" references. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/parquet-perf

5 hours ago

fix(parquet): emit schema-bearing empty batch in bulk path only Two failures fixed: - CI: cargo-machete flagged unused `tokio-stream` in daft-parquet — removed. - CI: `test_empty_input_on_fresh_store` (iceberg) regressed because `process_rg_streaming` was emitting a schema-bearing empty batch for 0-row RGs, which made the iceberg writer think it had data and land a zero-row snapshot. Fix: `process_rg_streaming` now returns `stream::empty()` for 0-row RGs, and `parquet_stream_v2` returns `(schema, stream)`. `parquet_read_v2` (bulk) uses the schema to build a 0-row schema-bearing batch when the stream produces nothing, preserving `MicroPartition.read_parquet` schema semantics on empty parquet inputs. The streaming path emits no batch in that case, so iceberg sees nothing and skips the snapshot. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/parquet-perf

6 hours ago

euan/nearest-asof

6 hours ago

perf(parquet): rewrite reader with arrow-rs public decoder API Replaces the parquet2-based stack (arrowrs_reader, async_reader, read_planner) with a flat (row_group × column) parallel decoder built on arrow-rs's public low-level APIs (SerializedPageReader, PrimitiveArrayReader, make_byte_array_reader, …). Key changes: - ParquetSource enum dispatches Local vs Url at one entry point (parquet_stream_v2 / parquet_read_v2) - LocalChunkSource: positioned coalesced preads, no mmap, no whole-file Bytes - RemoteChunkSource: per-RG coalesced range GETs with adjacency merge (≤1MB gap, ≤16MB per run, split at chunk boundaries above 24MB), fetches run in background while decode setup proceeds - Two-phase predicate pushdown: decode pred cols in parallel across RGs, evaluate per-RG to get bool masks, reuse pred arrays during assembly - Chunked-pred path for queries whose projection is fully covered by the predicate — streams pred cols chunk-by-chunk for within-RG limit early stop and better cache locality - Iceberg field-id mapping applied BEFORE filtering active leaves by name - Nested type decode via FieldReaderBuilder walking parquet+arrow schemas in lockstep (mirrors arrow-rs's private complex::Visitor) Module layout under arrowrs_v2/: - mod.rs entry points + phase-1 dispatch - chunk_source.rs byte sources (Local/Remote) - field_reader.rs schema walk + per-column decode - rg_processor.rs per-RG streaming + chunked-pred processors - util.rs pure helpers (RowSelection arithmetic, projection) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

colin/parquet-perf

7 hours ago

ci: retrigger to land CodSpeed on a consistent runner

BABTUNA:perf/packed-symbol-groupby

19 hours ago

feat: ASOF join benchmarking scripts (#6940) ## Summary Adds a self-contained benchmarking suite for Daft's `join_asof` operation: - **`data_generation.py`** — generates reproducible left/right parquet datasets at three scales (`small`, `medium`, `large`) with clustered timestamps and Zipf-skewed entity distribution, written to `benchmarking/data/asof_join/` - **`benchmark.py`** — runs a single asof-join using Daft's native or Ray runner, wrapped in a memray memory tracker, and prints a JSON result with wall time and memray output path - **`deployment.yaml`** — Ray cluster config for AWS (1 `m7i.large` head + 4 `r7i.4xlarge` workers) for distributed runs against S3 ## How to run From inside `benchmarking/asof_join/`: **1. Generate data (one-time)** ```bash python data_generation.py --scale small # or --scale medium / --scale large / --all ``` **2. Run locally (native runner)** ```bash python benchmark.py --scale small # Output: asof_join_memray.bin + JSON result on stdout ``` **3. Inspect memory profile** ```bash memray flamegraph asof_join_memray.bin ``` **Run on a Ray cluster** > Before running: update `DATA_ROOT` to your S3 bucket and uncomment `daft.set_runner_ray()` in `benchmark.py`. Also update the S3 bucket and IAM settings in `deployment.yaml`. Spin up the cluster: ```bash ray up benchmarking/asof_join/deployment.yaml ``` Forward the dashboard in one terminal: ```bash ray dashboard benchmarking/asof_join/deployment.yaml ``` Submit the job in another (after updating `DATA_ROOT` and `daft.set_runner_ray()` in `benchmark.py`): ```bash ray job submit --address "http://localhost:8265" --working-dir benchmarking/asof_join -- python benchmark.py --scale small ``` Tear down when done: ```bash ray down benchmarking/asof_join/deployment.yaml ```

main

19 hours ago

feat(show): env defaults and auto alignment for preview output (#6856) ## Summary Implements 1 and 5 bullet points of issue by improving `.show()` preview defaults and adding meaningful `align="auto"` behavior. This PR adds environment-variable defaults for `.show()` formatting options and makes `align="auto"` behave as expected: numeric/decimal columns are right-aligned and non-numeric columns are left-aligned. Auto-alignment is opt-in via the new env var or an explicit `align="auto"` argument; the hard-coded default stays `"left"` so existing docstring/doctest output (~40 examples) remains stable. ## Why The issue asks for more formatting control in `.show()` output. This PR focuses on two practical pieces: - Configurable default preview formatting without changing callsites. - Correct automatic alignment for mixed numeric/text previews when the user opts in. ## Changes Made - Add `.show()` default resolution helper in `daft/dataframe/preview.py`: - `DAFT_SHOW_FORMAT` - `DAFT_SHOW_VERBOSE` - `DAFT_SHOW_MAX_WIDTH` - `DAFT_SHOW_ALIGN` - Wire default resolution into `DataFrame.show(...)` in `daft/dataframe/dataframe.py`. - Update Rust preview alignment logic in `src/daft-recordbatch/src/preview.rs`: - `PreviewAlign::Auto` now maps numeric/decimal dtypes to right alignment. - Non-numeric dtypes remain left-aligned. - Add focused tests: - Rust unit test for auto alignment behavior. - Python tests for env default resolution, sentinel `None` handling, explicit-arg precedence, and the unchanged `"left"` fallback default. - Python formatting test covering `align="auto"` behavior. ## Behavior - Existing explicit `.show(...)` arguments keep precedence. - When callers leave defaults, env vars can now control show formatting defaults. - `align="auto"` (whether set via env var or explicit argument) produces numeric-right / non-numeric-left alignment in preview rendering. - The hard-coded default alignment remains `"left"` to keep existing examples stable; users opt into the new behavior with `DAFT_SHOW_ALIGN=auto` or `df.show(align="auto")`. ## Test Plan - `python -m ruff check daft/dataframe/preview.py daft/dataframe/dataframe.py tests/dataframe/test_show.py` - `cargo test -p daft-recordbatch test_auto_aligns_numeric_right_and_non_numeric_left -- --nocapture` - `DAFT_RUNNER=native pytest -q tests/dataframe/test_show.py -k 'resolve_show_defaults'` ## Related Issues - Part of #4114 <img width="887" height="702" alt="image" src="https://github.com/user-attachments/assets/eb81e69b-17b5-4992-83a9-e4de2b5805e0" /> <img width="968" height="618" alt="image" src="https://github.com/user-attachments/assets/45a98587-0910-400c-bc5c-f2655f4a5bb9" /> <img width="962" height="703" alt="image" src="https://github.com/user-attachments/assets/c8d5fe48-c896-4eb4-8b31-86fc5fefa983" /> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

main

1 day ago

Latest Branches

0%

fix: align hash implementations with equality#6955

4 hours ago

d2d7d87

3em0:fix/6954-hash-collisions

0%

perf(parquet): rewrite reader with arrow-rs public decoder API#6952

6 hours ago

0cef944

colin/parquet-perf

0%

feat: nearest asof joins#6953

6 hours ago

c731788

euan/nearest-asof

© 2026 CodSpeed Technology

Home Terms Privacy Docs