Avatar for the Eventual-Inc user
Eventual-Inc
Daft
BlogDocsChangelog

Performance History

Latest Results

perf(optimizer): eliminate sorts preceded by another sort Add an `EliminateRedundantSort` logical optimization rule that drops a `Sort` whose ordering is fully overwritten by a later `Sort`, since the outer sort re-establishes a total ordering from scratch. The rule looks through row-wise, order-insensitive operators (`Project`, `Filter`) that sit between the two sorts, and collapses arbitrarily long sort chains in a single pass. It intentionally stops at order-sensitive operators such as `Limit`, `Offset`, `TopN`, `Sample`, and `MonotonicallyIncreasingId`, where the inner ordering is observable and must be preserved. Closes #4226
hello-peter-tang:eliminate-redundant-sort
5 hours ago
Merge branch 'main' into feat/iceberg-overwrite-filter
atovk:feat/iceberg-overwrite-filter
5 hours ago
Merge branch 'main' into fix/iceberg-partition-evolution-row-filter
atovk:fix/iceberg-partition-evolution-row-filter
5 hours ago
fix(omnisharing): make depth_frames() independent of episode order CI failed on Linux while passing locally on macOS, and greptile flagged the same root cause: _check_depth_bounds() probed frame counts from episodes.limit(1), but which episode that lands on depends on file discovery order. With a mixed-length release the identical call either succeeded or raised IndexError depending on platform. Reproduced directly: two episodes of 8 and 3 frames, requesting [0, 5]. Long-first probes the 8-frame episode and succeeds; short-first probes the 3-frame one and raises. Same data, same arguments, different outcome. The premise that the first episode represents the release is simply wrong, so the pre-flight check is removed. Bounds are now decided per episode: each reads the frames it has and reports them in depth_frame_indices, which already existed for exactly this purpose. Negative indices are still rejected up front, since those are a caller error rather than a property of the data. This trades a clear up-front error for a per-episode result, so strict=True is added for callers who would rather fail. Its limitation is documented rather than hidden: the check runs inside the read, so the failure surfaces as a Daft execution error, and when every episode is short Daft reports its own concat error instead of the message naming the camera. The docstring recommends the default plus inspecting depth_frame_indices. Also addresses the P2 review note: 13 function-local `import h5py` and one `import re` are hoisted to module scope. h5py is imported after importorskip so a missing extra still skips the module rather than failing collection; av stays function-local in the datagen module for the same reason, now with a comment saying why. Verified by running: the order-dependence repro now gives identical results for all three orderings, and two new parametrized tests pin that invariant so the regression cannot return. 222 tests pass (up from 218), 13 doctests pass, ruff is clean and pre-commit is idempotent across consecutive runs.
XuQianJin-Stars:feat/omnisharing-dataset
6 hours ago
feat(omnisharing): add PX OmniSharing dataset reader PX OmniSharing is PaXini's omnimodal embodied-AI dataset. Unlike most manipulation datasets it records force and tactile sensing alongside vision: each episode captures a human wearing an instrumented exoskeleton glove, with 15 tactile pads per hand, a dozen synchronized RGB cameras, stereo RGBD cameras, hand proprioception, optional object poses, audio and a language instruction. Adds thirteen functions for the DF-2 HDF5 stage, streaming only the datasets asked for so the 0.4-3.2 GB episodes are never downloaded whole: - raw() catalogs episodes from filenames alone, with no file reads - describe() dumps the object tree, since layout varies between releases - episode_metadata() lifts task labels, the instruction and camera inventory - trajectory(), tactile() and audio() return per-episode tensors - objects() handles episodes with differing object counts - cameras(), camera_payloads(), camera_frames() and depth_frames() cover vision - stereo_extrinsics() parses the stereo calibration - frames() expands to one row per frame with correct alignment DF-3 is LeRobot v2.1 and is already served by daft.datasets.lerobot, so it is documented rather than reimplemented. ## Structure probed before implementing The published layout is wrong or silent in several places, so a real 440 MB episode was probed over hf:// range reads before any code was written. The findings are recorded in the fixture's module docstring: - camera payloads are described only as "1D compressed payload". RGB_Camera* is actually H.265/HEVC in Annex-B format, which has no container and needs an explicit format hint to decode; RGBD_* sub-streams are Matroska. - audio is described as a "compressed audio stream (includes text)". It is neither: raw float64 PCM, with the instruction in a txt attr. - an undocumented meta group holds the task labels. - aligned_depth exists on RGBD_0 but appears in no published layout, and carries no attributes at all, so its depth unit cannot be derived from the file. The docstring says so rather than assuming millimetres. ## Four traps this reader handles 1. episode_index is not unique. It restarts per capture group, so part_01 alone holds two different episodes numbered 1217. raw() emits a composite episode_key; joining on episode_index would silently cross-match. 2. handpose quaternions are qw-first, [x, y, z, qw, qx, qy, qz], the reverse of scipy's convention. HANDPOSE_ORDER is asserted in the suite so a drift to [qx, qy, qz, qw] cannot pass silently. 3. Cameras do not share the observation clock. RGB cameras run faster, produce more frames (207 vs 208-254), start at a different moment, and carry non-contiguous ids. RGBD cameras additionally keep a clock per eye (left_timestamp, right_timestamp). frames(align_cameras=...) therefore matches by nearest timestamp and reports the residual as timestamp_delta_us. 4. action leads observation by one frame: action[i] is the state at i+1 with the final action repeated. Reading the arrays as stored misaligns state and action, which would poison any policy trained on them. frames() applies the offset, and its fields argument is mandatory rather than defaulted because a single tactile row is 3465 floats wide. Nothing is hardcoded to DF-2 widths: DF-2R's 17 joints and 3750-wide tactile vector are read from the file and covered separately. ## Errors surfaced by verification, not by review Several defects were only caught by running things, and are worth recording because each would have been invisible to code reading: - An unnest=True UDF passed to with_column is silently dropped; the column count fell from 21 back to 10. - .h5 files were accepted by the filename regex but excluded by the glob. - An IndexError raised inside a UDF is swallowed by Daft, surfacing only as "Need at least 1 series to perform concat" with no mention of the camera or the bound. depth_frames() now validates bounds before execution. - Any UDF returning an all-null tensor column fails the same way, with or without unnest, while mixed null/non-null works. Absent audio and depth are therefore reported as empty arrays rather than nulls. - A zero-length aligned_depth raised while a missing one returned empty, even though both mean "nothing to read". Both now return empty. - trajectory()'s hand description was field-order dependent. - RGBD left/right have no extrinsics of their own; the fallback chain now reaches color/extrinsics. Two test assumptions were also wrong rather than the code: camera-alignment index skipping needs ~10 frames to manifest, so the 8-frame fixture test was split into an index test and a drift test; and the real episode has 14 cameras (11 RGB + 3 RGBD), not 15 as first miscounted. One premise was wrong outright: cameras() and frames() already resolved {stream}_timestamp with a timestamp fallback, so per-eye clocks worked from the start. The real gap was that the fixture gave all three RGBD clocks identical values, leaving the behaviour unverified rather than missing. ## Tests Three suites, split by what they need: - tests/datasets/test_omnisharing.py: 141 tests over a synthetic fixture, so CI never needs the 1.08 TB release. Decode tests run against real HEVC and Matroska streams that the fixture encodes locally with PyAV, keeping CC-BY-NC-SA data out of the repo. - tests/datasets/test_omnisharing_docs.py: 15 tests executing every snippet in the guide, so a renamed column cannot leave the documentation silently wrong. These check more than syntax: that the documented duplicate episode_index really is ambiguous, and that the np.roll recipe genuinely moves qw last. - tests/datasets/test_omnisharing_integration.py: 15 tests against the real release over hf://, marked integration so they stay out of the default run. The integration suite exists to catch values that are structurally valid but physically wrong, anchoring assertions to independently checkable quantities: - quaternion norms are exactly 1.000000 across all 207 frames, corroborating the qw-first layout rather than merely trusting the order attr - tactile widths sum to exactly 3465 across 15 pads, each slice matching its declared length - RGB decodes to 1200x1920x3 and RGBD to 720x1280x3, matching their intrinsics attrs, with per-frame std around 40 confirming real image content - audio()'s sample count matches episode_metadata's audio_samples exactly, cross-checking two independent code paths - depth comes back as (720, 1280) uint16, majority non-zero, a real depth map rather than padding - stereo_extrinsics() yields a ~59 mm baseline for RGBD_0, a physically plausible separation, which is strong evidence the matrix is read correctly rather than merely having the right shape - action[i] == raw action[i+1] holds for all 207 frames - observation frame 0 aligns to a non-zero RGB_Camera0 frame, because that camera started recording earlier; index-based alignment would offset every frame Two measurements are recorded rather than acted on. _open_for_scan() was written expecting a speedup from HDF5's 1 KiB scan buffer; walking all 20 camera streams over hf:// took 344s with it versus 362s with the 64 KiB default, so per-request latency dominates and its docstring says so. And the real left-eye residual is sub-frame, so this hardware is tightly synchronised and per-eye alignment is insurance rather than a correction; the fixture's deliberate 2 ms skew is what exercises the divergent path. Verified by running: 218 tests in tests/datasets, 13 doctests, and the integration suite against the real release. All Python pre-commit hooks pass including mypy, ruff and codespell. pyproject.toml is untouched: no new dependencies beyond the existing daft[hdf5] and daft[video] extras.
XuQianJin-Stars:feat/omnisharing-dataset
6 hours ago
perf: add dense integer argsort fast path
GENG-CHOVYYYY:perf/dense-integer-argsort
17 hours ago

Latest Branches

CodSpeed Performance Gauge
0%
perf(optimizer): eliminate sorts preceded by another sort#7438
5 hours ago
8b9552e
hello-peter-tang:eliminate-redundant-sort
CodSpeed Performance Gauge
0%
6 hours ago
a71a44a
atovk:feat/iceberg-overwrite-filter
CodSpeed Performance Gauge
0%
6 hours ago
16431e4
atovk:fix/iceberg-partition-evolution-row-filter
© 2026 CodSpeed Technology
Home Terms Privacy Docs