Latest Results
fix(omnisharing): make depth_frames() independent of episode order
CI failed on Linux while passing locally on macOS, and greptile flagged the same
root cause: _check_depth_bounds() probed frame counts from episodes.limit(1), but
which episode that lands on depends on file discovery order. With a mixed-length
release the identical call either succeeded or raised IndexError depending on
platform.
Reproduced directly: two episodes of 8 and 3 frames, requesting [0, 5].
Long-first probes the 8-frame episode and succeeds; short-first probes the
3-frame one and raises. Same data, same arguments, different outcome.
The premise that the first episode represents the release is simply wrong, so the
pre-flight check is removed. Bounds are now decided per episode: each reads the
frames it has and reports them in depth_frame_indices, which already existed for
exactly this purpose. Negative indices are still rejected up front, since those
are a caller error rather than a property of the data.
This trades a clear up-front error for a per-episode result, so strict=True is
added for callers who would rather fail. Its limitation is documented rather than
hidden: the check runs inside the read, so the failure surfaces as a Daft
execution error, and when every episode is short Daft reports its own concat
error instead of the message naming the camera. The docstring recommends the
default plus inspecting depth_frame_indices.
Also addresses the P2 review note: 13 function-local `import h5py` and one
`import re` are hoisted to module scope. h5py is imported after importorskip so a
missing extra still skips the module rather than failing collection; av stays
function-local in the datagen module for the same reason, now with a comment
saying why.
Verified by running: the order-dependence repro now gives identical results for
all three orderings, and two new parametrized tests pin that invariant so the
regression cannot return. 222 tests pass (up from 218), 13 doctests pass, ruff is
clean and pre-commit is idempotent across consecutive runs.XuQianJin-Stars:feat/omnisharing-dataset feat(omnisharing): add PX OmniSharing dataset reader
PX OmniSharing is PaXini's omnimodal embodied-AI dataset. Unlike most
manipulation datasets it records force and tactile sensing alongside vision: each
episode captures a human wearing an instrumented exoskeleton glove, with 15
tactile pads per hand, a dozen synchronized RGB cameras, stereo RGBD cameras,
hand proprioception, optional object poses, audio and a language instruction.
Adds thirteen functions for the DF-2 HDF5 stage, streaming only the datasets
asked for so the 0.4-3.2 GB episodes are never downloaded whole:
- raw() catalogs episodes from filenames alone, with no file reads
- describe() dumps the object tree, since layout varies between releases
- episode_metadata() lifts task labels, the instruction and camera inventory
- trajectory(), tactile() and audio() return per-episode tensors
- objects() handles episodes with differing object counts
- cameras(), camera_payloads(), camera_frames() and depth_frames() cover vision
- stereo_extrinsics() parses the stereo calibration
- frames() expands to one row per frame with correct alignment
DF-3 is LeRobot v2.1 and is already served by daft.datasets.lerobot, so it is
documented rather than reimplemented.
## Structure probed before implementing
The published layout is wrong or silent in several places, so a real 440 MB
episode was probed over hf:// range reads before any code was written. The
findings are recorded in the fixture's module docstring:
- camera payloads are described only as "1D compressed payload". RGB_Camera* is
actually H.265/HEVC in Annex-B format, which has no container and needs an
explicit format hint to decode; RGBD_* sub-streams are Matroska.
- audio is described as a "compressed audio stream (includes text)". It is
neither: raw float64 PCM, with the instruction in a txt attr.
- an undocumented meta group holds the task labels.
- aligned_depth exists on RGBD_0 but appears in no published layout, and carries
no attributes at all, so its depth unit cannot be derived from the file. The
docstring says so rather than assuming millimetres.
## Four traps this reader handles
1. episode_index is not unique. It restarts per capture group, so part_01 alone
holds two different episodes numbered 1217. raw() emits a composite
episode_key; joining on episode_index would silently cross-match.
2. handpose quaternions are qw-first, [x, y, z, qw, qx, qy, qz], the reverse of
scipy's convention. HANDPOSE_ORDER is asserted in the suite so a drift to
[qx, qy, qz, qw] cannot pass silently.
3. Cameras do not share the observation clock. RGB cameras run faster, produce
more frames (207 vs 208-254), start at a different moment, and carry
non-contiguous ids. RGBD cameras additionally keep a clock per eye
(left_timestamp, right_timestamp). frames(align_cameras=...) therefore
matches by nearest timestamp and reports the residual as timestamp_delta_us.
4. action leads observation by one frame: action[i] is the state at i+1 with the
final action repeated. Reading the arrays as stored misaligns state and
action, which would poison any policy trained on them. frames() applies the
offset, and its fields argument is mandatory rather than defaulted because a
single tactile row is 3465 floats wide.
Nothing is hardcoded to DF-2 widths: DF-2R's 17 joints and 3750-wide tactile
vector are read from the file and covered separately.
## Errors surfaced by verification, not by review
Several defects were only caught by running things, and are worth recording
because each would have been invisible to code reading:
- An unnest=True UDF passed to with_column is silently dropped; the column count
fell from 21 back to 10.
- .h5 files were accepted by the filename regex but excluded by the glob.
- An IndexError raised inside a UDF is swallowed by Daft, surfacing only as "Need
at least 1 series to perform concat" with no mention of the camera or the
bound. depth_frames() now validates bounds before execution.
- Any UDF returning an all-null tensor column fails the same way, with or
without unnest, while mixed null/non-null works. Absent audio and depth are
therefore reported as empty arrays rather than nulls.
- A zero-length aligned_depth raised while a missing one returned empty, even
though both mean "nothing to read". Both now return empty.
- trajectory()'s hand description was field-order dependent.
- RGBD left/right have no extrinsics of their own; the fallback chain now reaches
color/extrinsics.
Two test assumptions were also wrong rather than the code: camera-alignment index
skipping needs ~10 frames to manifest, so the 8-frame fixture test was split into
an index test and a drift test; and the real episode has 14 cameras (11 RGB +
3 RGBD), not 15 as first miscounted.
One premise was wrong outright: cameras() and frames() already resolved
{stream}_timestamp with a timestamp fallback, so per-eye clocks worked from the
start. The real gap was that the fixture gave all three RGBD clocks identical
values, leaving the behaviour unverified rather than missing.
## Tests
Three suites, split by what they need:
- tests/datasets/test_omnisharing.py: 141 tests over a synthetic fixture, so CI
never needs the 1.08 TB release. Decode tests run against real HEVC and
Matroska streams that the fixture encodes locally with PyAV, keeping
CC-BY-NC-SA data out of the repo.
- tests/datasets/test_omnisharing_docs.py: 15 tests executing every snippet in
the guide, so a renamed column cannot leave the documentation silently wrong.
These check more than syntax: that the documented duplicate episode_index
really is ambiguous, and that the np.roll recipe genuinely moves qw last.
- tests/datasets/test_omnisharing_integration.py: 15 tests against the real
release over hf://, marked integration so they stay out of the default run.
The integration suite exists to catch values that are structurally valid but
physically wrong, anchoring assertions to independently checkable quantities:
- quaternion norms are exactly 1.000000 across all 207 frames, corroborating the
qw-first layout rather than merely trusting the order attr
- tactile widths sum to exactly 3465 across 15 pads, each slice matching its
declared length
- RGB decodes to 1200x1920x3 and RGBD to 720x1280x3, matching their intrinsics
attrs, with per-frame std around 40 confirming real image content
- audio()'s sample count matches episode_metadata's audio_samples exactly,
cross-checking two independent code paths
- depth comes back as (720, 1280) uint16, majority non-zero, a real depth map
rather than padding
- stereo_extrinsics() yields a ~59 mm baseline for RGBD_0, a physically
plausible separation, which is strong evidence the matrix is read correctly
rather than merely having the right shape
- action[i] == raw action[i+1] holds for all 207 frames
- observation frame 0 aligns to a non-zero RGB_Camera0 frame, because that camera
started recording earlier; index-based alignment would offset every frame
Two measurements are recorded rather than acted on. _open_for_scan() was written
expecting a speedup from HDF5's 1 KiB scan buffer; walking all 20 camera streams
over hf:// took 344s with it versus 362s with the 64 KiB default, so per-request
latency dominates and its docstring says so. And the real left-eye residual is
sub-frame, so this hardware is tightly synchronised and per-eye alignment is
insurance rather than a correction; the fixture's deliberate 2 ms skew is what
exercises the divergent path.
Verified by running: 218 tests in tests/datasets, 13 doctests, and the
integration suite against the real release. All Python pre-commit hooks pass
including mypy, ruff and codespell. pyproject.toml is untouched: no new
dependencies beyond the existing daft[hdf5] and daft[video] extras.XuQianJin-Stars:feat/omnisharing-dataset Latest Branches
0%
hello-peter-tang:eliminate-redundant-sort 0%
atovk:feat/iceberg-overwrite-filter 0%
atovk:fix/iceberg-partition-evolution-row-filter © 2026 CodSpeed Technology