Latest Results
chore: centralize development and CI verbs in just (#4372)
* chore: extract just recipes from hatch replacement proposal
Extracts the Justfile from 98dcde719834bcd6fc010ffc4b07be5ae02fc2e4 (zarr-developers/zarr-python#4096). Environment and CI migration changes are adapted separately.
Assisted-by: Codex:GPT-6
* chore: use just for python helpers
Extracts the Justfile changes from 693f694e50b9ae4cd7c6b0d93238b9ee9f8f6865 (zarr-developers/zarr-python#4096).
Assisted-by: Codex:GPT-6
* chore: centralize development and CI verbs in just
Keep Hatch environment definitions while moving task commands into Justfile. Route root CI, Read the Docs, and contributor workflows through the same recipes, preserve quoted arguments, and ship recipes and helpers in the sdist.
Assisted-by: Codex:GPT-6
* chore: delegate package commands to their justfiles
Expose package recipes from the root while retaining each package’s command definitions and working directory.
Assisted-by: Codex:GPT-6
* fix(ci): harden the just migration's environment, pinning and release path
Review follow-ups to the Justfile centralization:
- `just gpu` read HATCH_ENV, so an exported CPU test environment silently
redirected `pytest -m gpu` into an env built without the gpu feature. It now
reads GPU_HATCH_ENV, which nothing else sets.
- Pin `uvx prek` to 0.5.3 and restore hook-environment caching in the lint
workflow, so lint no longer floats on whatever prek PyPI serves that day.
- Restore the mypy hook's `uv run --frozen mypy` entry: routing it through
`just typecheck` made just a prerequisite for committing in every existing
clone, and expanded to the identical command.
- Run the changelog filename check with `uv run --no-project python` instead of
building the `dev` hatch env (test + remote-tests + docs + mypy) for a script
that imports only sys and pathlib.
- Extend `just just-check` over packages/*/justfile, which the root delegation
recipes depend on and the formatting gate did not cover.
- Set up Python in the codspeed job before pip-installing into it, matching the
other workflows; that runner image was never given one.
- Revert the release and nightly-wheel jobs to `hatch build`, so the publishing
path does not fetch rust-just to run a pure alias.
- Renumber the changelog fragment to this pull request and credit #4096 in the
body, so the rendered note does not link to an unrelated upstream PR.
Assisted-by: ClaudeCode:claude-opus-5
* fix(ci): install prek persistently for git hooks; run setup steps in one just process
`just hooks-install` ran `uvx prek install`, and prek writes a hook shim that
hard-codes the binary it was installed from, falling back to `prek` on PATH.
Under uvx that path is an entry in uv's archive cache, so the first
`uv cache prune` (or a bump of the pinned version) broke every commit with
`prek: not found`, and nothing put `prek` on PATH for the fallback. Install it
as a pinned `uv tool` instead: a changed pin upgrades in place, a repeated run
is a no-op, and the shim points at a path that stays.
`setup` re-invoked `just list-env`, which dropped a `just hatch_env=... setup`
override on the floor because the child process re-read HATCH_ENV. Run it as a
subsequent dependency so both steps see the same value.
Also retire the last two comments naming the removed hatch scripts.
Assisted-by: ClaudeCode:claude-fable-5-1
* chore: renumber the changelog fragment to the upstream pull request
towncrier renders the fragment name as a link into
zarr-developers/zarr-python, so the fork's PR number would have pointed the
released note at an unrelated upstream pull request. #4372 is the PR that
merges this work.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* ci: install just as a binary in the benchmark job
Adding `actions/setup-python` to this job so that `pip install rust-just` had
an interpreter also changed which interpreter hatch built
`test.py3.12-minimal` on: the runner's own Python was replaced by
/opt/hostedtoolcache/Python/3.12.5/arm64. CodSpeed measured the result as a
12.97% degradation across 26 benchmarks, spread -12.76% to -13.99% -- a range
of 1.2 points over slice indexing, sharded Morton indexing, three array shapes
and two store types. Nothing in this branch touches src/, tests/benchmarks/ or
packages/; that uniformity is an interpreter swap, not a regression, and
CodSpeed flagged it as "different runtime environments detected".
Install just with extractions/setup-just instead, matching the three package
workflows. No interpreter is set up, so hatch resolves the same Python the
baseline used, and `just-version` is pinned to satisfy zizmor's unpinned-tools
audit.
The benchmarks need a re-run to produce a comparison against a matching
environment; the numbers on the previous run should not be acknowledged as a
regression.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* ci: install just from official release binaries; lock the docs toolchain
Follow-ups from an adversarial audit of this branch.
Supply chain. `rust-just` is a third-party repackaging of just on PyPI
(Repository: github.com/gnpaone/rust-just) with no attestation on any of its
17 wheels, and this branch had introduced it in ten places including the
self-hosted GPU runner and Read the Docs. All 23 just installs across the repo
now come from casey/just's own release binaries via
extractions/setup-crate@7577c1bd (v2.0.1), a node action whose SHA pins every
line of executed code -- rather than extractions/setup-just, a composite
wrapper that astral-sh/python-build-standalone#771 moved off for that reason.
This also retires the 12 pre-existing setup-just call sites, so the repo has
one mechanism instead of four. setup-crate does not verify checksums
(extractions/setup-just#20 is open); Read the Docs, where no action is
available, fetches the release tarball and verifies it against the SHA256SUMS
casey/just publishes.
Lockfile. docs.yml had stopped resolving from uv.lock: `uv sync --group docs`
became `hatch run docs:`, and hatch never reads the lockfile, so the entire
docs toolchain floated -- including numcodecs[msgpack], which carries no
version constraint. Dependabot's uv ecosystem exists to keep that lock fresh
and no job exercised it any more. The eight docs and changelog recipes now run
`uv run --frozen --group docs`, so single-version tooling comes from the lock
(as mypy already did) while hatch keeps the test environments, which exist per
interpreter and per dependency set and cannot live in one lockfile. The hatch
docs environment is removed as unused, and neither docs.yml nor Read the Docs
installs hatch at all now.
Argument forwarding. The three package justfiles splatted `{{ args }}`
unquoted, so `just zarr-metadata test -k 'a and b'` reached pytest as `-k a`
plus two stray paths, and command substitution in an argument executed. They
now use `set positional-arguments` and `"$@"` like the root, which is what the
contributing guide already claimed.
Also, regressions from my own earlier commits on this branch: `just lint` had
dropped `--show-diff-on-failure --color=always`, which j178/prek-action passed
by default, so a hook that rewrote a file failed with no diff; a four-line
comment above `hooks-install` displaced its `just --list` description, since
just reads only the last comment line; the prek cache key did not cover
prek_version and had a restore-keys fallback that could reuse a store built by
a different prek; `just just-check` ran before `just lint`, so a cosmetic
formatting nit hid every real lint result; and the `packages/*/justfile` glob
lacked nullglob, which breaks in the sdist, where /Justfile ships but
/packages deliberately does not.
Docs: the install instructions used `pip install`, which fails on an
externally-managed interpreter; `just just-check` was an undocumented required
check; two sentences still told contributors to activate an environment that
no longer exists; and the changelog fragment did not mention that the Hatch
script tables were removed. Adds a .gitattributes rule so a Windows checkout
does not get a CRLF Justfile that `just --fmt --check` rejects on every line.
Assisted-by: ClaudeCode:claude-opus-5
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Max Jones <14077947+maxrjones@users.noreply.github.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com> feat(zarr-indexing): factor chunk plans into a columnar GridPartition (#4310)
* feat(zarr-indexing): factor chunk plans into a columnar GridPartition
Restricting a transform to a chunk box distributes over output dimensions
whenever each output map reads its own input axis, which is every basic
and orthogonal selection. Chunk resolution therefore no longer intersects
the whole transform with every candidate chunk; it resolves each axis once
against its grid into a table (StridedSet / IndexedSet), sorts correlated
(vindex) index arrays into chunks once into a JointSet, and derives each
ChunkProjection as one row of each table. ChunkPlan.partition() and
partition_transform() expose the factored form, so a consumer can read
the tables directly instead of materializing an object graph per chunk.
The projections a plan yields are unchanged; the general whole-transform
walk remains for hand-built diagonals, which have no factored form.
Along the way: _intersect_general reuses a precomputed _CorrelatedBlock and
accepts survivor positions; checked_affine has identity and dtype-bounded
fast paths; ArrayMap._with_affine shares frozen index arrays on translate;
IndexDomain._unchecked / IndexTransform._unchecked skip validation for
objects derived from an already-valid transform.
Assisted-by: ClaudeCode:claude-fable-5-1
* docs(zarr-indexing): changelog fragment for the grid partition
Assisted-by: ClaudeCode:claude-fable-5-1
* docs(zarr-indexing): say why the suite runs from the repo root
The package and its tests import nothing from zarr; the old comments claimed
the chunk-resolution tests needed zarr's ChunkGrid, which stopped being true
once the package grew its own grids. The real reason is the shared pinned
test toolchain.
Assisted-by: ClaudeCode:claude-fable-5-1
* docs(zarr-indexing): document the grid partition and retire the per-chunk narrative
The module docstring described intersecting the whole transform with every
candidate chunk as "the algorithm"; that walk is now the fallback for
hand-built diagonals only. It now explains the factored form and its three
tables, and why they cost the sum of the touched chunks per axis.
The visual guide gains a final integrator section, "A plan is a product of
per-axis tables", with an executable snippet that reads the StridedSet,
IndexedSet and JointSet tables off real plans and checks the plan's
projections against the partition's rows. Integration boundaries gains
"Reading the tables directly", a consumer that assembles a strided box from
the tables with no projection materialized. The API index, landing page and
design notes (TensorStore lineage, the performance caveat, and the box/query
split) point at the new section.
Assisted-by: ClaudeCode:claude-fable-5-1
* refactor(zarr-indexing): one mechanism for chunk plans, and the review fixes
Adversarial review (roborev, a correctness reviewer, a complexity reviewer,
and ~24k differential examples against main) of the grid partition.
Cuts. The whole-transform walk that remained for hand-built diagonals is
gone: it was unreachable for every index-array shape, its key builder was
duplicated verbatim in _chunk_keys, and for the one shape it served it
produced wrong projections (a three-point diagonal yielded four projections
covering six cells, on main too). A DimensionMap diagonal is now rejected
with ValueError. With it go the sorted-1-D fast path, the three cell-transform
helpers, the block/positions parameters of _intersect_general, the
correlated-residual check that admitted a diagonal and then crashed,
GridPartition.__getitem__, partition_transform as public API, the
object-dtype column fallback (StridedSet.origin is now a position along the
request axis, so every column is intp), checked_affine's dtype-bound
shortcut (measured at noise; the identity shortcut stays and now accepts
bool via np.can_cast, as main did), and StridedSet.chunk_map/cell_map.
Fixes. GridPartition.n_rows is an exact integer and len raises OverflowError
instead of wrapping to zero; table columns are read-only, so a memoized
partition cannot drift under a consumer; the documented table consumer now
handles reversed axes, inserted axes and transposed transforms, and the
snippet checks all three.
Docs. Corrected the diagonal statement everywhere it appeared, the memoized
"fresh walk" wording, the "vectorized per axis" claim, and the TensorStore
correspondence (its strided sets are per input dimension; it keeps one index
array set per connected component). The guide no longer restates the class
docstrings.
Assisted-by: ClaudeCode:claude-fable-5-1
* fix(zarr-indexing): keep exact request-axis extents in strided tables
A zero-stride DimensionMap over a domain wider than np.intp is valid and
touches one storage cell; coercing every StridedSet column to intp made it
raise OverflowError where main returned one projection. `extent` and
`origin` are the two columns measured along the request axis, whose bounds
are arbitrary Python ints, so they now fall back to exact-int (object)
columns when a value does not fit. Chunk-local columns stay intp.
Also corrects the design note that said both affine-diagonal cases raise
NotImplementedError: two slice maps sharing an axis now raise ValueError.
Assisted-by: ClaudeCode:claude-fable-5-1
* test(zarr-indexing): partition oracle on clipped rectilinear grids and the minimal grid protocol
Every varying grid in the partition cases summed exactly to its extent, so
the boundary where a chunk's data extent is shorter than its declared size,
the rectilinear-specific case, was unpinned; so was a grid without
data_size. Both now run through the evaluation oracle for strided,
orthogonal and correlated selections.
Assisted-by: ClaudeCode:claude-fable-5-1
* Rename 316.feature.md to 4310.feature.md
* perf(indexing): preserve diagonal plans and reduce planning allocations
Assisted-by: Codex:GPT-6
* perf(indexing): partition independent index-array components
Assisted-by: Codex:GPT-6
* perf(indexing): streamline single-component projection walks
Assisted-by: Codex:GPT-6
* feat(indexing): prototype direct selector execution
Assisted-by: Codex:GPT-6
* refactor(indexing): prepare shared execution plans with explicit policies
Assisted-by: Codex:GPT-6
* docs(indexing): trace selections through chunk planning and execution
Assisted-by: Codex:GPT-6
* refactor(indexing): separate execution experiment from columnar plans
Retain public selection-flow documentation and fix singleton data-extent coverage. Move the execution prototype to a follow-up review.
Assisted-by: Codex:GPT-6
* fix(zarr-indexing): validate correlated bounds on any grid, own table columns, one planning mechanism
Adversarial review of the branch (roborev, a correctness reviewer, a
complexity reviewer, ~40k differential examples against main and NumPy).
Fixes. Correlated planning probed storage bounds only through the grid's
vectorized lookup, which zarr's grids do not validate, so an out-of-range
coordinate silently planned a chunk that does not exist; the joint table now
probes each component's extreme coordinates with the scalar lookup, as the
orthogonal table already did. Table columns were read-only by flag only,
which setflags(write=True) undoes; they are now re-homed over immutable
bytes, like ArrayMap's index array, so a memoized partition cannot drift.
Cuts, each with what it cost stated in the review: the affine-diagonal
iteration path (a second mechanism that reintroduced transform.intersect
per chunk for a shape no selection produces; diagonals now raise ValueError
from iteration as from partition()), the single-component iterator (14-23%
on a walk that stays 2-3x behind zarr's coordinate indexer either way),
chunk_coord_batches with its beyond-intp mixed-radix path and n_rows (no
consumer; a partition with row_shape writes the batching in three lines),
and the stored block_coordinates column (byte-identical to positions on
every 1-D block; now a memoized property). Tests that pinned memoization
identity and column flags collapse into one that asserts immutability.
Docs: the diagonal statement is now precise everywhere it appears,
StridedSet.full no longer claims "in order", the guide no longer says sets
holds every source axis, and the changelog describes the feature as shipped
rather than the branch's history.
Assisted-by: ClaudeCode:claude-fable-5-1
* fix(zarr-indexing): bounds-check unsigned index arrays before narrowing to intp
A uint64 value beyond the intp range wrapped to a negative index on the
cast and then passed as a wrapped position, selecting the last element
instead of raising. Unsigned values are never negative, so they are
checked as they are and narrowed only once in range.
Assisted-by: ClaudeCode:claude-fable-5-1
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Latest Branches
+1%
d-v-b:codex/justfile-verbs 0%
d-v-b:claude/zarr-python-issue-4174-77043e 0%
d-v-b:fix/codec-chain-validation © 2026 CodSpeed Technology