Eventual-Inc/Daft

Eventual-Inc

Daft

Blog Docs Changelog

Performance History

Latest Results

Merge branch 'main' into daft-lance-checkpoint

everySympathy:daft-lance-checkpoint

8 hours ago

test(ai/transformers): broaden HF 429 detection and surface retries in logs Address review feedback on the HuggingFace Hub retry helper: * Recognize the "HTTP Error 429 thrown while requesting ..." message shape produced by huggingface_hub.HfHubHTTPError, in addition to the requests/urllib3 "Too Many Requests" / "rate limit" phrasing. The earlier fallback required both "429" AND one of those keywords, which silently missed the very error format observed in CI. * Emit a warnings.warn before each backoff sleep so CI logs show the attempt count and delay instead of appearing to hang. * Fix import ordering and switch Callable to collections.abc to satisfy ruff (I001, UP035), and apply ruff-format.

XuQianJin-Stars:fix/transformers-classifier-tests-hf-429

20 hours ago

chore(grouped-agg): clarify SHARD_THRESHOLD doc covers both strategies

BABTUNA:perf/sharded-partition-then-agg

1 day ago

Merge branch 'main' into feat/string-distance-functions

nish2292:feat/string-distance-functions

1 day ago

feat: add Spark-compatible string functions (translate, substring_index, soundex, ascii, chr, space)

XuQianJin-Stars:feat/add-string-functions-translate-substring-index-soundex-ascii-chr-space

1 day ago

test(ai/transformers): tolerate HuggingFace Hub HTTP 429 in classifier tests The integration tests for TransformersImageClassifier and TransformersTextClassifier download models from huggingface.co. Shared CI runner egress IPs are occasionally rate-limited, causing flaky failures (e.g. 'HTTP Error 429 thrown while requesting HEAD https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/config.json') that are unrelated to the code under test. Wrap each Hub-touching call with a small retry helper (_hf_retry.call_with_hf_retry) that retries on 429 with linear backoff and skips the test if the limit persists, while re-raising any other exception unchanged.

XuQianJin-Stars:fix/transformers-classifier-tests-hf-429

1 day ago

fix: resolve Python lint errors in test_crypto.py - Remove unused imports: hashlib, pytest - Sort imports alphabetically (ruff I001) - Add noqa: F841 for df variables used implicitly by daft.sql - Fix duplicate test_crc32_sql method (F811) by renaming to test_crc32_sql_known_value

XuQianJin-Stars:feature/hash-crypto-functions

1 day ago

refactor(checkpoint): drop "seal" vocabulary from Rust API surface (#7078) ## Why The user-facing checkpoint lifecycle is `Staged → Checkpointed → Committed`, and the method is `checkpoint(id)`. The Rust trait docs and one error variant used a parallel vocabulary — "seal" / "sealed" — that appears nowhere in the Python API or user guide. A user reading the Python docs and a user reading `cargo doc` (or hitting an `AlreadySealed` traceback) saw two different words for the same transition. ## Changes - `store.rs` — trait + method docs rewritten using the `Checkpointed` state name and `checkpoint(id)` method name - `error.rs` — `AlreadySealed` → `AlreadyCheckpointed` (variant + `Display` string + all call sites) - `scan.rs` — module docs + comments - `impls/s3.rs`, `test_utils.rs`, `types.rs`, `tests/s3_store.rs` — internal helpers/comments follow suit (`sealed_manifests` → `checkpointed_manifests`, etc.) Not touched: `SEALED_KEYS_COLUMN` — that's the canonical on-disk column name for sealed-keys parquet files; the value already shipped and renaming it would change the on-disk format. Rust API break is fine: checkpointing isn't publicly announced yet and there are no external consumers of the trait. Note: `S3CheckpointStore` keeps its name deliberately — team consensus is "S3" reads as "S3-compatible" (it works against any IOClient-backed store incl. `file://`). ## Verification - `grep -rn -i "seal" src/daft-checkpoint/` → only `SEALED_KEYS_COLUMN` - `cargo doc -p daft-checkpoint --no-deps --features s3` renders only `Staged`/`Checkpointed`/`Committed` vocabulary - `cargo test -p daft-checkpoint --features s3,test-utils` → 39 passed, 0 failed 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

main

1 day ago

Latest Branches

feat(checkpoint): support checkpoint on lance, stage Lance DataSink write results#7035

8 hours ago

dcd1e06

everySympathy:daft-lance-checkpoint

test(ai/transformers): tolerate HuggingFace Hub HTTP 429 in classifier tests#7080

21 hours ago

973ee85

XuQianJin-Stars:fix/transformers-classifier-tests-hf-429

feat(grouped-agg): shard PartitionThenAgg execution per morsel#7079

1 day ago

3ad1f42

BABTUNA:perf/sharded-partition-then-agg

Home Terms Privacy Docs