Eventual-Inc/Daft

Daft

Performance History

Latest Results

fix(io): support writing to hdfs:// URLs that include a port parse_object_url split the object key at Position::AfterHost, which sits before the authority port. For a port-bearing authority (idiomatically HDFS, e.g. hdfs://namenode:9000), the port leaked into the key and the native writer produced an invalid path like namenode/:9000/..., rejected by HDFS as an invalid DFS filename. Keep the port with the bucket and split the key at Position::BeforePath. Port-less URLs (S3/GCS/Azure/OSS) are unaffected: with no port, AfterHost and BeforePath resolve to the same offset. Add unit tests for parse_object_url and build_object_path covering both port-less (S3) and port-bearing (HDFS) authorities. Both run on CI without requiring HDFS or a JVM.

Lucas61000:fix-hdfs-write-port

8 hours ago

fix(flotilla): enable in-place restarts for RaySwordfishActor to avoid node-level compute loss RaySwordfishActor was created with Ray's default max_restarts=0, so any actor crash (e.g. OOM kill) was terminal: in-flight calls raised ActorDiedError, the dispatcher evicted the whole node from the worker pool, and the node's compute stayed lost until a later worker-refresh cycle re-created an actor on it. Meanwhile all requeued tasks piled onto the remaining nodes, risking cascading OOMs. Set max_restarts (default 4, matching MAX_UDFACTOR_ACTOR_RESTARTS in ray_actor_pool_udf.py) on the swordfish actor options so Ray restarts a crashed actor on the same node. While restarting, in-flight calls surface ActorUnavailableError, which already maps to RayTaskResult.worker_unavailable() -> the task is requeued WITHOUT evicting the node. max_task_retries intentionally stays 0: task re-execution is owned by Daft's dispatcher. The value is overridable via DAFT_SWORDFISH_ACTOR_MAX_RESTARTS (-1 = infinite, 0 = restore old behavior). Upstream community context (Eventual-Inc/Daft): - PR #4628 (feat(flotilla): Fault tolerance) added dispatcher-level rescheduling on WorkerDied/WorkerUnavailable but never configured actor restarts, leaving the WorkerUnavailable fast path unreachable for crashed actors. - PR #7116 / issue #7104 bumped the ray floor to >=2.11.0 specifically for ActorUnavailableError; this change makes that signal actually fire for crashed-but-restartable actors. Tests: - start_ray_workers passes max_restarts>0 by default and honors the env override (tests/ray/test_flotilla_worker_startup.py) - env parsing fallback + RaySwordfishTaskHandle actor-error mapping contract (tests/ray/test_flotilla_actor_fault_tolerance.py)

DogerW666:test-bug-fix

9 hours ago

Merge branch 'main' into feat/rint-bround

cmirandavega:feat/rint-bround

19 hours ago

fix(sql): remove unused noqa comment to fix ruff RUF100 Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/doris-connectorx-fallback

1 day ago

fix(sql): add missing type annotations to fix mypy errors - Add Any, Engine imports for type annotations - Annotate __getstate__/__setstate__ with dict[str, Any] - Add return type Engine | None to _get_or_create_engine - Add type: ignore[import-untyped] for pymysql import - Add assert engine is not None guard for type narrowing Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/sql-engine-caching

1 day ago

perf(sql): cache SQLAlchemy engine with thread-safe double-checked locking Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/sql-engine-caching

1 day ago

fix(sql): fall back to SQLAlchemy on ConnectorX COM_STMT_PREPARE failure Signed-off-by: jiangxt2 <jiangxt2@vip.qq.com>

jiangxt2:fix/doris-connectorx-fallback

1 day ago

Merge remote-tracking branch 'origin/main' into fix/doris-connectorx-fallback # Conflicts: # daft/datasets/lerobot.py

jiangxt2:fix/doris-connectorx-fallback

1 day ago

Latest Branches

fix(io): support writing to hdfs:// URLs that include a port#7307

9 hours ago

1b99b7c

Lucas61000:fix-hdfs-write-port

fix(flotilla): enable in-place restarts for RaySwordfishActor to avoi…#7306

10 hours ago

5b03dec

DogerW666:test-bug-fix

feat: add rint and bround math functions#7222

20 hours ago

0011d5d

cmirandavega:feat/rint-bround

Home Terms Privacy Docs