Latest Results
fix(io): enforce timeouts on OpenDAL clients (#7523)
## Changes Made
- Layer OpenDAL's `TimeoutLayer` on the **read path** of every operator
built by `OpenDALSource::get_client`, using OpenDAL's defaults: 60s per
control operation (`timeout`) and 10s between body reads (`io_timeout`).
Without it, the OpenDAL backends (`oss`, `cos`, `obs`, `tos`, `goosefs`,
`hdfs`, ...) run without any deadline end to end, and a silently dropped
connection (e.g. an LB reaping idle connections without a RST) stalls
reads forever — see the linked issue for the full evidence chain.
- `io_timeout` bounds the gap between body reads, not the total duration
of a read, so large or slow transfers are unaffected; only genuinely
stalled IO is interrupted.
- **Writes and deletes are deliberately left unbounded** (unchanged
behavior): opendal's `io_timeout` wraps the *whole* body of a write
call, so any finite default would also cap how long a single part upload
may take (5 MB in 10 s requires ≥4 Mbps sustained) and how long
`CompleteMultipartUpload` may run, breaking slow-but-legitimate uploads.
The bounded read operator and unbounded write operator share the same
underlying accessor; a size-aware write deadline could be a follow-up.
- The deadlines can be overridden per backend through `timeout_ms` /
`io_timeout_ms` keys in the OpenDAL backend config map
(`IOConfig(opendal_backends={"tos": {"io_timeout_ms": "30000", ...}})`).
Values are milliseconds in `1..=86_400_000`; malformed or out-of-range
values (including `0`) fail client creation with the offending key named
in the error. A `log::warn!` is emitted when a bare `timeout` /
`io_timeout` key is present, since those look like OpenDAL's own
TimeoutLayer knobs but are silently ignored by the services.
- Enabled the `layers-timeout` opendal feature; added `services-http` as
a dev dependency for the stalled-response test.
- Tests:
- `test_opendal_stalled_read_times_out`: a loopback HTTP server that
returns response headers and never sends the body verifies that a
stalled read fails within the configured deadline instead of hanging
(verified locally that with an effectively-infinite deadline the same
read hangs forever, i.e. the test cannot pass vacuously). An outer 30s
`tokio::time::timeout` keeps this a fast red test rather than a hung CI
job if the layering ever regresses. Skipped with a printed notice when
`http_proxy`/`all_proxy` (either case) is set, since the proxy would
answer for the loopback endpoint; CI has no proxy configured.
- `test_opendal_memory_multipart_write_roundtrip`: multipart writer path
(`put_part` ×2 + `complete` + read-back), which previously had no test
coverage in this module.
- `test_opendal_invalid_timeout_config_fails_client_creation`: malformed
and out-of-range values fail client creation with the offending key
named.
- The existing memory-backend tests keep passing with the layer applied.
## Related Issues
Fixes #7522
Co-authored-by: wangzheyan <wangzheyan@bytedance.com> fix(io): enforce read timeouts on OpenDAL clients
OpenDAL's default operator stack carries no deadlines and its shared
HTTP client never times out, so reads through OpenDAL-backed sources
(oss, cos, obs, tos, goosefs, hdfs) hang forever when a connection is
silently dropped, e.g. a load balancer reaping idle connections
without a RST.
Layer OpenDAL's TimeoutLayer (60s per control operation, 10s between
body reads) on the read path of every operator built by
OpenDALSource, overridable per backend via the timeout_ms /
io_timeout_ms config keys (milliseconds, 1..=86400000; malformed or
out-of-range values fail client creation).
Writes and deletes keep their previous unbounded behavior: io_timeout
bounds the whole body of a write call, so any finite default would
also cap how long a part upload may take and break slow-but-legitimate
uploads.
Fixes #7522everySympathy:fix/opendal-timeout Latest Branches
0%
+1%
hqlalala:hequ/fix-limit-repartition-upstream 0%
FANNG1:deprecate-write-lance-merge © 2026 CodSpeed Technology