Branches performance
Pull requests
last run
3 hours ago
refactor(arrow2): misc deprecation warnings
4 hours ago
2ac9df2
universalmind303/misc-arrow2
0%
last run
4 hours ago
feat(swordfish): Streaming sources (#5978)
This PR enables sources to be streaming sources in swordfish. Currently,
all our sources (in memory / scan task / glob) require a fixed set of
inputs, and this PR refactors them such that they can continuously
accept inputs. This works by
1. Separate inputs from the plan. LocalPhysicalPlans as well as
Swordfish Pipeline nodes no longer store the inputs inline. Translation
from logical -> physical now produces a separate `Inputs` struct, and
the swordfish pipeline nodes now store a receiver, e.g.
`Receiver<(InputID, ScanTask)>`, that the sources pull from when they
are started.
2. The `NativeExecutor` holds the senders into the new streaming
sources, and is responsible for enqueuing the inputs.
FOR NOW, the pipelines themselves are still short lived,
`NativeExecutor.run` accepts both plan and inputs, and immediately
starts the plan and enqueues the inputs and thats it. In the next PR, I
will implement the ability to have the pipelines be long lived with
continuous input enqueuing. (this will require the input_id in the
morsels as well as flushing mechanism).
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
5 hours ago
5686bb5
colin/flight-shuffle-streaming
0%
last run
3 hours ago
ci: remove temporary restore-mtime verification job
Both Linux and macOS passed the smoke test. Remove the temporary job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 hours ago
bf183b0
ci/fix-restore-mtime-exit-code
0%
last run
4 hours ago
minor nits
4 hours ago
5c49897
universalmind303/remove-fromarrow2
0%