Branches performance
Pull requests
last run
14 hours ago
add empty
14 hours ago
f9af1f4
colin/remove-empty-scan
0%
last run
2 days ago
refactor(arrow2): misc deprecation warnings
2 days ago
2ac9df2
universalmind303/misc-arrow2
0%
last run
2 days ago
feat(swordfish): Streaming sources (#5978)
This PR enables sources to be streaming sources in swordfish. Currently,
all our sources (in memory / scan task / glob) require a fixed set of
inputs, and this PR refactors them such that they can continuously
accept inputs. This works by
1. Separate inputs from the plan. LocalPhysicalPlans as well as
Swordfish Pipeline nodes no longer store the inputs inline. Translation
from logical -> physical now produces a separate `Inputs` struct, and
the swordfish pipeline nodes now store a receiver, e.g.
`Receiver<(InputID, ScanTask)>`, that the sources pull from when they
are started.
2. The `NativeExecutor` holds the senders into the new streaming
sources, and is responsible for enqueuing the inputs.
FOR NOW, the pipelines themselves are still short lived,
`NativeExecutor.run` accepts both plan and inputs, and immediately
starts the plan and enqueues the inputs and thats it. In the next PR, I
will implement the ability to have the pipelines be long lived with
continuous input enqueuing. (this will require the input_id in the
morsels as well as flushing mechanism).
<!-- Link to related GitHub issues, e.g., "Closes #123" -->
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2 days ago
5686bb5
colin/flight-shuffle-streaming
0%
ci: remove temporary restore-mtime verification job
Both Linux and macOS passed the smoke test. Remove the temporary job.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2 days ago
bf183b0
ci/fix-restore-mtime-exit-code
0%