Latest Results
perf(core): merge streamed `AIMessageChunk`s once when accumulating runnable streams (#41697)
Related: #40826. Replaces #41468.
---
Adding two `AIMessageChunk`s re-parses all the tool call arguments
received so far. Runnable streams add each chunk to the running total
with `+`, so streaming a large tool call through a chain is quadratic in
the length of the arguments:
```python
chain = prompt | model.bind_tools([write_file])
for chunk in chain.stream(...): # slows down as the tool call's args grow
...
```
## Change
- Runnable streams now buffer leading `AIMessageChunk`s and merge them
in one `first + rest` addition when the total is read. This is the merge
`BaseChatModel.stream` already uses for its own output.
- Covers stream tracing (`_transform_stream_with_config`, used by every
runnable), the default `transform`/`atransform`, and `RunnableLambda`
input and output.
- The merged result is the same as adding the chunks one at a time. If
the single addition raises `TypeError`, the chunks are added one at a
time as before.
- `AIMessageChunk` is unchanged. #41468 deferred the parse inside the
message instead, but a chunk serialized under an `AIMessage`-typed field
then lost its `tool_calls`.
```mermaid
flowchart LR
S[stream chunks] --> B{AIMessageChunk and nothing merged yet?}
B -- yes --> L[buffer]
B -- no --> F[merge buffer, then add chunk with +]
L --> R[total read]
F --> R
R --> M["first + rest: one merge, one parse"]
```
| `chain.stream()`, 91 KB tool args in 20-char chunks | Before | After |
| --- | ---: | ---: |
| Wall time | 17.35 s | 0.07 s |
Not covered here:
- Code that accumulates chunks itself (`acc += chunk`). Calling
`add_ai_message_chunks(first, *rest)` once avoids the cost.
- Messages nested in other chunks, such as the dicts streamed by
`RunnablePassthrough.assign` and `RunnableParallel`.
- `astream_log`, and the accumulators in `RunnableWithFallbacks`,
`RunnableBranch` and `RunnablePassthrough`.
The accumulator is from the first version of #41502 by Elior Nataf
Lackritz.
---------
Co-authored-by: Elior Nataf Lackritz <elior.nataflackritz@langchain.dev> Latest Branches
0%
hunter/tool-exception-artifact -1%
byhow/langchain/nested-tool-executor -16%
mdrxy/langchain/summarize-with-call-model © 2026 CodSpeed Technology