exstream

Learn · Pipeline graphs

Branch and observe

Use a reliable branch when every record must arrive. Use an observer when the main pipeline must never wait for it.

Reliable fork

const source = exstream(records)
const database = source.fork()
const auditFile = source.fork()

await Promise.all([database.pipeTo(databaseWriter), auditFile.jsonlStringify().pipeTo(auditWriter)])
Open “Branch and observe” in the playground

Every fork() participates in backpressure. The shared source advances only when all reliable branches can make progress. A slow audit destination may therefore slow the database branch too.

That behavior is correct when both outputs are required.

Non-blocking observer

const metrics = source.observe({
  bufferLimit: 100,
  overflow: 'drop-oldest',
})

An observer does not slow the reliable flow. Because it can fall behind, it must have a buffer limit and overflow policy. Use it for metrics, sampling, and diagnostics—not required data.

Failure boundary

A failed destination cancels its own fork. Reliable sibling branches can continue. The code that owns all terminal promises decides whether one failed branch should also abort the others.

Decide in words first

For every branch, write one sentence:

  • “Every record must reach this destination.” Use fork().
  • “Missing observations are acceptable.” Use observe() with a bound.

If neither sentence is true, the delivery contract is still undefined.