Skip to content

Research references

This catalog records the 73 references supplied on October 17, 2026, with their proposed uses for fpstreams. Topics include lazy pipelines, logical and physical plans, mixed Python/Rust/Arrow/NumPy execution, spilling groups and joins, bounded async execution, and semantic parity.

Titles, categories, and links below follow the supplied list. Inclusion does not mean that a source has been verified, reviewed, implemented, or benchmarked. Extension marks a proposed capability that requires new functionality. It does not mean that the capability is available or scheduled. Review notes below distinguish source review from implemented and verified changes.

Papers, 1–25

No. Reference Proposed use
1 Stream Fusion: From Lists to Streams to Nothing at All Fuse map, filter, zip, and fold while reducing intermediate objects.
2 Stream Fusion, to Completeness Study fusion of nested streams, zip, and early termination.
3 Complete Fusion for Stateful Streams Establish verifiable transformation rules for stateful stream fusion.
4 MonetDB/X100: Hyper-Pipelining Query Execution Measure cache-friendly batch sizes and amortize per-row dispatch.
5 Efficiently Compiling Efficient Query Plans for Modern Hardware Compare data-centric code generation and pipeline fusion boundaries.
6 Vectorization vs. Compilation in Query Execution Decide which operators suit batch vectorization or compiled fusion.
7 Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask Calibrate mixed execution against computation and memory-access patterns.
8 Weld: A Common Runtime for High Performance Data Analytics Reduce data movement across Python, NumPy, and Arrow.
9 Tuplex: Data Science in Python at Native Code Speed Study common-type native paths and exceptional Python fallback.
10 Volcano—An Extensible and Parallel Query Evaluation System Compare iterator execution, operator lifecycles, and parallel exchange boundaries.
11 The Cascades Framework for Query Optimization Separate optimization rules, cost estimates, and physical strategy selection.
12 Morsel-Driven Parallelism Balance fine-grained batch scheduling, NUMA, and skew.
13 Eddies: Continuously Adaptive Query Processing Explore runtime adaptation only for pure operators that can safely be reordered.
14 How Good Are Query Optimizers, Really? Test join selection with cardinality errors and skewed data.
15 Robust External Hash Aggregation in the Solid State Age Improve spill layout and performance under memory pressure.
16 Simple, Efficient, and Robust Hash Tables for Join Processing Study join-specific hash tables, negative-probe filtering, and skew.
17 To Partition, or Not to Partition, That is the Join Question in a Real System Measure partitioning costs across join input sizes.
18 Global Hash Tables Strike Back! Compare shared and partitioned aggregation under memory pressure and skew.
19 In-Order Sliding-Window Aggregation in Worst-Case Constant Time Evaluate DABA to avoid repeating window computations.
20 The Dataflow Model Extension. Define event-time windows, watermarks, late data, and result revisions.
21 Naiad: A Timely Dataflow System Extension. Study progress tracking and cyclic incremental dataflow.
22 Umbra: A Disk-Based System with In-Memory Performance Compare buffer management across memory and disk execution.
23 Spark SQL: Relational Data Processing in Spark Study a shared optimizer for relational expressions and functional pipelines.
24 Velox: Meta’s Unified Execution Engine Separate reusable kernels, connectors, and resource management.
25 Adaptive Execution of Compiled Queries Study switching between interpreted and compiled execution for short queries.

Open-source projects and specifications, 26–63

No. Reference Proposed use
26 DuckDB Compare vectorized operators, external sorting, and external hash aggregation.
27 Polars Compare streaming plans, expression optimization, and columnar grouping.
28 Apache DataFusion Study Rust execution plans, optimizer rules, and shared spill budgets.
29 Apache Arrow Acero Compare batch dataflow, backpressure, and Arrow operator composition.
30 Velox Study encoded vectors, lazy materialization, and executor memory management.
31 Substrait, an open specification Explore serializable and inspectable relational plans across backends.
32 Ibis Compare compilation boundaries from one expression API to multiple backends.
33 Narwhals Improve DataFrame adaptation, static typing, and optional dependency isolation.
34 Daft Compare pipelines combining Python UDFs and Rust columnar execution.
35 Dask Study task fusion, chunk granularity, and scheduler overhead.
36 Ray Data Compare batch mapping, concurrency budgets, and producer/consumer backpressure.
37 Vaex Study memory mapping and virtual columns to reduce array copies.
38 PyO3 Check batch FFI, object borrowing, and GIL release boundaries.
39 rust-numpy Check shared NumPy buffer lifetimes, strides, and writability.
40 NumExpr Compare chunked expression evaluation and temporary array costs.
41 Numba Compare numeric specialization, compilation caches, and parallel reduction.
42 cytoolz Compare functional grouping, joins, and reduction internals.
43 more-itertools Check lazy windows, chunking, grouping, and iterator boundaries.
44 aiostream Compare ordered concurrent maps, task limits, and safe closing.
45 asyncstdlib Compare async iterator borrowing, scoped closing, and exception propagation.
46 ReactiveX for Python / RxPY Compare debounce, switch, merge, and time-operator semantics.
47 AnyIO Study task groups, cancellation scopes, and bounded object streams.
48 Trio Compare structured concurrency, cancellation, and task cleanup.
49 Streamz Compare branching dataflow, feedback, and slow-consumer backpressure.
50 Apache Flink Extension. Study windows, state backends, and backpressure diagnostics.
51 Apache Beam Extension. Study composable Combine, window triggers, and batch/stream semantics.
52 Differential Dataflow Extension. Explore incremental joins, grouping, and indexed intermediate state.
53 Strymonas, a research implementation Inspect generated loops for complex stream combinations without intermediate objects.
54 hashbrown Compare SIMD probing and compact hash layouts for native grouping.
55 Rayon Explore native parallel iteration, reduction, and thread-pool reuse.
56 pyperf Improve process warmup, environment capture, and regression comparison.
57 Hypothesis Generate boundary inputs for Python/native semantic equivalence checks.
58 IBM Sliding Window Aggregators Compare DABA and related window algorithms and benchmarks.
59 ConnectorX Study database-to-Arrow batch reads with fewer copies.
60 Apache Arrow nanoarrow Compare Arrow C Data/C Stream interoperability and lifetimes.
61 Apache DataSketches Evaluate optional bounded-state approximate cardinality and quantile collectors.
62 orjson Compare Rust/Python JSON encoding, decoding, and output types.
63 simdjson Explore batched JSONL parsing and field access on demand.

Commercial engines and services, 64–73

These entries are references to public documentation. They do not establish access to proprietary implementations or authorize use of a hosted service.

No. Reference Proposed use
64 Databricks Photon, a proprietary engine Compare documented vectorization coverage, fallback reasons, and diagnostics.
65 Tableau Hyper, a proprietary engine Compare batch inserts, type conversion, and embedded analytics interfaces.
66 Snowflake / Snowpark, a commercial backend with open-source clients Compare lazy pushdown, UDF costs, and local/backend switching.
67 MotherDuck, a commercial service based on DuckDB Study hybrid local/remote plans and input transfer costs.
68 Coiled, a commercial service based on Dask Compare separate compute, transfer, and spill times in reports.
69 Anyscale, a commercial service based on Ray Compare backpressure, memory-failure diagnostics, and batch task status.
70 Polars Cloud, a commercial service with an open-source Polars core Compare stage graphs and CPU, I/O, and shuffle diagnostics.
71 SingleStore, a proprietary engine Study interpretation before compilation and parameterized plan caching.
72 Actian Analytics Engine, formerly Vector, a proprietary engine Compare cache-friendly vectors, column compression, and block pruning.
73 CedarDB, a commercial engine Study tiered compilation, buffer management, and specialized hash joins.

Additional reference: StreamEx

StreamEx extends Java's Stream API. The initial comparison used its cheatsheet and selected implementations in the streamex-0.8.4 tag. This supplements the original 73 entries without renumbering them. Selected code was read in StreamEx and MoreCollectors. No upstream code was executed or copied into fpstreams. The 0.9.0 release notes were also checked. They add nullness annotations and null-terminated helpers, and report a fix for filtering entries with null keys or values. The source comparison below remains pinned to 0.8.4.

StreamEx feature fpstreams comparison and next question
pairMap, mapFirst, mapLast, scanLeft Existing pairwise().map(...), map_first, map_last, and scan cover related uses. Check callback order and allocation costs before adding shortcuts.
groupRuns, collapse, runLengths group_runs(key) groups equal adjacent keys; collapse(predicate, merger) supports pair predicates. These have different contracts. The 2.4.0 Collector overload reduces runs without a raw run buffer.
collapse(predicate, collector) Existing Rows.group_by_sorted(...).aggregate(...) covers ascending supported keys. The 2.4.0 collapse overload covers arbitrary adjacent predicates and async pipelines through the same Collector state program. Memory depends on the Collector's state; a run needs a boundary or source exhaustion to emit.
MoreCollectors.reducingWithZero Compare with the existing Collector.done protocol. An absorbing value requires an explicit user contract; ordinary reducers cannot safely infer early termination.
headTail Study iterator ownership, replay, and cleanup before exposing a live tail. Existing unfold, map_accum, and Gatherer already cover many stateful transformations.

For example, scanLeft(seed, accumulator) is a terminal operation that includes the seed in its result list. fpstreams scan(initial, function) is lazy and emits only states produced from input items. Copying the Java name would obscure that difference.

These are API comparisons, not a claim of semantic equivalence or measured performance. The 2.4.0 Collector overload preserves the existing binary merger form without adding a new method name. Java parallel streams do not establish fpstreams support for distributed execution, event-time windows, or watermarks.

Review notes

The roadmap records earlier comparisons and experiments separately. The October 18 follow-up below records the completed review scope. The six previously queued repositories received scoped, read-only analysis; experiments and implementation remain deferred.

October 17: prefix counts and iterator boundaries

Initial reading of reference 43's iterator recipes led to a check of fpstreams prefix operations. The Python path rejected floats only during execution, while the async path accepted them: take(2.5) emitted three items and take(float("nan")) emitted the entire finite input.

The 2.4.0 fix normalizes take, limit, drop, and skip counts once through the integer index protocol when constructing the operation. Both APIs accept integer-index objects and reject floats before opening a source. Integer behavior, zero-count boundaries, iterator cleanup, and existing execution paths remain subject to regression checks. No upstream implementation was copied or executed.

Reference 45's iterator-scoping guide distinguishes owning an iterator from borrowing it for several operations. fpstreams already closes owned async iterators on early termination. This comparison preserves that default; a borrowing API remains a separate candidate. The initially attempted iterator API URL was unavailable, so the guide was reached through the documentation index instead.

Versioned paper follow-up

These were the initial checks on October 17. Full-paper reading was completed in the October 18 follow-up below.

Metadata and abstracts were checked for references 2, 3, 18, and 19. Their supplied titles match the papers, including abbreviated titles for 3 and 18. These checks are not full-paper reviews.

The streaming model and scope sections of reference 3 describe an acyclic, single-consumer compilation model with declared state. Its allocation guarantee depends on individual operators avoiding calls and allocations. For fpstreams, our inference is to evaluate fusion within proven expression paths first, while preserving observable Python callbacks, exceptions, and iterator cleanup. The paper does not establish a speedup or unrestricted fusion for this library.

Reference 19 assumes an associative aggregate operation. A local probe using the existing reducer law checker accepted a small exact-integer sum sample and rejected the corresponding floating-point counterexample. This does not prove integer laws for arbitrary user reducers. DABA remains a candidate requiring an explicit algebraic contract; it cannot silently replace arbitrary window callbacks or change initial partial-window behavior.

October 18: completed review and adopted changes (2.4.0)

All 25 papers in this catalog were read in full. The project, specification, and commercial references were reviewed through relevant architecture documents, public documentation, and selected core implementations. Earlier links in the roadmap received the same scoped review. This does not constitute an audit of every upstream repository or inspection of closed-source engine internals.

The resulting changes reuse fpstreams' existing execution paths:

  • Query-wide task/file limits and task-scope limits normalize integer-index values once and reject nonintegers before resource acquisition.
  • Synchronous filter_map() removes tuple adaptation while preserving iterator boundaries. Async filter_map() uses the existing serial fusion stage. Both call the mapper once per input and discard only None.
  • Report metadata export adds a schema version, metric units, resource scope, and event limits. Plan dictionaries also carry a schema version and execution mode. These are fpstreams schemas, not OpenLineage RunEvents or executable Substrait plans.
  • Guarded Arrow comparison chains can share a selection mask, filter payload once, or count from the mask. Existing conversion, type, and source-boundary checks remain in place.
  • Spill decoding releases encoded buffers after decoding and completed decoded frames before reading the next frame.

The adopted changes passed the complete Python checks, existing performance gates, package checks, and installed-wheel checks on standard Linux CPython 3.11–3.14. These changes are included in 2.4.0. The review did not establish whole-application speedups.

A synchronous cross-node fusion prototype failed performance gates and was withdrawn. The existing window_reduce() uses Two-Stacks Lite; it was not replaced with DABA and does not promise worst-case constant work per operation. Event-time windows, distributed execution, approximate aggregations, and automatic reordering of arbitrary callbacks require separate contracts and were not adopted. Offline Optuna tuning informed the Arrow mask thresholds; the research dependencies and trial records are not product dependencies.

Cross-project follow-up (Unreleased)

The next batch adds explicit shared statistics, optional sketches, rolling numerical statistics, and conservative field dependencies. These changes use the existing Collector, map_accum, and plan explanation interfaces. The following sources supplement the earlier numbered list:

The field metadata is a static dependency description, not a per-record provenance implementation. Sketch estimates preserve the exact APIs and add their own type, missing-value, approximation, and merge contracts. See collectors, rolling statistics, and field dependencies.