Functionality At a Glance

Streamly allows the programmer to write high-performance code concisely and idiomatically using high level constructs, and with high expressivity.

Streams

Streaming: Streamly is a general purpose computing framework based on data flow programming paradigm also known as the streaming paradigm. Streaming enables writing modular and composable applications declaratively.

High Performance: Streamly focuses on high performance in all areas comparable to low programming languages like C. To achieve that streamly uses a GHC optimization popularly known as stream fusion. All abstractions in streamly are designed for stream fusion. It makes sure that stream fusion works reliably. The abstractions in streamly also allow nested stream fusion, for example, the Unfold abstraction is specifically designed for nested fusion (an alternative to concatMap) which does not fuse.

Unified Abstractions: Furthermore, streamly provides a range of unified streaming abstractions for representing real life applications. In general, it includes stream producers and consumers. In particular, streaming abstractions include Stream representing the stream producers, Scan the stream transformers, Fold representing the stream consumers, Parser representing the stream consumers with failure and backtracking. All these abstractions are unified, interwork with each other, they are all designed based on the same underlying principles. All other functionality in streamly and applications based on streamly are designed based on these fundamental abstractions.

Concise: The functionality of lists, list-transformer, logic-programming, streaming, streaming folds, parsers which is covered by numerous library in the Haskell ecosystem are all represented efficiently, with the highest possible performance with just these few abstractions.

Interworking: Streamly streams can be converted to and from other streaming types in the Haskell ecosystem. See the interop examples in the streamly-examples repository.

Arrays

Arrays: Arrays complement streams. While streams are used for in-flight data processing, arrays are used for storing data-at-rest and for random access. If you look carefully, the core functionality in streamly is only streams and arrays, the remaining part is just high level functionality built on these two. All you need to build any application is streams and arrays. Streamly unifies these two fundamental concepts, they are intertwined, some stream functionality requires arrays and some array functionality requires streams. That is also the reason why we cannot separate these two cleanly in different packages.

High Performance: Similar to streams, arrays are designed for high performance. While arrays also provide native and high-performance operations utilizing the random and chunked access nature of arrays, in most cases they can be processed and transformed efficiently using streams. Thus, providing a concise API utilizing streams. Unboxed arrays provide the highest performance.

Unified Abstractions: Streamly provides a wide range of abstractions using arrays to express all types of applications. Unboxed arrays provide the highest performance whereas boxed arrays provide more flexibility. Immutable arrays (the Array type) guarantee that the data does not change whereas mutable arrays (the MutArray type) provide in-place mutation for performance where needed. Pinned arrays provide interfacing with the OS whereas unpinned arrays provide freedom from fragmentation of memory. The Unbox and Serialize type classes assist in high-performance serialization of Haskell data to and from arrays.

Concise: The functionality of array processing and serialization which is covered by numerous library in the Haskell ecosystem are all represented efficiently, well-integrated with streams, with the highest possible performance, and in a concise and unified way. Especially, the functionality of bytestring, text and vector packages can all be represented by the single Array type in streamly and MutArray if you want them to be mutable.

Interworking: Streamly arrays can be converted to and from other popular array types (e.g. bytestring, text and vector) in the Haskell ecosystem at zero cost i.e. without copying. See the streamly-bytestring repository.

Declarative Concurrency

Streamly introduces concurrency to the streaming paradigm preserving the modularity and composability of serial composition. It enables the programmer to write concurrent programs in a high-level declarative manner, without using low level concurrency primitives like threads and synchronization. No explicit thread pools are needed. The degree of concurrency can be automatically adjusted dynamically based on the demand by the consumer.

Arrays are processed using streams and streams are concurrent, therefore, arrays can be processed concurrently.

Reactive Programming

Streaming and concurrency together enable expressing reactive applications conveniently. See the AcidRain game in Streamly Examples for a simple reactive programming example. See the CirclingSquare example in Streamly Examples for a simple SDL based reactive programming example. To summarize, streamly provides a unified computing framework for streaming, non-determinism and functional reactive programming in an elegant and simple API that is a natural extension of pure lists to monadic streams.

Batteries Included

As we discussed above the core abstractions in streamly are streams and arrays. For basic programming needs we also need console IO, file IO, network IO and unicode text processing. All this functionality is provided by streamly using the core streaming and array abstractions.