Functional Stream Processing in Scala by Zainab Ali (Early Access)

Modern business applications are moving towards incremental processing. We design realtime data pipelines, event-driven architectures, reactive systems, and use stream-based I/O with databases, files and websockets. You may have heard of streams in media, but stream processing has many wider applications.

Just as functional programming has transformed the way we build applications, functional stream processing lets us write simple, safe, scalable incremental pipelines.

This book is a practical deep-dive into building reliable stream processing pipelines with Scala 3 and fs2.

Contents

Part 1: Introduction

• What is functional stream processing?: We’ll explore areas where stream processing is typically used, and compare it to conventional processing.
• What is fs2?: We’ll be introduced to the fs2 functional stream processing library.
• Designing incremental pipelines: We’ll learn how to chain operations to compose larger pipelines.
• Reasoning about stream processing: We’ll learn to think of streams as iterative programs, develop a mental model of their execution, and use that to predict their output.
• Combining streams: We’ll explore the different methods of combining functional streams.
• Combining streams concurrently: We’ll explore how to use fs2’s concurrent combination operators to write fan-out and fan-in pipelines.

Part 2: Handling side effects

• What are effect systems?: We’ll discuss the key properties of effect systems, and why they are important.
• Working with cats-effect IO: We’ll take a brief tour of cats-effect IO and its suspension of side effects.
• Working with effectful streams: We’ll see how IO lets us write many more practical stream programs.
• Error propagation, recovery and retries: We’ll learn how to recover from errors in an incremental process.
• Bracketing with streams: We’ll learn how to encode traditional try-catch-finally logic in an incremental process.
• Performance optimization: We’ll see how to detect performance problems in streams and address them with chunking.

Part 3: Concurrent systems

• Concurrency and parallelism: We’ll learn how to use streams to write safe, simple concurrent processes.

Part 4: Streams at scale

• Scaling challenges in event-driven systems: We’ll explore the impact of load in incremental pipelines and the phenomena of backpressure.
• Handling backpressure with queues: We’ll use different types of queues to ensure a predictable behaviour under load.

Part 5: Application

We’ll use everything we’ve learned to create an interactive game. We’ll write an event-based stream for receiving user input, have a publisher-subscriber model for handling input and updating game state, and a separate concurrent render loop for displaying state to the screen. We’ll then enhance it with error handling, metrics, and logging, as we would any production-ready application.