Go concurrency applied to data pipelines

A different approach to batch processing, and how to potentiate the power of data pipelines throughout the use of the Go concurrency model.

Introduction to pipelines

Previous problem and solution

Example of architecture for handling in batches

A better approach for data pipelines: streams of data

Implementation example

The pipeline invocation
Generator pattern to stream values by using a channel for it
Fetching users and return them on the channel
Filtering inactive users out
Aggregating user’s profiles to the payload
Converting the payload to a reduced version of it

Final thoughts

Staff Software Engineer at Pomelo