Go concurrency applied to data pipelines

A different approach to batch processing, and how to potentiate the power of data pipelines throughout the use of the Go concurrency model.

Introduction to pipelines

Previous problem and solution

Example of architecture for handling in batches

A better approach for data pipelines: streams of data

Implementation example

The pipeline invocation
Generator pattern to stream values by using a channel for it
Fetching users and return them on the channel
Filtering inactive users out
Aggregating user’s profiles to the payload
Converting the payload to a reduced version of it

Final thoughts

IT Project Lead at Mercadolibre

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store