Orchestration

Wondering how to execute a spark job on an AWS EMR cluster, based on a file upload event on S3? Then this post if for you. In this post we go over how to trigger spark jobs on an AWS EMR cluster, using AWS Lambda. The lambda function will execute in response to an S3 upload event. We will go over this event driven pattern with code snippets and set up a fully functioning pipeline.

Data engineering project for beginners, stream edition. In this post we design and build a simple data streaming pipeline using Apache Kafka, Apache Flink and PostgreSQL DB. We will also review the design and understand some common issues to avoid while building distributed stream processing systems.

Create a streaming pipeline using Docker, Kafka, and Kafka Connect

A step-by-step guide on how to start with Airflow: from your local set-up to creating simple tasks.

A step by step guide on how to build an Event-driven architecture in Azure.

bytewax - Stream Processing Python framework