Homepage – bytewax
bytewax - Stream Processing Python framework
Kafka, for your data pipeline? Why not?
Create a streaming pipeline using Docker, Kafka, and Kafka Connect
A Beginner Guide to Airflow
A step-by-step guide on how to start with Airflow: from your local set-up to creating simple tasks.
Event-driven architecture with Azure Eventgrid (Databricks-Azure Data Factory)
A step by step guide on how to build an Event-driven architecture in Azure.
How to trigger a spark job from AWS Lambda
Wondering how to execute a spark job on an AWS EMR cluster, based on a file upload event on S3? Then this post if for you. In this post we go over how to trigger spark jobs on an AWS EMR cluster, using AWS Lambda. The lambda function will execute in response to an S3 upload event. We will go over this event driven pattern with code snippets and set up a fully functioning pipeline.
Data Engineering Project: Stream Edition · Start Data Engineering
Data engineering project for beginners, stream edition. In this post we design and build a simple data streaming pipeline using Apache Kafka, Apache Flink and PostgreSQL DB. We will also review the design and understand some common issues to avoid while building distributed stream processing systems.