Data Engineering

See how to write Rust code to create a Delta Table in S3 and add data to it, using your local Windows development environment.

Learn various ways to perform error handling in Rust explained using an example in this easy-to-follow-along blog.

This session will provide answers for some of the biggest questions in the universe: namely, how to take full advantage of Delta Lake streaming. You will be ...

Course schedule with slides, lecture notes, and videos.

Course schedule with slides, lecture notes, and videos.

Discover the most used messaging patterns and how data movement affects operational complexity, costs, and sustainability.

Read writing about Data Architecture in Data Engineer Things. Things learned in our data engineering journey and ideas on data and engineering.

Find out how to use the optimizations of your Delta Lake to increase the performance of the operations and save costs.

This post discusses the pros and cons of Hive-style partioning.

bytewax - Stream Processing Python framework

We are excited to ship the first version of pgroll, a command line tool that offers safe and reversible schema migrations for PostgreSQL

(empty introductory line to avoid a formatting issue with Medium editor)

Most data engineers correlate delta format with Spark and Databricks. That's not true. Delta can be used by so many other tools and most cloud providers have added delta support to their analytics tools. In this post we will see how to use delta from a Rust client.

If you’ve been following along in this series, we’ve previously deployed a Delta Sharing server on AWS. Providing a similar tutorial for…

And how to run it on Kubernetes.

In recent years, I’ve done dozens of training on various data platform topics, for all kinds of audiences. When teaching various data platform concepts and techniques, I find one of the concepts particularly intimidating for many business analysts, especially those who are just starting their journe

Example for article Running Spark 3 with standalone Hive Metastore 3.0

Intro

A cheat sheet to refer to when you run into performance issues with your Spark application.

jar dependencies are very finicky

MinIO is a cloud object storage that offers high-performance, S3 compatible. Native to Kubernetes, MinIO is the only object storage suite…

I'm trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn't include the GlobalStorageStati...

Collection of Minio recipes. Contribute to nitisht/cookbook development by creating an account on GitHub.

I'm launching a pyspark program: $ export SPARK_HOME= $ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip $ python And the py code: from pyspark import SparkContext,

Disaggregated compute and storage for the apprentice Data Engineer

When working on real data products, we will register an account on cloud providers such as Amazon, Azure, or Google so that we are able to…

There are 300+ databases; what do they all do?

In this course, we will explore the vast majority of performance problems in an Apache Spark application: skew, spill, shuffle, storage, and serialization.

YTsaurus is a scalable and fault-tolerant open-source big data platform. - ytsaurus/ytsaurus: YTsaurus is a scalable and fault-tolerant open-source big data platform.