Data-Lab

Build your own Apache Spark cluster in standalone mode on Docker with a JupyterLab interface

Read our step-by-step guide to building an Apache Spark cluster based on the Docker virtual environment with JupyterLab and the Apache Livy REST interface.

Set up a Spark cluster in Docker from scratch

This post explains how to setup Apache Spark and run Spark applications on the Hadoop with the Yarn cluster manager that is used to run spark examples as deployment mode client and master as yarn. You can also try running the Spark application in cluster mode. Prerequisites : If you don't have Hadoop & Yarn installed, please Install and Setup Hadoop cluster and setup Yarn on Cluster before proceeding with this article.. Spark Install and Setup In order to install and setup Apache Spark on Hadoop cluster, access Apache Spark Download site and go to the Download Apache Spark section

Spark Data processing in an easy, cost-effective and serverless way

A guide on how to set up an environment to work with Airflow and Spark

Want to install Apache Spark using Apache Bigtop? Step by step tutorial. Bigtop is a package manager for Spark, HBase, Hadoop and other Apache projects related to big data. This tutorial is for Machine Learning engineers and Data Scientists looking for a convenient way to manage big data components of their ecosystem.

I’ve never seen so many posts about Apache Spark before, not sure if it’s 3.0, or because the world is burning down. I’ve written about Spark a few times, even 2 years ago, but it still seems to be steadily increasing in popularity, albeit still missing from many companies tech stacks. With the continued rise […]

Base Docker image with just essentials: Hadoop, Hive and Spark. - GitHub - panovvv/hadoop-hive-spark-docker: Base Docker image with just essentials: Hadoop, Hive and Spark.

Example for article Running Spark 3 with standalone Hive Metastore 3.0

Intro

jar dependencies are very finicky

MinIO is a cloud object storage that offers high-performance, S3 compatible. Native to Kubernetes, MinIO is the only object storage suite…

I'm trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn't include the GlobalStorageStati...

Collection of Minio recipes. Contribute to nitisht/cookbook development by creating an account on GitHub.

I'm launching a pyspark program: $ export SPARK_HOME= $ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip $ python And the py code: from pyspark import SparkContext,

Disaggregated compute and storage for the apprentice Data Engineer

When working on real data products, we will register an account on cloud providers such as Amazon, Azure, or Google so that we are able to…