Apache Hadoop — Background
Technologies come and go, but data is here to grow! Introduction Data is the core of every modern application, driving insights, automation, and user experiences. Over time, technologies have evolved
Beyond Backfills
Beyond Backfills is a space to explore the art and craft of data engineering. The goal isn’t just moving bytes—it’s understanding the systems that shape them, and the human curiosity that follows.
Latest articles
AWS EMR – Private Subnets
Introduction In this guide, we’ll walk through creating an Amazon EMR cluster in a private subnet using Terraform. By placing your EMR cluster in a private subnet, you prevent it from being directly accessible from the internet — improving both secur...
Apache Spark Unified Memory
Introduction Apache Spark, a powerful distributed processing engine, relies heavily on memory management for optimal performance.The Unified Memory Manager (UMM) — introduced in Spark 1.6 — plays a ke
Serverless Framework
Introduction In this series, we’ll focus on Serverless Framework, one of the fastest ways to build and deploy serverless applications. If you’ve already worked with Terraform, you know how great it is
AWS S3 Multi-Part Uploads
Once you’ve created an S3 bucket, you’ll likely need to transfer large files — sometimes in the order of gigabytes or even terabytes. AWS S3 allows object sizes of up to 5 TB, but a single PUT upload is limited to 5 GB. To handle larger uploads, S3 o...
Terraform Dynamic Blocks
Introduction In this part, we’ll explore some advanced features of Terraform. You’ll learn about variable types, constructs like for_each, and how to use dynamic blocks to make your Terraform code cle
Terraform Modules
divide and rule
What is Terraform State
Objective In this section, we discuss the concept of Terraform state. Terraform is responsible for creating or altering resources on a platform (for example, deploying pods to Kubernetes or creating resources on AWS). To manage this, Terraform needs ...
Basics of IaC with Terraform
hands on terraform