Cost effective way of running spark in AWS EMR

Using transient clusters

Running spark in EMR can help in analyzing big datasets in a very little time, but when it comes to cost it’s definitely not cheap. It starts becoming very expensive as we get billed by number of nodes & depending upon machine types * duration till it was running. I definitely love the fact amazon can provision these EMR clusters with 100s of nodes in the matter of minutes but can I still use it in a cost effective so that I can get the better value out of it ?

Let’s take an example where twice a day I get a drop of this huge dataset (in CSV format) which I need to process using spark (running it in EMR) in 1 hour each. In order to meet my SLA I would need an EMR cluster with at least 10 core nodes + 1 master node with m4.2xlarge configuration. Now let’s do a cost calculation with this configuration for a month:

This is HUGE, is there a way to can just turn on these clusters when I need and shutdown once we are done with the processing ? Yes, you can and we are going to see how to do that

AWS came up with the concept of transient cluster using which you can build them on demand basis and shutdown once the spark job is complete. In order to that through AWS console, do the following:

So this will take care of running the spark application once the cluster is up and…

Related posts:

Hazbin Hotel is an adult musical dramedy by Vivienne “Viziepop” Medrano, featuring the voices of Jill Harris, Monica Franco, Michael Kovach and presented by Spindle Horse Toons. The pilot for Hazbin…

Ever since its advent, ICOs have been popularly trending among the cryptosphere and is one of the most trending fundraising methods preferred by entrepreneurs for their startups. ICO- Initial Coin…

Most remote work guides I see during these days of COVID-19, are all about optimizing for remote work in a remote company where everyone will keep working remotely forever. I haven’t yet seen any…