[Spark] 제플린에서 spark context 옵션에 대해 살펴보기

스파크와 제플린을 실행시킨다.

스파크는 스탠드얼론으로 실행됐고,

마스터노드 1개, 워커노드 3개로 구성돼 있다.

워커노드는 각각 3개의 코어와 5기가의 메모리를 갖고있다.

 

스파크 ui

제플린

 

먼저 실행이 잘 되는지 확인해보자

sc

 

종료

sc.stop

 

기본 옵션을 살펴보자

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));

 

드라이버 메모리 : 1g

익스큐터당 메모리 : 1g

익스큐터당 코어 : 1

익스큐터당 인스턴스 : 2

 

스파크에서 실행된 애플리케이션을 클릭해보자.

 

해당 애플리케이션을 종료시키고 다른 옵션으로 실행시켰을 때 어떨지 한번 보자

sc.stop

 

디폴트옵션을 사용하지말고 직접 옵션을 넣어주자

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

conf.set("spark.driver.memory", "2g")
conf.set("spark.executor.memory", "2g")
conf.set("spark.executor.cores", "2")
conf.set("spark.executor.instances", "1")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));

 

드라이버 메모리 : 2g

익스큐터당 메모리 : 2g

익스큐터당 코어 : 2

익스큐터당 인스턴스 : 1

 

 

종료시켜보자

sc.stop

 

옵션을 수정해보자

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

conf.set("spark.driver.memory", "1g")
conf.set("spark.executor.memory", "512m")
conf.set("spark.executor.cores", "1")
conf.set("spark.executor.instances", "2")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));

드라이버 메모리 : 1g

익스큐터당 메모리 : 512m

익스큐터당 코어 : 1

익스큐터당 인스턴스 : 2

 

 

종료시키기

sc.stop

 

옵션 수정

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

conf.set("spark.driver.memory", "1g")
conf.set("spark.executor.memory", "512m")
conf.set("spark.executor.cores", "1")
conf.set("spark.executor.instances", "1")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));

 

드라이버 메모리 : 1g

익스큐터당 메모리 : 512m

익스큐터당 코어 : 1

익스큐터당 인스턴스 : 1

 

 

종료

sc.stop

 

옵션 수정

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

conf.set("spark.driver.memory", "1g")
conf.set("spark.executor.memory", "512m")
conf.set("spark.executor.cores", "2")
conf.set("spark.executor.instances", "1")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));

 

드라이버 메모리 : 1g

익스큐터당 메모리 : 512m

익스큐터당 코어 : 2

익스큐터당 인스턴스 : 1

 

 

종료

sc.stop

 

마지막 옵션 수정 (내 성능을 초과해서 실행시켰을 때) - 코어초과

import org.apache.spark.{SparkConf, SparkContext}

val conf = new SparkConf().setAppName("Park-Gyeong-Tae")
                          .setMaster("spark://spark-master:7077")

conf.set("spark.driver.memory", "1g")
conf.set("spark.executor.memory", "512m")
conf.set("spark.executor.cores", "2")
conf.set("spark.executor.instances", "2")

val sc = new SparkContext(conf)

println("APP Name : " + sc.appName);
println("Master : " + sc.master);
println("Deploy Mode : " + sc.deployMode);

println("spark.driver.memory : " + conf.get("spark.driver.memory"));
println("spark.executor.memory : " + conf.get("spark.executor.memory"));
println("spark.executor.cores : " + conf.get("spark.executor.cores"));
println("spark.executor.instances : " + conf.get("spark.executor.instances"));