[Spark] Spark Cluster 간단하게 만들어보기 (standalone)

모든 설정파일은 아래 깃허브에 저장시켜놓았다.

https://github.com/ParkGyeongTae/spark-pgt/tree/main/1_spark-cluster

GitHub - ParkGyeongTae/spark-pgt

 

파일의 형태는 다음과 같다.

 

스파크를 한두번 설치해보면 어떤파일이 무엇인지 단번에 알 수 있을 것 같다...

먼저 도커파일을 살펴보자

FROM ubuntu:18.04
LABEL maintainer "ParkGyeongTae"

# apt 설치시 입력요청 무시
ENV DEBIAN_FRONTEND=noninteractive

# apt 미러서버 미국(default) -> 한국 변경
RUN sed -i 's@archive.ubuntu.com@kr.archive.ubuntu.com@g' /etc/apt/sources.list

# 자주 사용하는 패키지 설치
RUN apt-get update && \
    apt-get install net-tools -y && \
    apt-get install iputils-ping -y && \
    apt-get install vim -y && \
    apt-get install wget -y

# 작업영역 /home
WORKDIR /home

# jdk
RUN apt-get install openjdk-8-jdk -y

# spark-3.2.1-bin-hadoop3.2
RUN wget https://dlcdn.apache.org/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz && \
    tar -xvf spark-3.2.1-bin-hadoop3.2.tgz && \
    mv spark-3.2.1-bin-hadoop3.2 spark && \
    rm -rf spark-3.2.1-bin-hadoop3.2.tgz

# python 3.8.0
RUN apt-get install python3.8 -y && \
    apt-get install python3-pip -y && \
    rm -rf /usr/bin/python3 && \
    ln -s /usr/bin/python3.8 /usr/bin/python3 && \
    ln -s /usr/bin/python3.8 /usr/bin/python

# pip3 설정
RUN mkdir /root/.pip && \
    set -x \
    && { \
    echo '[global]'; \
    echo 'timeout = 60'; \
    echo '[freeze]'; \
    echo 'timeout = 10'; \
    echo '[list]'; \
    echo 'format = columns'; \
    } > /root/.pip/pip.conf && \
    pip3 install --upgrade pip
    

# 환경설정
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64
ENV SPARK_HOME /home/spark
ENV PATH $PATH:$JAVA_HOME/bin:$SPARK_HOME/bin

# spark 설정파일 수정
COPY ./spark-cluster/spark-cluster-conf/spark-env.sh /home/spark/conf/spark-env.sh
COPY ./spark-cluster/spark-cluster-conf/log4j.properties /home/spark/conf/log4j.properties

RUN rm -rf /home/spark/conf/spark-env.sh.template && \
    rm -rf /home/spark/conf/log4j.properties.template && \
    rm -rf /home/spark/bin/*.cmd

# 컨테이너 실행시 spark 자동실행
COPY ./spark-cluster/spark-cluster-entrypoint/entrypoint-spark.sh /usr/local/bin/

ENTRYPOINT ["entrypoint-spark.sh"]

 

다음은 docker-compose.yml 파일을 살펴보자.

version: '2.1'

services:

  spark-master:
    hostname: spark-master
    container_name: spark-master
    image: spark-cluster:0.02
    restart: always
    ports:
      - 18080:18080 # 스파크 웹 UI
    stdin_open: true
    tty: true
    environment:
      - SPARK_MODE=master
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=17077
      - SPARK_MASTER_WEBUI_PORT=18080

  spark-slave-1:
    hostname: spark-slave-1
    container_name: spark-slave-1
    image: spark-cluster:0.02
    restart: always
    stdin_open: true
    tty: true
    ports:
      - 8081:8081 # 스파크 워커 웹 UI
    environment:
      - SPARK_MODE=slave
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=17077
      - SPARK_MASTER_WEBUI_PORT=18080
      - SPARK_WORKER_CORES=4
      - SPARK_WORKER_MEMORY=4g
      - SPARK_WORKER_WEBUI_PORT=8081
      - SPARK_WORKER_PORT=18081
    depends_on:
      - spark-master

  spark-slave-2:
    hostname: spark-slave-2
    container_name: spark-slave-2
    image: spark-cluster:0.02
    restart: always
    stdin_open: true
    tty: true
    ports:
      - 8082:8082 # 스파크 워커 웹 UI
    environment:
      - SPARK_MODE=slave
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=17077
      - SPARK_MASTER_WEBUI_PORT=18080
      - SPARK_WORKER_CORES=4
      - SPARK_WORKER_MEMORY=4g
      - SPARK_WORKER_WEBUI_PORT=8082
      - SPARK_WORKER_PORT=18081
    depends_on:
      - spark-master

  spark-slave-3:
    hostname: spark-slave-3
    container_name: spark-slave-3
    image: spark-cluster:0.02
    restart: always
    stdin_open: true
    tty: true
    ports:
      - 8083:8083 # 스파크 워커 웹 UI
    environment:
      - SPARK_MODE=slave
      - SPARK_MASTER_HOST=spark-master
      - SPARK_MASTER_PORT=17077
      - SPARK_MASTER_WEBUI_PORT=18080
      - SPARK_WORKER_CORES=4
      - SPARK_WORKER_MEMORY=4g
      - SPARK_WORKER_WEBUI_PORT=8083
      - SPARK_WORKER_PORT=18081
    depends_on:
      - spark-master

 

현재 내 도커 상태를 확인해보자

docker ps -a
docker images

 

내가 만든 도커파일로 도커 이미지를 만들어보자!

./full_build.sh

 

docker-compose up -d

 

컨테이너가 잘 실행됐는지 확인해보자!

docker ps
docker images

 

localhost:18080에 접속해보자

 

각 워커 웹에 접속해보자!

localhost:8081, localhost:8082, localhost:8083 순서이다.

 

 

 

잘 연결된 것을 확인할 수 있다.