데이터 엔지니어

Programming Language/Python

[Python] 강화학습 CartPole-v0을 실행해보자 (with python 3.7)

https://github.com/ParkGyeongTae/dqn-atari-breakout/tree/main/0_cartpole-v0-example GitHub - ParkGyeongTae/dqn-atari-breakout Contribute to ParkGyeongTae/dqn-atari-breakout development by creating an account on GitHub. github.com conda create -n py37 python=3.7 conda activate py37 pip install gym==0.23.1 pip install pygame==2.1.2 0_cartpole-v0-example.py import gym env = gym.make('CartPole-v0') ..

Programming Language/Python

[Python] No module named 'pyglet'

pip install pyglet python 0_cartpole-v0-example.py 0_cartpole-v0-example.py import gym env = gym.make('CartPole-v0') env.reset() for i in range(500): env.render() env.step(env.action_space.sample()) env.close() ''' import gym env = gym.make('CartPole-v0') # 새로운 에피소드를 시작 env.reset() # 500 이라는 시간 동안 (약 10초 정도) for i in range(500): # 행동 이전 관찰값 env.render() # 행동 이후 관찰값 env.step(env.action_space.samp..

Programming Language/Python

[Python] 아나콘다로 python 3.6, python 3.7 가상환경 만들고 삭제하는 방법

conda info --envs conda create -n py36 python=3.6 conda create -n py37 python=3.7 conda info --envs conda activate py36 python --version pip list conda deactivate conda activate py37 python --version pip list conda deactivate conda info --envs conda env remove -n py36 conda env remove -n py37 conda info --envs

Data Engineering/Airflow

[Airflow] airflow에 연결된 postgreSQL 테이블 확인하기

https://github.com/ParkGyeongTae/airflow-pgt/tree/main/0_airflow GitHub - ParkGyeongTae/airflow-pgt Contribute to ParkGyeongTae/airflow-pgt development by creating an account on GitHub. github.com sudo -u postgres psql -U postgres -c "\list" sudo -u postgres psql -U postgres -d airflow -c "\list" sudo -u postgres psql -U postgres -d airflow -c "\dt" sudo -u postgres psql -U postgres -d airflow -..

Data Engineering/Airflow

[Airflow] 다양한 Dag Graph 실습해보자 (Bash operator)

https://github.com/ParkGyeongTae/airflow-pgt/tree/main/0_airflow GitHub - ParkGyeongTae/airflow-pgt Contribute to ParkGyeongTae/airflow-pgt development by creating an account on GitHub. github.com from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime dag = DAG ( dag_id = 'my_bash_dag', start_date = datetime(2022, 4, 16), schedule_interval ..

카테고리 없음

[Airflow] Dag의 Bash Operator 만들어보기

https://github.com/ParkGyeongTae/airflow-pgt/tree/main/0_airflow GitHub - ParkGyeongTae/airflow-pgt Contribute to ParkGyeongTae/airflow-pgt development by creating an account on GitHub. github.com from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime dag = DAG ( dag_id = 'my_bash_dag', start_date = datetime(2022, 4, 16), schedule_interval ..

Data Engineering/Airflow

[Airflow] Apache Airflow 실행하기

https://github.com/ParkGyeongTae/airflow-pgt/tree/main/0_airflow GitHub - ParkGyeongTae/airflow-pgt Contribute to ParkGyeongTae/airflow-pgt development by creating an account on GitHub. github.com docker-compose.yml version: '2.1' services: airflow: hostname: airflow container_name: airflow image: airflow-pgt:0.01 restart: always stdin_open: true tty: true ports: - 28080:8080 volumes: - type: bi..

Data Engineering/Spark

[Spark] Zeppelin 사용하여 여러가지 RDD 만드는 방법 parallelize 사용

https://github.com/ParkGyeongTae/spark-pgt/tree/main/2_spark-cluster-zeppelin GitHub - ParkGyeongTae/spark-pgt Contribute to ParkGyeongTae/spark-pgt development by creating an account on GitHub. github.com 입력 %spark val data = sc.parallelize(1 to 100) data.count 출력 data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[3] at parallelize at :26 res13: Long = 100 입력 %spark val data = sc.paral..

Data Engineering/Spark

[Spark] Zeppelin 실행시 초기 로그 확인하기

사용중인 파일 모음 https://github.com/ParkGyeongTae/spark-pgt/tree/main/2_spark-cluster-zeppelin GitHub - ParkGyeongTae/spark-pgt Contribute to ParkGyeongTae/spark-pgt development by creating an account on GitHub. github.com 로그 파일 위치 pwd ll cat zeppelin--zeppelin.out cat zeppelin--zeppelin.log root@zeppelin:/home/zeppelin/logs# cat zeppelin--zeppelin.log WARN [2022-04-15 07:03:09,425] ({main} ZeppelinCo..

Data Engineering/Spark

[Spark] Apache Zeppelin 로그 파일 위치 확인하기

모든 설정파일은 아래에서 확인이 가능합니다. https://github.com/ParkGyeongTae/spark-pgt/tree/main/2_spark-cluster-zeppelin GitHub - ParkGyeongTae/spark-pgt Contribute to ParkGyeongTae/spark-pgt development by creating an account on GitHub. github.com 나는 제플린의 로그가 어디에 쌓이는지 확인하고 싶었다. 기본적으로 제플린의 로그는 제플린 폴더 안에 /logs 라는 곳에 들어간다. pwd ll cd logs ll pwd 로그의 위치는 zeppelin-env.sh 에서 설정할 수 있다. ZEPPELIN_LOG_DIR 만약 로깅되는 위치를 수정하고싶..

박경태
'분류 전체보기' 카테고리의 글 목록 (71 Page)
상단으로

티스토리툴바