데이터 엔지니어

Operating System/Network

[network] forward proxy and reverse proxy

1. proxy 프록시는 보통 중계기라고 해석 2. forward proxy 사람 -> proxy서버 -> 인터넷 -> 구글, 네이버, 다음 ... 3. reverse proxy 사람 -> 인터넷 -> proxy서버 -> 서비스(내부서버) 4. 프록시를 사용하는 이유 - 보안 강화 - 속도 향상 - 안전성 - 신뢰성

Operating System/Docker

[docker] docker compose로 nginx 컨테이너 만들기 (docker-compose nginx)

파일구조 ./docker-nginx/docker-compose.yml ./docker-nginx/docker-compose.yml version: '3' services: nginx: hostname: nginx container_name: nginx image: nginx:latest ports: - "80:80"

Operating System/Docker

[Docker] docker compose로 elk stack 컨테이너 만들기 (Docker-compose elasticsearch logstash kibana)

파일구조 ./docker_elk ./docker_elk/elasticsearch/config/elasticsearch.yml ./docker_elk/logstash/config/logstash.yml ./docker_elk/logstash/config/pipelines.yml ./docker_elk/logstash/pipeline/0_api_file.conf ./docker_elk/kibana/test/0_api_file.csv ./docker_elk/kibana/config/kibana.yml ./docker_elk/docker-compose.yml ./docker_elk/elasticsearch/config/elasticsearch.yml --- ## Default Elasticsearch confi..

Database/ElasticSearch

[logstash] input file output mysql pipeline .conf

input file output mysql input { file { path => "/usr/share/logstash/test/2_minio_file.csv" start_position => "beginning" sincedb_path => "/dev/null" codec => "json" } } filter { } output { jdbc { driver_jar_path => "/usr/share/logstash/jarfile/mysql-connector-java-8.0.15.jar" driver_class => "com.mysql.jdbc.Driver" connection_string => "jdbc:mysql://mysql:3306/test_db?user=root&password=root" st..

Database/ElasticSearch

[logstash] input file output minio pipeline .conf

input file output minio input { file { path => "/usr/share/logstash/test/2_minio_file.csv" start_position => "beginning" sincedb_path => "/dev/null" codec => "json" } } filter { } output { s3 { access_key_id => "minio" secret_access_key => "minio123" bucket => "bucketcpi" endpoint => "http://nginx:9000" additional_settings => { force_path_style => true follow_redirects => false } rotation_strate..

Database/ElasticSearch

[logstash] input api output elasticsearch pipeline .conf

input api output elasticsearch input { http_poller { urls => { myurl => "http://api.openweathermap.org/data/2.5/weather?q=Seoul&appid=5af8b7bbeacd98abe5e5e4b84a46d520" } schedule => { cron => "*/1 * * * * UTC" } codec => "json" } } filter { mutate { copy => { "[main][pressure]" => "[main_pressure]" "[main][humidity]" => "[main_humidity]" "[main][temp_max]" => "[main_temp_max]" "[main][grnd_level..

Database/ElasticSearch

[logstash] input api output kafka pipeline .conf

input api output kafka input { http_poller { urls => { myurl => "http://api.openweathermap.org/data/2.5/weather?q=Seoul&appid=5af8b7bbeacd98abe5e5e4b84a46d520" } schedule => { cron => "*/1 * * * * UTC" } codec => "json" } } filter { mutate { remove_field => ["weather"] } ruby { code => "event.set('@timestamp', LogStash::Timestamp.new(event.get('@timestamp')+(9*60*60)))" } } output { kafka { boot..

Database/ElasticSearch

[logstash] input minio output file pipeline .conf

input minio output file input { s3 { access_key_id => "minio" secret_access_key => "minio123" bucket => "bucketfpi" endpoint => "http://nginx:9000" additional_settings => { force_path_style => true follow_redirects => false } codec => "json" } } filter { } output { file { path => "/usr/share/logstash/test/2_minio_file.csv" } }

Database/ElasticSearch

[logstash] input api output minio pipeline .conf

input api output minio input { http_poller { urls => { myurl => "http://api.openweathermap.org/data/2.5/weather?q=Seoul&appid=5af8b7bbeacd98abe5e5e4b84a46d520" } schedule => { cron => "*/1 * * * * UTC" } codec => "json" } } filter { } output { s3 { access_key_id => "minio" secret_access_key => "minio123" bucket => "apibucket" endpoint => "http://nginx:9000" additional_settings => { force_path_st..

Database/ElasticSearch

[logstash] input api output file pipeline .conf

input api output file input { http_poller { urls => { myurl => "http://api.openweathermap.org/data/2.5/weather?q=Seoul&appid=5af8b7bbeacd98abe5e5e4b84a46d520" } schedule => { cron => "*/1 * * * * UTC" } codec => "json" } } filter { } output { file { path => "/usr/share/logstash/test/0_api_file.csv" # start_position => "beginning" # sincedb_path => "/dev/null" } }

박경태
'분류 전체보기' 카테고리의 글 목록 (93 Page)