Data Engineering/Kafka

Data Engineering/Kafka

[kafka] 주키퍼와 카프카의 모든 볼륨 마운트하기

docker-compose.yml version: '2.1' volumes: zookeeper-1-data: zookeeper-1-datalog: zookeeper-1-logs: zookeeper-2-data: zookeeper-2-datalog: zookeeper-2-logs: zookeeper-3-data: zookeeper-3-datalog: zookeeper-3-logs: kafka-1-data: kafka-1-secrets: kafka-2-data: kafka-2-secrets: kafka-3-data: kafka-3-secrets: services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3..

Data Engineering/Kafka

[kafka] 카프카의 메세지는 파티션이 1이면 순서가 보장된다.

kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 3 --partitions 1 --topic mytopic-3-1 kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 3 --partitions 3 --topic mytopic-3-3 kafka-topics --list --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 kafka-console-producer --broker-list kafka-1..

Data Engineering/Kafka

[kafka] 토픽 생성시 --replication-factor --partitions 확인하기

kafka-topics --list --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 1 --partitions 1 --topic mytopic-1-1 kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 2 --partitions 2 --topic mytopic-2-2 kafka-topics --create --bootstrap-server kaf..

Data Engineering/Kafka

[kafka] Error while executing topic command : Replication factor: 4 larger than available brokers: 3.

카프카 브로커가 3개인데 레플리카 펙터가 4개인 토픽을 만들 때 발생하는 에러이다. [appuser@kafka-1 ~]$ kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 1 --partitions 1 --topic mytopic-1-1 Created topic mytopic-1-1. [appuser@kafka-1 ~]$ kafka-topics --create --bootstrap-server kafka-1:9092,kafka-2:9093,kafka-3:9094 --replication-factor 2 --partitions 2 --topic mytopic-2-2 Create..

Data Engineering/Kafka

[kafka] kafka cluster 구성 후 토픽 생성시의 현상 확인해보기

docker-compose.yml version: '2.1' services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3.6 restart: always ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zookeeper-1:2888:3888;2181 server.2=zookeeper-2:2888:3888;2181 server.3=zookeeper-3:2888:3888;2181 volumes: - type: bind source: ./data/zk-cluster/zookeeper-1/data target: /data read_only..

Data Engineering/Kafka

[Kafka] 카프카 클러스터 만들기 (confluentinc/cp-kafka:7.1.0)

docker-compose.yml version: '2.1' services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3.6 restart: always ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zookeeper-1:2888:3888;2181 server.2=zookeeper-2:2888:3888;2181 server.3=zookeeper-3:2888:3888;2181 volumes: - type: bind source: ./data/zk-cluster/zookeeper-1/data target: /data read_only..

Data Engineering/Kafka

[Zookeeper] 주키퍼에 특정 znode에 어떤 znode가 들어있는지 확인하는 방법

docker exec zookeeper-1 zkCli.sh ls /zookeeper docker exec zookeeper-2 zkCli.sh ls /zookeeper docker exec zookeeper-3 zkCli.sh ls /zookeeper

Data Engineering/Kafka

[Zookeeper] 주키퍼에 어떤 znode가 들어있는지 확인하는 방법

docker-compose.yml version: '2.1' services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3.6 restart: always ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zookeeper-1:2888:3888;2181 server.2=zookeeper-2:2888:3888;2181 server.3=zookeeper-3:2888:3888;2181 volumes: - type: bind source: ./data/zk-cluster/zookeeper-1/data target: /data read_only..

Data Engineering/Kafka

[Zookeeper] 주키퍼 마스터 찾는 방법

docker-compose.yml version: '2.1' services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3.6 restart: always ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zookeeper-1:2888:3888;2181 server.2=zookeeper-2:2888:3888;2181 server.3=zookeeper-3:2888:3888;2181 volumes: - type: bind source: ./data/zk-cluster/zookeeper-1/data target: /data read_only..

Data Engineering/Kafka

[Zookeeper] Zookeeper cluster + Zookeeper UI

docker-compose.yml version: '2.1' services: zookeeper-1: hostname: zookeeper-1 container_name: zookeeper-1 image: zookeeper:3.6 restart: always ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zookeeper-1:2888:3888;2181 server.2=zookeeper-2:2888:3888;2181 server.3=zookeeper-3:2888:3888;2181 volumes: - type: bind source: ./data/zk-cluster/zookeeper-1/data target: /data read_only..

박경태
'Data Engineering/Kafka' 카테고리의 글 목록 (2 Page)