[Kubernetes] k8s, helm으로 kafka 설치하는 방법

kafka 네임스페이스 생성

# kubectl -n kafka apply -f namespace.yml
apiVersion: v1
kind: Namespace
metadata:
  name: kafka

 

Helm 차트 추가 및 업데이트 및 확인

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm repo list | grep bitnami

 

버전 확인 및 values.yaml 추출

helm search repo bitnami/kafka --versions
helm show values bitnami/kafka --version=31.1.0 > values.yaml

 

values.yaml 원본은 놔두고 복제

cp values.yaml values_override.yaml

 

values_override.yaml 수정

controller:
  replicaCount: 3 -> 4
  persistence:
    enabled: true -> false
broker:
  persistence:
    enabled: true -> false
listeners:
  client:
    protocol: SASL_PLAINTEXT -> PLAINTEXT

 

Helm 차트로 kafka 설치

helm -n kafka upgrade --install kafka bitnami/kafka \
--values values_override.yaml \
--version 31.1.0

 

설치하면 다음과 같은 출력이 나옴

Release "kafka" has been upgraded. Happy Helming!
NAME: kafka
LAST DEPLOYED: Sun Dec 15 14:21:00 2024
NAMESPACE: kafka
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
CHART NAME: kafka
CHART VERSION: 31.1.0
APP VERSION: 3.9.0

Did you know there are enterprise versions of the Bitnami catalog? For enhanced secure software supply chain features, unlimited pulls from Docker, LTS support, or application customization, see Bitnami Premium or Tanzu Application Catalog. See https://www.arrow.com/globalecs/na/vendors/bitnami for more information.

** Please be patient while the chart is being deployed **

Kafka can be accessed by consumers via port 9092 on the following DNS name from within your cluster:

    kafka.kafka.svc.cluster.local

Each Kafka broker can be accessed by producers via port 9092 on the following DNS name(s) from within your cluster:

    kafka-controller-0.kafka-controller-headless.kafka.svc.cluster.local:9092
    kafka-controller-1.kafka-controller-headless.kafka.svc.cluster.local:9092
    kafka-controller-2.kafka-controller-headless.kafka.svc.cluster.local:9092
    kafka-controller-3.kafka-controller-headless.kafka.svc.cluster.local:9092

To create a pod that you can use as a Kafka client run the following commands:

    kubectl run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.9.0-debian-12-r1 --namespace kafka --command -- sleep infinity
    kubectl exec --tty -i kafka-client --namespace kafka -- bash

    PRODUCER:
        kafka-console-producer.sh \
            --bootstrap-server kafka.kafka.svc.cluster.local:9092 \
            --topic test

    CONSUMER:
        kafka-console-consumer.sh \
            --bootstrap-server kafka.kafka.svc.cluster.local:9092 \
            --topic test \
            --from-beginning

WARNING: There are "resources" sections in the chart not set. Using "resourcesPreset" is not recommended for production. For production installations, please set the following values according to your workload needs:
  - controller.resources
+info https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

 

파드들 잘 올라왔는지 확인

kubectl -n kafka get all

 

카프카 클라이언트 생성 및 접속

kubectl -n kafka run kafka-client --restart='Never' --image docker.io/bitnami/kafka:3.9.0-debian-12-r1 --command -- sleep infinity
kubectl -n kafka exec --tty -i kafka-client -- bash

 

프로듀서 및 컨슈머 명령어

# 프로듀서 접속
kafka-console-producer.sh \
--bootstrap-server kafka.kafka.svc.cluster.local:9092 \
--topic test

# 컨슈머 접속
kafka-console-consumer.sh \
--bootstrap-server kafka.kafka.svc.cluster.local:9092 \
--topic test \
--from-beginning