Commit 6b115c3c by Guangbo Chen

Bump Kafka to v0.8.0 with confluent

parent 380bd960
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
# Images and archives
*.png
*.tgz
scripts/
screenshots/
docs/index.html
\ No newline at end of file
apiVersion: v1
description: A Helm chart for Confluent Open Source
name: kafka
version: 0.8.0
appVersion: 5.3.0
keywords:
- kafka
- zookeeper
- kafka statefulset
home: https://kafka.apache.org/
sources:
- https://github.com/confluentinc/cp-helm-charts
icon: file://./logo.png
maintainers:
- name: qshao-pivotal
email: qshao@pivotal.io
# Confluent Open Source Helm Chart
**The Confluent Platform Helm charts are in developer preview and are not supported for production use.**
The [Confluent Platform Helm charts](https://github.com/confluentinc/cp-helm-charts) enable you to deploy Confluent Platform services on Kubernetes for development, test, and proof of concept environments.
## Installing Charts
```
helm repo add confluentinc https://confluentinc.github.io/cp-helm-charts/
helm repo update
```
## Documentation
The Confluent Helm Chart documentation is located at [docs.confluent.io](https://docs.confluent.io/current/quickstart/cp-helm-charts/docs/index.html).
## Contributing
We welcome any contributions:
- Report all enhancements, bugs, and tasks as [GitHub issues](https://github.com/confluentinc/cp-helm-charts/issues)
- Provide fixes or enhancements by opening pull requests in GitHub
## Thanks
Huge thanks to:
- [Kafka helm chart](https://github.com/kubernetes/charts/tree/master/incubator/kafka)
- [ZooKeeper helm chart](https://github.com/kubernetes/charts/tree/master/incubator/zookeeper)
- [Schema Registry helm chart](https://github.com/kubernetes/charts/tree/master/incubator/schema-registry)
- [kubernetes-kafka](https://github.com/Yolean/kubernetes-kafka)
- [docker-kafka](https://github.com/solsson/dockerfiles)
# Kafka
Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
## Confluent Kafka Chart
**The Confluent Kafka charts are in developer preview and are not supported for production use.**
The [Confluent Platform Helm charts](https://github.com/confluentinc/cp-helm-charts) enable you to deploy Confluent Kafka services on Kubernetes for development, test, and proof of concept environments.
## Chart Details
This chart bootstraps a [Confluent](https://docs.confluent.io/current/) Kafka platform v5.3.0. The chart has the following components,
- confluent-kafka(v2.3.0)
- confluent-zookeeper
- confluent-Kafka-schema-registry
- confluent-Kafka-rest
- confluent-kafka-ksql
- confluent-kafka-connect
- confluent-control-center(free 30 days trail, enterprise only)
- Kafka-topics-ui (This project is licensed under the [BSL](http://www.landoop.com/bsl) license.)
**Warning:** upgrade from previous version is currently not supported, please re-crate a new kafka app.
\ No newline at end of file
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart Confluent Control Center for Kubernetes
name: cp-control-center
version: 0.1.0
This chart installs a Confluent Control Center
To connect to Confluent Control Center directly from outside the k8s cluster:
1. Execute the following command to setup port forward from k8s to localhost
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "cp-control-center.name" . }}" -o jsonpath="{.items[0].metadata.name}")
kubectl port-forward $POD_NAME {{ .Values.serviceHttpPort }}
2. Open a browser and visit http://localhost:{{ .Values.serviceHttpPort }}
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-control-center.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-control-center.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-control-center.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified kafka headless name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-control-center.cp-kafka-headless.fullname" -}}
{{- $name := "cp-kafka-headless" -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Kafka URL. If Kafka is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-control-center.kafka.bootstrapServers" -}}
{{- if .Values.kafka.bootstrapServers -}}
{{- .Values.kafka.bootstrapServers -}}
{{- else -}}
{{- printf "PLAINTEXT://%s:9092" (include "cp-control-center.cp-kafka-headless.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified schema registry name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-control-center.cp-schema-registry.fullname" -}}
{{- $name := default "cp-schema-registry" (index .Values "cp-schema-registry" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-control-center.cp-schema-registry.service-name" -}}
{{- if (index .Values "cp-schema-registry" "url") -}}
{{- printf "%s" (index .Values "cp-schema-registry" "url") -}}
{{- else -}}
{{- printf "http://%s:8081" (include "cp-control-center.cp-schema-registry.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified connect name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-control-center.cp-kafka-connect.fullname" -}}
{{- $name := default "cp-kafka-connect" (index .Values "cp-kafka-connect" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-control-center.cp-kafka-connect.service-name" -}}
{{- if (index .Values "cp-kafka-connect" "url") -}}
{{- printf "%s" (index .Values "cp-kafka-connect" "url") -}}
{{- else -}}
{{- printf "http://%s:8083" (include "cp-control-center.cp-kafka-connect.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified ksql name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-control-center.cp-ksql-server.fullname" -}}
{{- $name := default "cp-ksql-server" (index .Values "cp-ksql-server" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-control-center.cp-ksql-server.service-name" -}}
{{- if (index .Values "cp-ksql-server" "url") -}}
{{- printf "%s" (index .Values "cp-ksql-server" "url") -}}
{{- else -}}
{{- printf "http://%s:8088" (include "cp-control-center.cp-ksql-server.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified zookeeper name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-control-center.cp-zookeeper.fullname" -}}
{{- $name := default "cp-zookeeper" (index .Values "cp-zookeeper" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Zookeeper URL. If zookeeper is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-control-center.cp-zookeeper.service-name" }}
{{- if (index .Values "cp-zookeeper" "enabled") -}}
{{- $clientPort := default 2181 (index .Values "cp-zookeeper" "clientPort") | int -}}
{{- printf "%s:%d" (include "cp-control-center.cp-zookeeper.fullname" .) $clientPort }}
{{- else -}}
{{- $zookeeperConnect := printf "%s" (index .Values "cp-zookeeper" "url") }}
{{- $zookeeperConnectOverride := (index .Values "configurationOverrides" "zookeeper.connect") }}
{{- default $zookeeperConnect $zookeeperConnectOverride }}
{{- end -}}
{{- end -}}
\ No newline at end of file
{{- if .Values.global.enterprise }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "cp-control-center.fullname" . }}
labels:
app: {{ include "cp-control-center.name" . }}
chart: {{ include "cp-control-center.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ include "cp-control-center.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ include "cp-control-center.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: {{ .Values.imagePullPolicy }}
ports:
- name: cc-http
containerPort: {{ .Values.serviceHttpPort}}
protocol: TCP
resources:
{{- toYaml .Values.resources | indent 12 }}
env:
- name: CONTROL_CENTER_BOOTSTRAP_SERVERS
value: {{ template "cp-control-center.kafka.bootstrapServers" . }}
- name: CONTROL_CENTER_ZOOKEEPER_CONNECT
value: {{ template "cp-control-center.cp-zookeeper.service-name" . }}
- name: CONTROL_CENTER_CONNECT_CLUSTER
value: {{ template "cp-control-center.cp-kafka-connect.service-name" . }}
- name: CONTROL_CENTER_KSQL_URL
value: {{ template "cp-control-center.cp-ksql-server.service-name" . }}
- name: CONTROL_CENTER_KSQL_ADVERTISED_URL
value: {{ template "cp-control-center.cp-ksql-server.service-name" . }}
- name: CONTROL_CENTER_SCHEMA_REGISTRY_URL
value: {{ template "cp-control-center.cp-schema-registry.service-name" . }}
- name: KAFKA_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{- range $key, $value := .Values.configurationOverrides }}
- name: {{ printf "CONTROL_CENTER_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- range $key, $value := .Values.customEnv }}
- name: {{ $key | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- end }}
{{- if .Values.global.enterprise }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-control-center.fullname" . }}
labels:
app: {{ template "cp-control-center.name" . }}
chart: {{ template "cp-control-center.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
type: {{ .Values.serviceType }}
ports:
- name: cc-http
port: {{ .Values.serviceHttpPort }}
selector:
app: {{ template "cp-control-center.name" . }}
release: {{ .Release.Name }}
{{- end }}
# Default values for cp-control-center.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-kafka/
image: confluentinc/cp-enterprise-control-center
imageTag: 5.2.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
serviceHttpPort: 9021
serviceType: NodePort
## Control Center properties
## ref:
configurationOverrides:
"replication.factor": "3"
# "internal.topics.replication": "1"
# "monitoring.interceptor.topic.replication": "1"
# "metrics.topic.replication": "1"
# "command.topic.replication": "1"
# "streams.num.stream.threads": "2"
# "streams.consumer.request.timeout.ms": "960032"
# "rest.listeners": "http://0.0.0.0:9021"
## Kafka Connect JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
## Additional env variables
customEnv: {}
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## Kafka JMX Settings
## ref: https://docs.confluent.io/current/kafka/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
## You can list load balanced service endpoint, or list of all brokers (which is hard in K8s). e.g.:
## bootstrapServers: "PLAINTEXT://dozing-prawn-kafka-headless:9092"
kafka:
bootstrapServers: ""
## If the Kafka Chart is disabled a URL and port are required to connect
## e.g. gnoble-panther-cp-schema-registry:8081
cp-schema-registry:
url: ""
cp-kafka-connect:
url: ""
cp-ksql-server:
url: ""
cp-zookeeper:
## If the Zookeeper Chart is disabled a URL and port are required to connect
url: ""
\ No newline at end of file
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent Kafka Connect on Kubernetes
name: cp-kafka-connect
version: 0.1.0
# CP-Kafka Connect Helm Chart
This chart bootstraps a deployment of a Confluent Kafka Connect
## Prerequisites
* Kubernetes 1.9.2+
* Helm 2.8.2+
* A healthy and accessible Kafka Cluster
## Developing Environment:
* [Pivotal Container Service (PKS)](https://pivotal.io/platform/pivotal-container-service)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/)
## Docker Image Source:
* [DockerHub -> ConfluentInc](https://hub.docker.com/u/confluentinc/)
## Installing the Chart
### Install along with cp-helm-charts
```console
git clone https://github.com/confluentinc/cp-helm-charts.git
helm install cp-helm-charts
```
To install with a specific name, you can do:
```console
helm install --name my-confluent cp-helm-charts
```
### Install with a existing CP-Kafka release
```console
helm install --set kafka.bootstrapServers="PLAINTEXT://lolling-chinchilla-cp-kafka-headless:9092",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-kafka-connect
```
### Installed Components
You can use `helm status <release name>` to view all of the installed components.
For example:
```console
$ helm status kissing-macaw
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kissing-macaw-cp-kafka-connect ClusterIP 10.19.253.217 <none> 8083/TCP 34m
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kissing-macaw-cp-kafka-connect 1 1 1 1 34m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
kissing-macaw-cp-kafka-connect-6c77b8f5fd-cqlzq 1/1 Running 0 34m
```
There are
1. A [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) `kissing-macaw-cp-kafka-connect` which contains 1 Kafka Connect [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/): `kissing-macaw-cp-kafka-connect-6c77b8f5fd-cqlzq`.
2. A [Service](https://kubernetes.io/docs/concepts/services-networking/service/) `kissing-macaw-cp-kafka-connect` for clients to connect to Kafka Connect REST endpoint.
## Configuration
You can specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```console
helm install --name my-kafka-connect -f my-values.yaml ./cp-kafka-connect
```
> **Tip**: A default [values.yaml](values.yaml) is provided
### Kafka Connect Deployment
The configuration parameters in this section control the resources requested and utilized by the `cp-kafka-connect` chart.
| Parameter | Description | Default |
| ----------------- | ------------------------------------- | --------- |
| `replicaCount` | The number of Kafka Connect Servers. | `1` |
### Image
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `image` | Docker Image of Confluent Kafka Connect. | `confluentinc/cp-kafka-connect` |
| `imageTag` | Docker Image Tag of Confluent Kafka Connect. | `5.3.0` |
| `imagePullPolicy` | Docker Image Tag of Confluent Kafka Connect. | `IfNotPresent` |
| `imagePullSecrets` | Secrets to be used for private registries. | see [values.yaml](values.yaml) for details |
### Port
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `servicePort` | The port on which the Kafka Connect will be available and serving requests. | `8082` |
### Kafka Connect Worker Configurations
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `configurationOverrides` | Kafka Connect [configuration](https://docs.confluent.io/current/connect/references/allconfigs.html) overrides in the dictionary format. | `{}` |
| `customEnv` | Custom environmental variables | `{}` |
### Kafka Connect JVM Heap Options
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `heapOptions` | The JVM Heap Options for Kafka Connect | `"-Xms512M -Xmx512M"` |
### Resources
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `resources.requests.cpu` | The amount of CPU to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.memory` | The amount of memory to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit CPU usage for a Kafka Connect Pod. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit memory usage for a Kafka Connect Pod. | see [values.yaml](values.yaml) for details |
### Annotations
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `podAnnotations` | Map of custom annotations to attach to the pod spec. | `{}` |
### JMX Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `jmx.port` | The jmx port which JMX style metrics are exposed. | `5555` |
### Prometheus JMX Exporter Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `prometheus.jmx.enabled` | Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. | `true` |
| `prometheus.jmx.image` | Docker Image for Prometheus JMX Exporter container. | `solsson/kafka-prometheus-jmx-exporter@sha256` |
| `prometheus.jmx.imageTag` | Docker Image Tag for Prometheus JMX Exporter container. | `6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143` |
| `prometheus.jmx.imagePullPolicy` | Docker Image Pull Policy for Prometheus JMX Exporter container. | `IfNotPresent` |
| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping. | `5556` |
| `prometheus.jmx.resources` | JMX Exporter resources configuration. | see [values.yaml](values.yaml) for details |
### Deployment Topology
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `nodeSelector` | Dictionary containing key-value-pairs to match labels on nodes. When defined pods will only be scheduled on nodes, that have each of the indicated key-value pairs as labels. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) | `{}`
| `tolerations`| Array containing taint references. When defined, pods can run on nodes, which would otherwise deny scheduling. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) | `{}`
## Dependencies
### Kafka
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `kafka.bootstrapServers` | Bootstrap Servers for Kafka Connect | `""` |
This chart installs a Confluent Kafka Connect
https://docs.confluent.io/current/connect/index.html
\ No newline at end of file
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-kafka-connect.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-kafka-connect.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-kafka-connect.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified kafka headless name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka-connect.cp-kafka-headless.fullname" -}}
{{- $name := "cp-kafka-headless" -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Kafka URL. If Kafka is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-kafka-connect.kafka.bootstrapServers" -}}
{{- if .Values.kafka.bootstrapServers -}}
{{- .Values.kafka.bootstrapServers -}}
{{- else -}}
{{- printf "PLAINTEXT://%s:9092" (include "cp-kafka-connect.cp-kafka-headless.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified schema registry name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka-connect.cp-schema-registry.fullname" -}}
{{- $name := default "cp-schema-registry" (index .Values "cp-schema-registry" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-kafka-connect.cp-schema-registry.service-name" -}}
{{- if (index .Values "cp-schema-registry" "url") -}}
{{- printf "%s" (index .Values "cp-schema-registry" "url") -}}
{{- else -}}
{{- printf "http://%s:8081" (include "cp-kafka-connect.cp-schema-registry.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Default GroupId to Release Name but allow it to be overridden
*/}}
{{- define "cp-kafka-connect.groupId" -}}
{{- if .Values.overrideGroupId -}}
{{- .Values.overrideGroupId -}}
{{- else -}}
{{- .Release.Name -}}
{{- end -}}
{{- end -}}
\ No newline at end of file
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "cp-kafka-connect.fullname" . }}
labels:
app: {{ template "cp-kafka-connect.name" . }}
chart: {{ template "cp-kafka-connect.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "cp-kafka-connect.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "cp-kafka-connect.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-kafka-connect/jmx-kafka-connect-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 12 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-kafka-connect
{{- end }}
- name: {{ template "cp-kafka-connect.name" . }}-server
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- name: kafka-connect
containerPort: {{ .Values.servicePort}}
protocol: TCP
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
resources:
{{ toYaml .Values.resources | indent 12 }}
env:
- name: CONNECT_REST_ADVERTISED_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: CONNECT_BOOTSTRAP_SERVERS
value: {{ template "cp-kafka-connect.kafka.bootstrapServers" . }}
- name: CONNECT_GROUP_ID
value: {{ template "cp-kafka-connect.groupId" . }}
- name: CONNECT_CONFIG_STORAGE_TOPIC
value: {{ template "cp-kafka-connect.fullname" . }}-config
- name: CONNECT_OFFSET_STORAGE_TOPIC
value: {{ template "cp-kafka-connect.fullname" . }}-offset
- name: CONNECT_STATUS_STORAGE_TOPIC
value: {{ template "cp-kafka-connect.fullname" . }}-status
- name: CONNECT_KEY_CONVERTER_SCHEMA_REGISTRY_URL
value: {{ template "cp-kafka-connect.cp-schema-registry.service-name" .}}
- name: CONNECT_VALUE_CONVERTER_SCHEMA_REGISTRY_URL
value: {{ template "cp-kafka-connect.cp-schema-registry.service-name" .}}
- name: KAFKA_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{- range $key, $value := .Values.configurationOverrides }}
- name: {{ printf "CONNECT_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- range $key, $value := .Values.customEnv }}
- name: {{ $key | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.jmx.port }}
- name: KAFKA_JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-kafka-connect.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-kafka-connect.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-kafka-connect.name" . }}
chart: {{ template "cp-kafka-connect.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-kafka-connect-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern : "kafka.connect<type=connect-worker-metrics>([^:]+):"
name: "cp_kafka_connect_connect_worker_metrics_$1"
- pattern : "kafka.connect<type=connect-metrics, client-id=([^:]+)><>([^:]+)"
name: "cp_kafka_connect_connect_metrics_$1_$2"
- pattern : "kafka.connect<type=connector-task-metrics, connector=([^:]+), task=([^:]+)><>status: ([^:]+)"
name: "cp_kafka_connect_connect_connector_metrics"
value: 1
labels:
connector: $1
task: $2
status: $3
{{- end }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka-connect.fullname" . }}
labels:
app: {{ template "cp-kafka-connect.name" . }}
chart: {{ template "cp-kafka-connect.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- name: kafka-connect
port: {{ .Values.servicePort }}
selector:
app: {{ template "cp-kafka-connect.name" . }}
release: {{ .Release.Name }}
# Default values for cp-kafka-connect.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-kafka/
image: confluentinc/cp-kafka-connect
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
servicePort: 8083
## Kafka Connect properties
## ref: https://docs.confluent.io/current/connect/userguide.html#configuring-workers
configurationOverrides:
"plugin.path": "/usr/share/java"
"key.converter": "io.confluent.connect.avro.AvroConverter"
"value.converter": "io.confluent.connect.avro.AvroConverter"
"key.converter.schemas.enable": "false"
"value.converter.schemas.enable": "false"
"internal.key.converter": "org.apache.kafka.connect.json.JsonConverter"
"internal.value.converter": "org.apache.kafka.connect.json.JsonConverter"
"config.storage.replication.factor": "3"
"offset.storage.replication.factor": "3"
"status.storage.replication.factor": "3"
## Kafka Connect JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
## Additional env variables
customEnv: {}
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## Kafka Connect JMX Settings
## ref: https://kafka.apache.org/documentation/#connect_monitoring
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
## You can list load balanced service endpoint, or list of all brokers (which is hard in K8s). e.g.:
## bootstrapServers: "PLAINTEXT://dozing-prawn-kafka-headless:9092"
kafka:
bootstrapServers: ""
## If the Kafka Chart is disabled a URL and port are required to connect
## e.g. gnoble-panther-cp-schema-registry:8081
cp-schema-registry:
url: ""
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent Kafka REST Proxy on Kubernetes
name: cp-kafka-rest
version: 0.1.0
# CP-REST Proxy Helm Chart
This chart bootstraps a deployment of a Confluent REST Proxy
## Prerequisites
* Kubernetes 1.9.2+
* Helm 2.8.2+
* A healthy and accessible Zookeeper Ensemble of the Kafka Cluster
## Developing Environment
* [Pivotal Container Service (PKS)](https://pivotal.io/platform/pivotal-container-service)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/)
## Docker Image Source
* [DockerHub -> ConfluentInc](https://hub.docker.com/u/confluentinc/)
## Installing the Chart
### Install along with cp-helm-charts
```console
git clone https://github.com/confluentinc/cp-helm-charts.git
helm install cp-helm-charts
```
To install with a specific name, you can do:
```console
helm install --name my-confluent cp-helm-charts
```
### Install with a existing cp-kafka and cp-schema-registry release
```console
helm install --set cp-zookeeper.url="unhinged-robin-cp-zookeeper:2181",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-kafka-rest
```
### Installed Components
You can use `helm status <release name>` to view all of the installed components.
For example:
```console
$ helm status lolling-chinchilla
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hopping-salamander-cp-kafka-rest ClusterIP 10.19.250.118 <none> 8082/TCP 1m
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
hopping-salamander-cp-kafka-rest 1 1 1 1 1m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
hopping-salamander-cp-kafka-rest-67b86cff98-qxrd8 1/1 Running 0 1m
==> v1/ConfigMap
NAME DATA AGE
hopping-salamander-cp-kafka-rest-jmx-configmap 1 1s
```
There are
1. A [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) `hopping-salamander-cp-kafka-rest` which contains 1 REST Proxy [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/): `hopping-salamander-cp-kafka-rest-67b86cff98-qxrd8`.
1. A [Service](https://kubernetes.io/docs/concepts/services-networking/service/) `hopping-salamander-cp-kafka-rest` for clients to connect to REST Proxy.
1. A [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) which contains configuration for Prometheus JMX Exporter.
1. (Optional) A [Service](https://kubernetes.io/docs/concepts/services-networking/service/) `hopping-salamander-cp-kafka-restproxy-external` for clients to connect to REST Proxy from outside.
## Configuration
You can specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```console
helm install --name my-rest-proxy -f my-values.yaml ./cp-kafka-rest
```
> **Tip**: A default [values.yaml](values.yaml) is provided
### REST Proxy Deployment
The configuration parameters in this section control the resources requested and utilized by the `cp-kafka-rest` chart.
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `replicaCount` | The number of REST Proxy Servers. | `1` |
### Image
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `image` | Docker Image of Confluent REST Proxy. | `confluentinc/cp-kafka-rest` |
| `imageTag` | Docker Image Tag of Confluent REST Proxy. | `5.3.0` |
| `imagePullPolicy` | Docker Image Tag of Confluent REST Proxy. | `IfNotPresent` |
| `imagePullSecrets` | Secrets to be used for private registries. | see [values.yaml](values.yaml) for details |
### Port
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `servicePort` | The port on which the REST Proxy will be available and serving requests. | `8082` |
### Confluent Kafka REST Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `configurationOverrides` | Kafka REST [configuration](https://docs.confluent.io/current/kafka-rest/docs/config.html) overrides in the dictionary format | `{}` |
### Confluent Kafka REST JVM Heap Options
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `heapOptions` | The JVM Heap Options for Confluent Kafka REST | `"-Xms512M -Xmx512M"` |
### Resources
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `resources.requests.cpu` | The amount of CPU to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.memory` | The amount of memory to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit CPU usage for a REST Proxy Pod. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit memory usage for a REST Proxy Pod. | see [values.yaml](values.yaml) for details |
### Annotations
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `podAnnotations` | Map of custom annotations to attach to the pod spec. | `{}` |
### JMX Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `jmx.port` | The jmx port which JMX style metrics are exposed. | `5555` |
### Prometheus JMX Exporter Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `prometheus.jmx.enabled` | Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. | `true` |
| `prometheus.jmx.image` | Docker Image for Prometheus JMX Exporter container. | `solsson/kafka-prometheus-jmx-exporter@sha256` |
| `prometheus.jmx.imageTag` | Docker Image Tag for Prometheus JMX Exporter container. | `6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143` |
| `prometheus.jmx.imagePullPolicy` | Docker Image Pull Policy for Prometheus JMX Exporter container. | `IfNotPresent` |
| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping. | `5556` |
| `prometheus.jmx.resources` | JMX Exporter resources configuration. | see [values.yaml](values.yaml) for details |
### External Access
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `external.enabled` | whether or not to allow external access to Kafka REST Proxy | `false` |
| `external.type` | `Kubernetes Service Type` to expose Kafka REST Proxy to external | `LoadBalancer` |
| `external.port` | External service port to expose Kafka REST Proxy to external | `8082` |
| `external.annotations` | Map of annotations to attach to external Kafka REST Proxy service | `nil` |
| `external.externalTrafficPolicy` | Configures `.spec.externalTrafficPolicy` which controls if load balancing occurs across all nodes (value of `Cluster`) or only active nodes (value of `Local`) | `Cluster` |
| `external.loadBalancerSourceRanges` | Configures `.spec.loadBalancerSourceRanges` which controls a list of source IP ranges permitted access to the load balancer | `["0.0.0.0/0"]` |
### Deployment Topology
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `nodeSelector` | Dictionary containing key-value-pairs to match labels on nodes. When defined pods will only be scheduled on nodes, that have each of the indicated key-value pairs as labels. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) | `{}`
| `tolerations`| Array containing taint references. When defined, pods can run on nodes, which would otherwise deny scheduling. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) | `{}`
## Dependencies
### Zookeeper
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `cp-zookeeper.url` | Service name of Zookeeper cluster (Not needed if this is installed along with cp-kafka chart). | `""` |
| `cp-zookeeper.clientPort` | Port of Zookeeper Cluster | `2181` |
### Schema Registry (optional)
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `cp-schema-registry.url` | Service name of Schema Registry (Not needed if this is installed along with cp-kafka chart). | `""` |
| `cp-schema-registry.port` | Port of Schema Registry Service | `8081` |
This chart installs a Confluent Kafka REST Proxy
https://docs.confluent.io/current/kafka-rest/docs/intro.html
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-kafka-rest.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-kafka-rest.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-kafka-rest.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified zookeeper name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka-rest.cp-zookeeper.fullname" -}}
{{- $name := default "cp-zookeeper" (index .Values "cp-zookeeper" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Zookeeper URL. If zookeeper is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-kafka-rest.cp-zookeeper.service-name" }}
{{- if (index .Values "cp-zookeeper" "url") -}}
{{- printf "%s" (index .Values "cp-zookeeper" "url") }}
{{- else -}}
{{- $clientPort := default 2181 (index .Values "cp-zookeeper" "clientPort") | int -}}
{{- printf "%s-headless:%d" (include "cp-kafka-rest.cp-zookeeper.fullname" .) $clientPort }}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified schema registry name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka-rest.cp-schema-registry.fullname" -}}
{{- $name := default "cp-schema-registry" (index .Values "cp-schema-registry" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-kafka-rest.cp-schema-registry.service-name" -}}
{{- if (index .Values "cp-schema-registry" "url") -}}
{{- printf "%s" (index .Values "cp-schema-registry" "url") -}}
{{- else -}}
{{- printf "http://%s:8081" (include "cp-kafka-rest.cp-schema-registry.fullname" .) -}}
{{- end -}}
{{- end -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "cp-kafka-rest.fullname" . }}
labels:
app: {{ template "cp-kafka-rest.name" . }}
chart: {{ template "cp-kafka-rest.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "cp-kafka-rest.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "cp-kafka-rest.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-kafka-rest/jmx-kafka-rest-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 12 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-kafka-rest
{{- end }}
- name: {{ template "cp-kafka-rest.name" . }}-server
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- name: rest-proxy
containerPort: {{ .Values.servicePort}}
protocol: TCP
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
resources:
{{ toYaml .Values.resources | indent 12 }}
env:
- name: KAFKA_REST_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: KAFKA_REST_ZOOKEEPER_CONNECT
value: {{ template "cp-kafka-rest.cp-zookeeper.service-name" . }}
- name: KAFKA_REST_SCHEMA_REGISTRY_URL
value: {{ template "cp-kafka-rest.cp-schema-registry.service-name" . }}
- name: KAFKAREST_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{- range $key, $value := .Values.configurationOverrides }}
- name: {{ printf "KAFKA_REST_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.jmx.port }}
- name: KAFKA_REST_JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-kafka-rest.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.external.enabled -}}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka-rest.fullname" . }}-external
{{- if .Values.external.annotations }}
annotations:
{{- range $key, $value := .Values.external.annotations }}
{{ $key }}: "{{ $value }}"
{{- end }}
{{- end }}
labels:
app: {{ template "cp-kafka-rest.name" . }}
chart: {{ template "cp-kafka-rest.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
type: {{ .Values.external.type }}
externalTrafficPolicy: {{ .Values.external.externalTrafficPolicy }}
{{- if .Values.external.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- range $cidr := .Values.external.loadBalancerSourceRanges }}
- {{ $cidr }}
{{- end }}
{{- end }}
ports:
- name: rest-proxy
port: {{ .Values.external.port }}
targetPort: {{ .Values.servicePort }}
selector:
app: {{ template "cp-kafka-rest.name" . }}
release: {{ .Release.Name }}
{{- end -}}
\ No newline at end of file
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-kafka-rest.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-kafka-rest.name" . }}
chart: {{ template "cp-kafka-rest.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-kafka-rest-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern : 'kafka.rest<type=jetty-metrics>([^:]+):'
name: "cp_kafka_rest_jetty_metrics_$1"
- pattern : 'kafka.rest<type=jersey-metrics>([^:]+):'
name: "cp_kafka_rest_jersey_metrics_$1"
{{- end }}
\ No newline at end of file
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka-rest.fullname" . }}
labels:
app: {{ template "cp-kafka-rest.name" . }}
chart: {{ template "cp-kafka-rest.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- name: rest-proxy
port: {{ .Values.servicePort }}
selector:
app: {{ template "cp-kafka-rest.name" . }}
release: {{ .Release.Name }}
# Default values for cp-kafka-rest.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-kafka/
image: confluentinc/cp-kafka-rest
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
servicePort: 8082
## Kafka rest JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Kafka REST configuration options
## ref: https://docs.confluent.io/current/kafka-rest/docs/config.html
configurationOverrides:
# "consumer.request.timeout.ms": 5000
## Monitoring
## Kafka REST JMX Settings
## ref: https://docs.confluent.io/current/kafka-rest/docs/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
## External Access
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer
external:
enabled: false
type: LoadBalancer
externalTrafficPolicy: Cluster
port: 8082
## If the Kafka Chart is disabled a URL of zookeeper are required to connect
## e.g. ungaged-sparrow-cp-zookeeper:2181
cp-zookeeper:
url: ""
## If the Kafka Chart is disabled a URL and port are required to connect
## e.g. gnoble-panther-cp-schema-registry:8081
cp-schema-registry:
url: ""
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent Kafka on Kubernetes
name: cp-kafka
version: 0.1.0
dependencies:
- name: cp-zookeeper
version: 0.1.0
repository: file://../cp-zookeeper
condition: cp-zookeeper.enabled
\ No newline at end of file
This chart installs a {{ default 3 .Values.servers }} nodes Confluent Kafka Cluster.
To connect from a client pod:
1. Deploy a kafka client pod with configuration:
apiVersion: v1
kind: Pod
metadata:
name: kafka-client
namespace: {{ .Release.Namespace }}
spec:
containers:
- name: kafka-client
image: {{ .Values.image }}:{{ .Values.imageTag }}
command:
- sh
- -c
- "exec tail -f /dev/null"
2. Log into the Pod
kubectl exec -it kafka-client -- /bin/bash
3. Explore with kafka commands:
# Delete the topic if it exists
kafka-topics --zookeeper {{ template "cp-kafka.cp-zookeeper.service-name" . }} --topic {{ template "cp-kafka.fullname" . }}-topic --delete --if-exists
# Create the topic
kafka-topics --zookeeper {{ template "cp-kafka.cp-zookeeper.service-name" . }} --topic {{ template "cp-kafka.fullname" . }}-topic --create --partitions 1 --replication-factor 1 --if-not-exists
# Create a message
MESSAGE="`date -u`"
# Produce a test message to the topic
echo "$MESSAGE" | kafka-console-producer --broker-list {{ template "cp-kafka.fullname" . }}:9092 --topic {{ template "cp-kafka.fullname" . }}-topic && \
# Consume a test message from the topic
kafka-console-consumer --bootstrap-server {{ template "cp-kafka.fullname" . }}-headless:9092 --topic {{ template "cp-kafka.fullname" . }}-topic --from-beginning --timeout-ms 2000 | grep "$MESSAGE"
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-kafka.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-kafka.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-kafka.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified zookeeper name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka.cp-zookeeper.fullname" -}}
{{- $name := default "cp-zookeeper" (index .Values "cp-zookeeper" "nameOverride") -}}
{{- printf "%s-%s-headless" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Zookeeper URL. If zookeeper is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-kafka.cp-zookeeper.service-name" }}
{{- if (index .Values "cp-zookeeper" "enabled") -}}
{{- $clientPort := default 2181 (index .Values "cp-zookeeper" "clientPort") | int -}}
{{- printf "%s:%d" (include "cp-kafka.cp-zookeeper.fullname" .) $clientPort }}
{{- else -}}
{{- $zookeeperConnect := printf "%s" (index .Values "cp-zookeeper" "url") }}
{{- $zookeeperConnectOverride := (index .Values "configurationOverrides" "zookeeper.connect") }}
{{- default $zookeeperConnect $zookeeperConnectOverride }}
{{- end -}}
{{- end -}}
{{/*
Form the Advertised Listeners. We will use the value of nodeport.firstListenerPort to create the
external advertised listeners if configurationOverrides.advertised.listeners is not set.
*/}}
{{- define "cp-kafka.configuration.advertised.listeners" }}
{{- if (index .Values "configurationOverrides" "advertised.listeners") -}}
{{- printf ",%s" (first (pluck "advertised.listeners" .Values.configurationOverrides)) }}
{{- else -}}
{{- printf ",EXTERNAL://${HOST_IP}:$((%s + ${KAFKA_BROKER_ID}))" (.Values.nodeport.firstListenerPort | toString) }}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified kafka headless name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka.cp-kafka-headless.fullname" -}}
{{- $name := "cp-kafka-headless" -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a variable containing all the datadirs created.
*/}}
{{- define "cp-kafka.log.dirs" -}}
{{- range $k, $e := until (.Values.persistence.disksPerBroker|int) -}}
{{- if $k}}{{- printf ","}}{{end}}
{{- printf "/opt/kafka/data-%d/logs" $k -}}
{{- end -}}
{{- end -}}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka.fullname" . }}-headless
labels:
app: {{ template "cp-kafka.name" . }}
chart: {{ template "cp-kafka.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- port: 9092
name: broker
clusterIP: None
selector:
app: {{ template "cp-kafka.name" . }}
release: {{ .Release.Name }}
\ No newline at end of file
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-kafka.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-kafka.name" . }}
chart: {{ template "cp-kafka.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-kafka-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern : kafka.server<type=ReplicaManager, name=(.+)><>(Value|OneMinuteRate)
name: "cp_kafka_server_replicamanager_$1"
- pattern : kafka.controller<type=KafkaController, name=(.+)><>Value
name: "cp_kafka_controller_kafkacontroller_$1"
- pattern : kafka.server<type=BrokerTopicMetrics, name=(.+)><>OneMinuteRate
name: "cp_kafka_server_brokertopicmetrics_$1"
- pattern : kafka.network<type=RequestMetrics, name=RequestsPerSec, request=(.+)><>OneMinuteRate
name: "cp_kafka_network_requestmetrics_requestspersec_$1"
- pattern : kafka.network<type=SocketServer, name=NetworkProcessorAvgIdlePercent><>Value
name: "cp_kafka_network_socketserver_networkprocessoravgidlepercent"
- pattern : kafka.server<type=ReplicaFetcherManager, name=MaxLag, clientId=(.+)><>Value
name: "cp_kafka_server_replicafetchermanager_maxlag_$1"
- pattern : kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate
name: "cp_kafka_kafkarequesthandlerpool_requesthandleravgidlepercent"
- pattern : kafka.controller<type=ControllerStats, name=(.+)><>OneMinuteRate
name: "cp_kafka_controller_controllerstats_$1"
- pattern : kafka.server<type=SessionExpireListener, name=(.+)><>OneMinuteRate
name: "cp_kafka_server_sessionexpirelistener_$1"
{{- end }}
\ No newline at end of file
{{- if .Values.nodeport.enabled }}
{{- $fullName := include "cp-kafka.fullname" . }}
{{- $brokers := .Values.brokers | int }}
{{- $servicePort := .Values.nodeport.servicePort }}
{{- $root := . }}
{{- range $i, $e := until $brokers }}
{{- $externalListenerPort := add $root.Values.nodeport.firstListenerPort $i }}
{{- $responsiblePod := printf "%s-%d" (printf "%s" $fullName) $i }}
---
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka.fullname" $root }}-{{ $i }}-nodeport
labels:
app: {{ include "cp-kafka.name" $root }}
chart: {{ template "cp-kafka.chart" $root }}
release: {{ $root.Release.Name }}
heritage: {{ $root.Release.Service }}
pod: {{ $responsiblePod }}
spec:
type: NodePort
ports:
- name: external-broker
port: {{ $servicePort }}
targetPort: {{ $externalListenerPort }}
nodePort: {{ $externalListenerPort }}
protocol: TCP
selector:
app: {{ include "cp-kafka.name" $root }}
release: {{ $root.Release.Name }}
statefulset.kubernetes.io/pod-name: {{ $responsiblePod | quote }}
{{- end }}
{{- end }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-kafka.fullname" . }}
labels:
app: {{ template "cp-kafka.name" . }}
chart: {{ template "cp-kafka.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- port: 9092
name: broker
selector:
app: {{ template "cp-kafka.name" . }}
release: {{ .Release.Name }}
\ No newline at end of file
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "cp-kafka.fullname" . }}
labels:
app: {{ template "cp-kafka.name" . }}
chart: {{ template "cp-kafka.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
selector:
matchLabels:
app: {{ template "cp-kafka.name" . }}
release: {{ .Release.Name }}
serviceName: {{ template "cp-kafka.fullname" . }}-headless
podManagementPolicy: {{ .Values.podManagementPolicy }}
replicas: {{ default 3 .Values.brokers }}
updateStrategy:
type: {{ .Values.updateStrategy }}
template:
metadata:
labels:
app: {{ template "cp-kafka.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- {{ template "cp-kafka.name" . }}
- key: "release"
operator: In
values:
- {{ .Release.Name }}
topologyKey: "kubernetes.io/hostname"
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-kafka/jmx-kafka-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 10 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-kafka
{{- end }}
- name: {{ template "cp-kafka.name" . }}-broker
{{- if .Values.global.enterprise }}
image: "{{ .Values.enterpriseImage }}:{{ .Values.imageTag }}"
{{- else }}
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
{{- end }}
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- containerPort: 9092
name: kafka
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
{{- if .Values.nodeport.enabled }}
{{- $brokers := .Values.brokers | int }}
{{- $root := . }}
{{- range $i, $e := until $brokers }}
- containerPort: {{ add $root.Values.nodeport.firstListenerPort $i }}
name: nodeport-{{ $i }}
{{- end }}
{{- end }}
resources:
{{ toYaml .Values.resources | indent 10 }}
env:
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: KAFKA_HEAP_OPTS
value: {{ .Values.heapOptions }}
- name: KAFKA_ZOOKEEPER_CONNECT
value: {{ include "cp-kafka.cp-zookeeper.service-name" . | quote }}
- name: KAFKA_LOG_DIRS
value: {{ include "cp-kafka.log.dirs" . | quote }}
- name: KAFKA_METRIC_REPORTERS
value: "io.confluent.metrics.reporter.ConfluentMetricsReporter"
- name: CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS
value: {{ printf "PLAINTEXT://%s:9092" (include "cp-kafka.cp-kafka-headless.fullname" .) | quote }}
{{- range $key, $value := .Values.configurationOverrides }}
- name: {{ printf "KAFKA_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- range $key, $value := .Values.customEnv }}
- name: {{ $key | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.jmx.port }}
- name: KAFKA_JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
# This is required because the Downward API does not yet support identification of
# pod numbering in statefulsets. Thus, we are required to specify a command which
# allows us to extract the pod ID for usage as the Kafka Broker ID.
# See: https://github.com/kubernetes/kubernetes/issues/31218
command:
- sh
- -exc
- |
export KAFKA_BROKER_ID=${HOSTNAME##*-} && \
export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${POD_NAME}.{{ template "cp-kafka.fullname" . }}-headless.${POD_NAMESPACE}:9092{{ include "cp-kafka.configuration.advertised.listeners" . }} && \
exec /etc/confluent/docker/run
volumeMounts:
{{- $disksPerBroker := .Values.persistence.disksPerBroker | int }}
{{- range $k, $e := until $disksPerBroker }}
- name: datadir-{{$k}}
mountPath: /opt/kafka/data-{{$k}}
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{- if not .Values.persistence.enabled }}
- name: datadir-0
emptyDir: {}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-kafka.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.persistence.enabled }}
volumeClaimTemplates:
{{- $disksPerBroker := .Values.persistence.disksPerBroker | int }}
{{- $root := . }}
{{- range $k, $e := until $disksPerBroker }}
- metadata:
name: datadir-{{$k}}
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "{{ $root.Values.persistence.size }}"
{{- if $root.Values.persistence.storageClass }}
{{- if (eq "-" $root.Values.persistence.storageClass) }}
storageClassName: ""
{{- else }}
storageClassName: "{{ $root.Values.persistence.storageClass }}"
{{- end }}
{{- end }}
{{- end }}
{{- end }}
apiVersion: v1
kind: Pod
metadata:
name: "{{ .Release.Name }}-canary"
annotations:
"helm.sh/hook": test-success
"helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded"
spec:
containers:
- name: {{ .Release.Name }}-canary
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
command:
- sh
- -c
- |
# Delete the topic if it exists
kafka-topics --zookeeper {{ template "cp-kafka.cp-zookeeper.service-name" . }} --topic {{ template "cp-kafka.fullname" . }}-canary-topic --delete --if-exists
# Create the topic
kafka-topics --zookeeper {{ template "cp-kafka.cp-zookeeper.service-name" . }} --topic {{ template "cp-kafka.fullname" . }}-canary-topic --create --partitions 1 --replication-factor 1 --if-not-exists && \
# Create a message
MESSAGE="`date -u`" && \
# Produce a test message to the topic
echo "$MESSAGE" | kafka-console-producer --broker-list {{ template "cp-kafka.fullname" . }}:9092 --topic {{ template "cp-kafka.fullname" . }}-canary-topic && \
# Consume a test message from the topic
kafka-console-consumer --bootstrap-server {{ template "cp-kafka.fullname" . }}-headless:9092 --topic {{ template "cp-kafka.fullname" . }}-canary-topic --from-beginning --timeout-ms 2000 | grep "$MESSAGE"
restartPolicy: Never
\ No newline at end of file
# Default values for cp-kafka.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
## Number of Kafka brokers
brokers: 3
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-kafka/
image: confluentinc/cp-kafka
enterpriseImage: confluentinc/cp-enterprise-kafka
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
## StatefulSet Config
## Start and stop pods in Parallel or OrderedReady (one-by-one.)
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
podManagementPolicy: OrderedReady
## The StatefulSet Update Strategy which Kafka will use when changes are applied: OnDelete or RollingUpdate
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy: RollingUpdate
## Kafka Server properties
## ref: https://kafka.apache.org/documentation/#configuration
configurationOverrides:
"offsets.topic.replication.factor": "3"
# "default.replication.factor": 3
# "min.insync.replicas": 2
# "auto.create.topics.enable": false
## Options required for external access via NodePort
## ref:
## - http://kafka.apache.org/documentation/#security_configbroker
## - https://cwiki.apache.org/confluence/display/KAFKA/KIP-103%3A+Separation+of+Internal+and+External+traffic
##
## Advertised listeners will use the firstListenerPort value as it's default unless overridden here.
## Setting "advertised.listeners" here appends to "PLAINTEXT://${POD_IP}:9092,"
# "advertised.listeners": |-
# EXTERNAL://${HOST_IP}:$((31090 + ${KAFKA_BROKER_ID}))
"listener.security.protocol.map": |-
PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
## Additional env variables
customEnv: {}
# KAFKA_METRIC_REPORTERS: "io.confluent.metrics.reporter.ConfluentMetricsReporter"
# CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: "localhost:9092"
persistence:
enabled: true
## The size of the PersistentVolume to allocate to each Kafka Pod in the StatefulSet. For
## production servers this number should likely be much larger.
size: 1Gi
## Kafka data Persistent Volume Storage Class
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
## set, choosing the default provisioner. (gp2 on AWS, standard on
## GKE, AWS & OpenStack)
##
# storageClass: ""
disksPerBroker: 1
## Kafka JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## Kafka JMX Settings
## ref: https://docs.confluent.io/current/kafka/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
nodeport:
enabled: false
servicePort: 19092
firstListenerPort: 31090
## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
cp-zookeeper:
## If true, install the cp-zookeeper chart alongside cp-kafka
## ref: ../cp-zookeeper
enabled: true
servers: 3
persistence:
enabled: true
dataDirSize: 5Gi
dataLogDirSize: 5Gi
## If the Zookeeper Chart is disabled a URL and port are required to connect
url: ""
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent KSQL Server on Kubernetes
name: cp-ksql-server
version: 0.1.0
# KSQL Server Helm Chart
This chart bootstraps a deployment of a Confluent KSQL Server.
This is an example deployment which runs KSQL Server in non-interactive
mode.
The included queries file `queries.sql` is a stub provided to illustrate one possible approach to mounting queries in the server container via ConfigMap.
## Prerequisites
* Kubernetes 1.9.2+
* Helm 2.8.2+
* A healthy and accessible Kafka Cluster
## Developing Environment
* [Pivotal Container Service (PKS)](https://pivotal.io/platform/pivotal-container-service)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/)
## Docker Image Source
* [DockerHub -> ConfluentInc](https://hub.docker.com/u/confluentinc/)
## Installing the Chart
### Install along with cp-helm-charts
```console
git clone https://github.com/confluentinc/cp-helm-charts.git
helm install cp-helm-charts
```
To install with a specific name, you can do:
```console
helm install --name my-confluent cp-helm-charts
```
### Install with a existing cp-kafka and cp-schema-registry release
```console
helm install --set cp-zookeeper.url="unhinged-robin-cp-zookeeper:2181",cp-schema-registry.url="lolling-chinchilla-cp-schema-registry:8081" cp-helm-charts/charts/cp-ksql-server
```
### Installed Components
You can use `helm status <release name>` to view all of the installed components.
For example:
```console
$ helm status excited-lynx
STATUS: DEPLOYED
RESOURCES:
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
excited-lynx-cp-ksql-server ClusterIP 10.31.253.70 <none> 8088/TCP 10s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
excited-lynx-cp-ksql-server 1 1 1 0 10s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
excited-lynx-cp-ksql-server-d4848ff94-x5fmn 2/2 Running 1 10s
==> v1/ConfigMap
NAME DATA AGE
excited-lynx-cp-ksql-server-jmx-configmap 1 10s
excited-lynx-cp-ksql-server-ksql-queries-configmap 1 10s
NOTES:
This chart installs Confluent KSQL Server.
https://docs.confluent.io/current/ksql/docs
```
There are
1. A [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) `excited-lynx-cp-ksql-server` which contains 1 KSQL Server instance [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/): `excited-lynx-cp-ksql-server-d4848ff94-x5fmn`.
1. A [Service](https://kubernetes.io/docs/concepts/services-networking/service/) `excited-lynx-cp-ksql-server` for clients to connect to KSQL Server.
1. A [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) which contains configuration for Prometheus JMX Exporter.
1. A [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) which contains SQL queries for the server to run in non-interactive mode.
## Configuration
You can specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```console
helm install --name my-ksql-server -f my-values.yaml ./cp-ksql-server
```
> **Tip**: A default [values.yaml](values.yaml) is provided
### KSQL Server Deployment
The configuration parameters in this section control the resources requested and utilized by the cp-ksql-server chart.
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `replicaCount` | The number of KSQL Server instances. | `1` |
### Image
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `image` | Docker Image of Confluent KSQL Server. | `confluentinc/cp-ksql-server` |
| `imageTag` | Docker Image Tag of Confluent KSQL Server. | `5.3.0` |
| `imagePullPolicy` | Docker Image Tag of Confluent KSQL Server. | `IfNotPresent` |
| `imagePullSecrets` | Secrets to be used for private registries. | see [values.yaml](values.yaml) for details |
### KSQL Configuration
Parameter | Description | Default |
| --------- | ----------- | ------- |
| `configurationOverrides` | KSQL [configuration](https://docs.confluent.io/current/ksql/docs/installation/server-config/config-reference.html) overrides in the dictionary format | `{}` |
### Port
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `servicePort` | The port on which the KSQL Server will be available and serving requests. | `8088` |
### KSQL JVM Heap Options
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `heapOptions` | The JVM Heap Options for KSQL | `"-Xms512M -Xmx512M"` |
### Resources
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `resources.requests.cpu` | The amount of CPU to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.memory` | The amount of memory to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit CPU usage for a KSQL Server Pod. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit memory usage for a KSQL Server Pod. | see [values.yaml](values.yaml) for details |
### Annotations
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `podAnnotations` | Map of custom annotations to attach to the pod spec. | `{}` |
### JMX Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `jmx.port` | The jmx port which JMX style metrics are exposed. | `5555` |
### Prometheus JMX Exporter Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `prometheus.jmx.enabled` | Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. | `true` |
| `prometheus.jmx.image` | Docker Image for Prometheus JMX Exporter container. | `solsson/kafka-prometheus-jmx-exporter@sha256` |
| `prometheus.jmx.imageTag` | Docker Image Tag for Prometheus JMX Exporter container. | `6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143` |
| `prometheus.jmx.imagePullPolicy` | Docker Image Pull Policy for Prometheus JMX Exporter container. | `IfNotPresent` |
| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping. | `5556` |
| `prometheus.jmx.resources` | JMX Exporter resources configuration. | see [values.yaml](values.yaml) for details |
### External Access
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `external.enabled` | whether or not to allow external access to KSQL Server | `false` |
| `external.type` | `Kubernetes Service Type` to expose KSQL Server to external | `LoadBalancer` |
| `external.port` | External service port to expose KSQL Server to external | `8082` |
| `external.annotations` | Map of annotations to attach to external KSQL Server service | `nil` |
| `external.externalTrafficPolicy` | Configures `.spec.externalTrafficPolicy` which controls if load balancing occurs across all nodes (value of `Cluster`) or only active nodes (value of `Local`) | `Cluster` |
| `external.loadBalancerSourceRanges` | Configures `.spec.loadBalancerSourceRanges` which specifies the list of source IP ranges permitted access to the load balancer | `["0.0.0.0/0"]` |
### Deployment Topology
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `nodeSelector` | Dictionary containing key-value-pairs to match labels on nodes. When defined pods will only be scheduled on nodes, that have each of the indicated key-value pairs as labels. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) | `{}`
| `tolerations`| Array containing taint references. When defined, pods can run on nodes, which would otherwise deny scheduling. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) | `{}`
## Dependencies
### Schema Registry (optional)
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `cp-schema-registry.url` | Service name of Schema Registry (Not needed if this is installed along with cp-kafka chart). | `""` |
| `cp-schema-registry.port` | Port of Schema Registry Service | `8081` |
-- From http://docs.confluent.io/current/ksql/docs/tutorials/basics-docker.html#create-a-stream-and-table
-- Create a stream pageviews_original from the Kafka topic pageviews, specifying the value_format of DELIMITED
CREATE STREAM pageviews_original (viewtime bigint, userid varchar, pageid varchar) WITH (kafka_topic='pageviews', value_format='DELIMITED');
-- Create a table users_original from the Kafka topic users, specifying the value_format of JSON
CREATE TABLE users_original (registertime BIGINT, gender VARCHAR, regionid VARCHAR, userid VARCHAR) WITH (kafka_topic='users', value_format='JSON', key = 'userid');
-- Create a persistent query by using the CREATE STREAM keywords to precede the SELECT statement
CREATE STREAM pageviews_enriched AS SELECT users_original.userid AS userid, pageid, regionid, gender FROM pageviews_original LEFT JOIN users_original ON pageviews_original.userid = users_original.userid;
-- Create a new persistent query where a condition limits the streams content, using WHERE
CREATE STREAM pageviews_female AS SELECT * FROM pageviews_enriched WHERE gender = 'FEMALE';
-- Create a new persistent query where another condition is met, using LIKE
CREATE STREAM pageviews_female_like_89 WITH (kafka_topic='pageviews_enriched_r8_r9') AS SELECT * FROM pageviews_female WHERE regionid LIKE '%_8' OR regionid LIKE '%_9';
-- Create a new persistent query that counts the pageviews for each region and gender combination in a tumbling window of 30 seconds when the count is greater than one
CREATE TABLE pageviews_regions WITH (VALUE_FORMAT='avro') AS SELECT gender, regionid , COUNT(*) AS numusers FROM pageviews_enriched WINDOW TUMBLING (size 30 second) GROUP BY gender, regionid HAVING COUNT(*) > 1;
\ No newline at end of file
This chart installs Confluent KSQL Server.
https://docs.confluent.io/current/ksql/docs
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-ksql-server.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-ksql-server.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-ksql-server.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified kafka headless name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-ksql-server.cp-kafka-headless.fullname" -}}
{{- $name := "cp-kafka-headless" -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Kafka URL. If Kafka is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-ksql-server.kafka.bootstrapServers" -}}
{{- if .Values.kafka.bootstrapServers -}}
{{- .Values.kafka.bootstrapServers -}}
{{- else -}}
{{- printf "PLAINTEXT://%s:9092" (include "cp-ksql-server.cp-kafka-headless.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Default Server Pool Id to Release Name but allow it to be overridden
*/}}
{{- define "cp-ksql-server.serviceId" -}}
{{- if .Values.overrideServiceId -}}
{{- .Values.overrideServiceId -}}
{{- else -}}
{{- .Release.Name -}}
{{- end -}}
{{- end -}}
{{/*
Create a default fully qualified schema registry name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-ksql-server.cp-schema-registry.fullname" -}}
{{- $name := default "cp-schema-registry" (index .Values "cp-schema-registry" "nameOverride") -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "cp-ksql-server.cp-schema-registry.service-name" -}}
{{- if (index .Values "cp-schema-registry" "url") -}}
{{- printf "%s" (index .Values "cp-schema-registry" "url") -}}
{{- else -}}
{{- printf "http://%s:8081" (include "cp-ksql-server.cp-schema-registry.fullname" .) -}}
{{- end -}}
{{- end -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "cp-ksql-server.fullname" . }}
labels:
app: {{ template "cp-ksql-server.name" . }}
chart: {{ template "cp-ksql-server.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "cp-ksql-server.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "cp-ksql-server.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-ksql-server/jmx-ksql-server-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 12 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-ksql-server
{{- end }}
- name: {{ template "cp-ksql-server.name" . }}
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- name: server
containerPort: {{ .Values.servicePort}}
protocol: TCP
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
resources:
{{ toYaml .Values.resources | indent 12 }}
volumeMounts:
{{- if .Values.ksql.headless }}
- name: ksql-queries
mountPath: /etc/ksql/queries
{{- end }}
env:
- name: KSQL_BOOTSTRAP_SERVERS
value: {{ template "cp-ksql-server.kafka.bootstrapServers" . }}
- name: KSQL_KSQL_SERVICE_ID
value: {{ template "cp-ksql-server.serviceId" . }}
- name: KSQL_KSQL_SCHEMA_REGISTRY_URL
value: {{ template "cp-ksql-server.cp-schema-registry.service-name" . }}
- name: KSQL_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{- if .Values.ksql.headless }}
- name: KSQL_KSQL_QUERIES_FILE
value: /etc/ksql/queries/queries.sql
{{- else }}
- name: KSQL_LISTENERS
value: http://0.0.0.0:8088
{{- end }}
{{- range $key, $value := .Values.configurationOverrides }}
- name: {{ printf "KSQL_%s" $key | replace "." "_" | upper | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.jmx.port }}
- name: KSQL_JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-ksql-server.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.ksql.headless }}
- name: ksql-queries
configMap:
name: {{ template "cp-ksql-server.fullname" . }}-ksql-queries-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.external.enabled -}}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-ksql-server.fullname" . }}-external
{{- if .Values.external.annotations }}
annotations:
{{- range $key, $value := .Values.external.annotations }}
{{ $key }}: "{{ $value }}"
{{- end }}
{{- end }}
labels:
app: {{ template "cp-ksql-server.name" . }}
chart: {{ template "cp-ksql-server.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
type: {{ .Values.external.type }}
externalTrafficPolicy: {{ .Values.external.externalTrafficPolicy }}
{{- if .Values.external.loadBalancerSourceRanges }}
loadBalancerSourceRanges:
{{- range $cidr := .Values.external.loadBalancerSourceRanges }}
- {{ $cidr }}
{{- end }}
{{- end }}
ports:
- name: server-proxy
port: {{ .Values.external.port }}
targetPort: {{ .Values.servicePort }}
selector:
app: {{ template "cp-ksql-server.name" . }}
release: {{ .Release.Name }}
{{- end -}}
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-ksql-server.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-ksql-server.name" . }}
chart: {{ template "cp-ksql-server.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-ksql-server-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern : 'io.confluent.ksql.metrics<type=ksql-engine-query-stats>([^:]+):'
name: "cp_ksql_server_metrics_$1"
{{- end }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-ksql-server.fullname" . }}-ksql-queries-configmap
labels:
app: {{ template "cp-ksql-server.name" . }}
chart: {{ template "cp-ksql-server.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
{{- $files := .Files }}
{{- range tuple "queries.sql" }}
{{ . }}: |-
{{ $files.Get . | indent 4 }}
{{- end }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-ksql-server.fullname" . }}
labels:
app: {{ template "cp-ksql-server.name" . }}
chart: {{ template "cp-ksql-server.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- name: ksql-server
port: {{ .Values.servicePort }}
selector:
app: {{ template "cp-ksql-server.name" . }}
release: {{ .Release.Name }}
# Default values for cp-ksql-server.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-ksql-server/
image: confluentinc/cp-ksql-server
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
servicePort: 8088
## KSQL JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## JMX Settings
## ref: https://docs.confluent.io/current/ksql/docs/operations.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
## External Access
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-loadbalancer
external:
enabled: false
type: LoadBalancer
externalTrafficPolicy: Cluster
port: 8088
## Headless mode
## ref: https://docs.confluent.io/current/ksql/docs/installation/server-config/index.html
ksql:
headless: true
## You can list load balanced service endpoint, or list of all brokers (which is hard in K8s). e.g.:
## bootstrapServers: "PLAINTEXT://dozing-prawn-kafka-headless:9092"
kafka:
bootstrapServers: ""
## e.g. gnoble-panther-cp-schema-registry:8081
cp-schema-registry:
url: ""
# KSQL configuration options
## ref: https://docs.confluent.io/current/ksql/docs/installation/server-config/config-reference.html
configurationOverrides: {}
# "ksql.streams.producer.retries": "2147483647"
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent Schema Registry on Kubernetes
name: cp-schema-registry
version: 0.1.0
\ No newline at end of file
# CP-Schema Registry Helm Chart
This chart bootstraps a deployment of a Confluent Schema Registry
## Prerequisites
* Kubernetes 1.9.2+
* Helm 2.8.2+
* A healthy and accessible Kafka Cluster
## Developing Environment
* [Pivotal Container Service (PKS)](https://pivotal.io/platform/pivotal-container-service)
* [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/)
## Docker Image Source
* [DockerHub -> ConfluentInc](https://hub.docker.com/u/confluentinc/)
## Installing the Chart
### Install along with cp-helm-charts
```console
git clone https://github.com/confluentinc/cp-helm-charts.git
helm install cp-helm-charts
```
To install with a specific name, you can do:
```console
helm install --name my-confluent cp-helm-charts
```
### Install with a existing cp-kafka release
```console
helm install --set kafka.bootstrapServers="PLAINTEXT://unhinged-robin-cp-kafka-headless:9092" cp-helm-charts/charts/cp-schema-registry
```
### Installed Components
You can use `helm status <release name>` to view all of the installed components.
For example:
```console{%raw}
$ helm status lolling-chinchilla
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
lolling-chinchilla-cp-schema-registry 1 1 1 0 1s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
lolling-chinchilla-cp-schema-registry-58f854bd47-jxrcj 0/1 ContainerCreating 0 1s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
lolling-chinchilla-cp-schema-registry ClusterIP 10.19.245.11 <none> 8081/TCP 1s
==> v1/ConfigMap
NAME DATA AGE
lolling-chinchilla-cp-schema-registry-jmx-configmap 1 1s
```
There are
1. A [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) `lolling-chinchilla-cp-schema-registry` which contains 1 Schema Registry [Pod](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/): `lolling-chinchilla-cp-schema-registry-58f854bd47-jxrcj`.
1. A [Service](https://kubernetes.io/docs/concepts/services-networking/service/) `lolling-chinchilla-cp-schema-registry` for clients to connect to Schema Registry.
1. A [ConfigMap](https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/) which contains configuration for Prometheus JMX Exporter.
## Configuration
You can specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
```console
helm install --name my-schema-registry -f my-values.yaml ./cp-schema-registry
```
> **Tip**: A default [values.yaml](values.yaml) is provided
### Schema Registry Deployment
The configuration parameters in this section control the resources requested and utilized by the cp-schema-registry chart.
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `replicaCount` | The number of Schema Registry Servers. | `1` |
### Image
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `image` | Docker Image of Confluent Schema Registry. | `confluentinc/cp-schema-registry` |
| `imageTag` | Docker Image Tag of Confluent Schema Registry. | `5.3.0` |
| `imagePullPolicy` | Docker Image Tag of Confluent Schema Registry. | `IfNotPresent` |
| `imagePullSecrets` | Secrets to be used for private registries. | see [values.yaml](values.yaml) for details |
### Confluent Schema Registry Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `configurationOverrides` | Schema Registry [configuration](https://docs.confluent.io/current/schema-registry/docs/config.html) overrides in the dictionary format. | `{}` |
| `customEnv` | Custom environmental variables | `{}` |
### Port
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `servicePort` | The port on which the Schema Registry will be available and serving requests. | `8081` |
### Kafka
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `kafka.bootstrapServers` | Bootstrap Servers for Schema Registry | `""` |
### Confluent Schema Registry JVM Heap Options
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `heapOptions` | The JVM Heap Options for Confluent Schema Registry | `"-Xms512M -Xmx512M"` |
### Resources
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `resources.requests.cpu` | The amount of CPU to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.memory` | The amount of memory to request. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit CPU usage for a Schema Registry Pod. | see [values.yaml](values.yaml) for details |
| `resources.requests.limit` | The upper limit memory usage for a Schema Registry Pod. | see [values.yaml](values.yaml) for details |
### Annotations
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `podAnnotations` | Map of custom annotations to attach to the pod spec. | `{}` |
### JMX Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `jmx.port` | The jmx port which JMX style metrics are exposed. | `5555` |
### Prometheus JMX Exporter Configuration
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `prometheus.jmx.enabled` | Whether or not to install Prometheus JMX Exporter as a sidecar container and expose JMX metrics to Prometheus. | `true` |
| `prometheus.jmx.image` | Docker Image for Prometheus JMX Exporter container. | `solsson/kafka-prometheus-jmx-exporter@sha256` |
| `prometheus.jmx.imageTag` | Docker Image Tag for Prometheus JMX Exporter container. | `6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143` |
| `prometheus.jmx.imagePullPolicy` | Docker Image Pull Policy for Prometheus JMX Exporter container. | `IfNotPresent` |
| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping. | `5556` |
| `prometheus.jmx.resources` | JMX Exporter resources configuration. | see [values.yaml](values.yaml) for details |
### Deployment Topology
| Parameter | Description | Default |
| --------- | ----------- | ------- |
| `nodeSelector` | Dictionary containing key-value-pairs to match labels on nodes. When defined pods will only be scheduled on nodes, that have each of the indicated key-value pairs as labels. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/) | `{}`
| `tolerations`| Array containing taint references. When defined, pods can run on nodes, which would otherwise deny scheduling. Further information can be found in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) | `{}`
This chart installs a Confluent Kafka Schema Registry
https://github.com/confluentinc/schema-registry
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-schema-registry.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-schema-registry.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-schema-registry.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified kafka headless name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "cp-kafka-rest.cp-kafka-headless.fullname" -}}
{{- $name := "cp-kafka-headless" -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Form the Kafka URL. If Kafka is installed as part of this chart, use k8s service discovery,
else use user-provided URL
*/}}
{{- define "cp-schema-registry.kafka.bootstrapServers" -}}
{{- if .Values.kafka.bootstrapServers -}}
{{- .Values.kafka.bootstrapServers -}}
{{- else -}}
{{- printf "PLAINTEXT://%s:9092" (include "cp-kafka-rest.cp-kafka-headless.fullname" .) -}}
{{- end -}}
{{- end -}}
{{/*
Default GroupId to Release Name but allow it to be overridden
*/}}
{{- define "cp-schema-registry.groupId" -}}
{{- if .Values.overrideGroupId -}}
{{- .Values.overrideGroupId -}}
{{- else -}}
{{- .Release.Name -}}
{{- end -}}
{{- end -}}
\ No newline at end of file
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "cp-schema-registry.fullname" . }}
labels:
app: {{ template "cp-schema-registry.name" . }}
chart: {{ template "cp-schema-registry.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "cp-schema-registry.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "cp-schema-registry.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-schema-registry/jmx-schema-registry-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 12 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-schema-registry
{{- end }}
- name: {{ template "cp-schema-registry.name" . }}-server
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- name: schema-registry
containerPort: {{ .Values.servicePort }}
protocol: TCP
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
resources:
{{ toYaml .Values.resources | indent 12 }}
env:
- name: SCHEMA_REGISTRY_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: SCHEMA_REGISTRY_LISTENERS
value: http://0.0.0.0:{{ .Values.servicePort }}
- name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
value: {{ template "cp-schema-registry.kafka.bootstrapServers" . }}
- name: SCHEMA_REGISTRY_KAFKASTORE_GROUP_ID
value: {{ template "cp-schema-registry.groupId" . }}
- name: SCHEMA_REGISTRY_MASTER_ELIGIBILITY
value: "true"
- name: SCHEMA_REGISTRY_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{ range $configName, $configValue := .Values.configurationOverrides }}
- name: SCHEMA_REGISTRY_{{ $configName | replace "." "_" | upper }}
value: {{ $configValue | quote }}
{{ end }}
{{- range $key, $value := .Values.customEnv }}
- name: {{ $key | quote }}
value: {{ $value | quote }}
{{- end }}
{{- if .Values.schemaRegistryOpts }}
- name: SCHEMA_REGISTRY_OPTS
value: "{{ .Values.schemaRegistryOpts }}"
{{- end }}
{{- if .Values.jmx.port }}
- name: JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-schema-registry.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-schema-registry.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-schema-registry.name" . }}
chart: {{ template "cp-schema-registry.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-schema-registry-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern : 'kafka.schema.registry<type=jetty-metrics>([^:]+):'
name: "cp_kafka_schema_registry_jetty_metrics_$1"
- pattern : 'kafka.schema.registry<type=master-slave-role>([^:]+):'
name: "cp_kafka_schema_registry_master_slave_role"
- pattern : 'kafka.schema.registry<type=jersey-metrics>([^:]+):'
name: "cp_kafka_schema_registry_jersey_metrics_$1"
{{- end }}
\ No newline at end of file
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-schema-registry.fullname" . }}
labels:
app: {{ template "cp-schema-registry.name" . }}
chart: {{ template "cp-schema-registry.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- name: schema-registry
port: {{ .Values.servicePort }}
selector:
app: {{ template "cp-schema-registry.name" . }}
release: {{ .Release.Name }}
# Default values for cp-schema-registry.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
## ------------------------------------------------------
## Schema Registry
## ------------------------------------------------------
## Number of Scheme Registry Pod
replicaCount: 1
## Image Info
## ref: https://hub.docker.com/r/confluentinc/cp-schema-registry/
image: confluentinc/cp-schema-registry
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
## Schema Registry Settings Overrides
## Configuration Options can be found here: https://docs.confluent.io/current/schema-registry/docs/config.html
configurationOverrides: {}
## Additional env variables
customEnv: {}
## The port on which the Schema Registry will be available and serving requests
servicePort: 8081
## Schema registry JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
## You can list load balanced service endpoint, or list of all brokers (which is hard in K8s). e.g.:
## bootstrapServers: "PLAINTEXT://dozing-prawn-kafka-headless:9092"
## Charts uses Kafka Coordinator Master Election: https://docs.confluent.io/current/schema-registry/docs/design.html#kafka-coordinator-master-election
kafka:
bootstrapServers: ""
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## Schema Registry JMX Settings
## ref: https://docs.confluent.io/current/schema-registry/docs/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
resources: {}
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for Confluent Zookeeper on Kubernetes
name: cp-zookeeper
version: 0.1.0
\ No newline at end of file
This chart installs a {{ default 3 .Values.servers }} nodes Confluent Zookeeper ensemble for Confluent Kafka Cluster.
Connection string for Confluent Kafka:
{{ template "cp-zookeeper.fullname" . }}-0.{{ template "cp-zookeeper.fullname" . }}-headless:{{ .Values.clientPort }},{{ template "cp-zookeeper.fullname" . }}-1.{{ template "cp-zookeeper.fullname" . }}-headless:{{ .Values.clientPort }},...
To connect from a client pod:
1. Deploy a zookeeper client pod with configuration:
apiVersion: v1
kind: Pod
metadata:
name: zookeeper-client
namespace: {{ .Release.Namespace }}
spec:
containers:
- name: zookeeper-client
image: {{ .Values.image }}:{{ .Values.imageTag }}
command:
- sh
- -c
- "exec tail -f /dev/null"
2. Log into the Pod
kubectl exec -it zookeeper-client -- /bin/bash
3. Use zookeeper-shell to connect in the zookeeper-client Pod:
zookeeper-shell {{ template "cp-zookeeper.fullname" . }}:{{ .Values.clientPort }}
4. Explore with zookeeper commands:
https://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_ConnectingToZooKeeper
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-zookeeper.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-zookeeper.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-zookeeper.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a server list string based on fullname, namespace, # of servers
in a format like "zkhost1:port:port;zkhost2:port:port"
*/}}
{{- define "cp-zookeeper.serverlist" -}}
{{- $namespace := .Release.Namespace }}
{{- $name := include "cp-zookeeper.fullname" . -}}
{{- $serverPort := .Values.serverPort -}}
{{- $leaderElectionPort := .Values.leaderElectionPort -}}
{{- $zk := dict "servers" (list) -}}
{{- range $idx, $v := until (int .Values.servers) }}
{{- $noop := printf "%s-%d.%s-headless.%s:%d:%d" $name $idx $name $namespace (int $serverPort) (int $leaderElectionPort) | append $zk.servers | set $zk "servers" -}}
{{- end }}
{{- printf "%s" (join ";" $zk.servers) | quote -}}
{{- end -}}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-zookeeper.fullname" . }}-headless
labels:
app: {{ template "cp-zookeeper.name" . }}
chart: {{ template "cp-zookeeper.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- port: {{ .Values.serverPort }}
name: server
- port: {{ .Values.leaderElectionPort }}
name: leader-election
clusterIP: None
selector:
app: {{ template "cp-zookeeper.name" . }}
release: {{ .Release.Name }}
\ No newline at end of file
{{- if and .Values.prometheus.jmx.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "cp-zookeeper.fullname" . }}-jmx-configmap
labels:
app: {{ template "cp-zookeeper.name" . }}
chart: {{ template "cp-zookeeper.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
jmx-zookeeper-prometheus.yml: |+
jmxUrl: service:jmx:rmi:///jndi/rmi://localhost:{{ .Values.jmx.port }}/jmxrmi
lowercaseOutputName: true
lowercaseOutputLabelNames: true
ssl: false
rules:
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+)><>(\\w+)"
name: "cp_zookeeper_$2"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+)><>(\\w+)"
name: "cp_zookeeper_$3"
labels:
replicaId: "$2"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+)><>(\\w+)"
name: "cp_zookeeper_$4"
labels:
replicaId: "$2"
memberType: "$3"
- pattern: "org.apache.ZooKeeperService<name0=ReplicatedServer_id(\\d+), name1=replica.(\\d+), name2=(\\w+), name3=(\\w+)><>(\\w+)"
name: "cp_zookeeper_$4_$5"
labels:
replicaId: "$2"
memberType: "$3"
{{- end }}
\ No newline at end of file
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: {{ template "cp-zookeeper.fullname" . }}-pdb
labels:
app: {{ template "cp-zookeeper.name" . }}
chart: {{ template "cp-zookeeper.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
selector:
matchLabels:
app: {{ template "cp-zookeeper.name" . }}
release: {{ .Release.Name }}
{{- if .Values.minAvailable }}
minAvailable: {{ .Values.minAvailable }}
{{- else }}
maxUnavailable: 1
{{- end }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "cp-zookeeper.fullname" . }}
labels:
app: {{ template "cp-zookeeper.name" . }}
chart: {{ template "cp-zookeeper.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
type: {{ .Values.serviceType }}
ports:
- port: {{ .Values.clientPort }}
name: client
selector:
app: {{ template "cp-zookeeper.name" . }}
release: {{ .Release.Name }}
\ No newline at end of file
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "cp-zookeeper.fullname" . }}
labels:
app: {{ template "cp-zookeeper.name" . }}
chart: {{ template "cp-zookeeper.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
selector:
matchLabels:
app: {{ template "cp-zookeeper.name" . }}
release: {{ .Release.Name }}
serviceName: {{ template "cp-zookeeper.fullname" . }}-headless
podManagementPolicy: {{ .Values.podManagementPolicy }}
replicas: {{ default 3 .Values.servers }}
updateStrategy:
type: {{ .Values.updateStrategy }}
template:
metadata:
labels:
app: {{ template "cp-zookeeper.name" . }}
release: {{ .Release.Name }}
{{- if or .Values.podAnnotations .Values.prometheus.jmx.enabled }}
annotations:
{{- range $key, $value := .Values.podAnnotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
prometheus.io/scrape: "true"
prometheus.io/port: {{ .Values.prometheus.jmx.port | quote }}
{{- end }}
{{- end }}
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
podAffinityTerm:
labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- {{ template "cp-zookeeper.name" . }}
- key: "release"
operator: In
values:
- {{ .Release.Name }}
topologyKey: "kubernetes.io/hostname"
containers:
{{- if .Values.prometheus.jmx.enabled }}
- name: prometheus-jmx-exporter
image: "{{ .Values.prometheus.jmx.image }}:{{ .Values.prometheus.jmx.imageTag }}"
imagePullPolicy: "{{ .Values.prometheus.jmx.imagePullPolicy }}"
command:
- java
- -XX:+UnlockExperimentalVMOptions
- -XX:+UseCGroupMemoryLimitForHeap
- -XX:MaxRAMFraction=1
- -XshowSettings:vm
- -jar
- jmx_prometheus_httpserver.jar
- {{ .Values.prometheus.jmx.port | quote }}
- /etc/jmx-zookeeper/jmx-zookeeper-prometheus.yml
ports:
- containerPort: {{ .Values.prometheus.jmx.port }}
resources:
{{ toYaml .Values.prometheus.jmx.resources | indent 10 }}
volumeMounts:
- name: jmx-config
mountPath: /etc/jmx-zookeeper
{{- end }}
- name: {{ template "cp-zookeeper.name" . }}-server
image: "{{ .Values.image }}:{{ .Values.imageTag }}"
imagePullPolicy: "{{ .Values.imagePullPolicy }}"
ports:
- containerPort: {{ .Values.clientPort }}
name: client
- containerPort: {{ .Values.serverPort }}
name: server
- containerPort: {{ .Values.leaderElectionPort }}
name: leader-election
{{- if .Values.prometheus.jmx.enabled }}
- containerPort: {{ .Values.jmx.port }}
name: jmx
{{- end }}
resources:
{{ toYaml .Values.resources | indent 10 }}
env:
- name : KAFKA_HEAP_OPTS
value: "{{ .Values.heapOptions }}"
{{- if .Values.jmx.port }}
- name : KAFKA_JMX_PORT
value: "{{ .Values.jmx.port }}"
{{- end }}
- name : ZOOKEEPER_TICK_TIME
value: "{{ .Values.tickTime }}"
- name : ZOOKEEPER_SYNC_LIMIT
value: "{{ .Values.syncLimit }}"
- name : ZOOKEEPER_INIT_LIMIT
value: "{{ .Values.initLimit }}"
- name : ZOOKEEPER_MAX_CLIENT_CNXNS
value: "{{ .Values.maxClientCnxns }}"
- name : ZOOKEEPER_AUTOPURGE_SNAP_RETAIN_COUNT
value: "{{ .Values.autoPurgeSnapRetainCount}}"
- name : ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL
value: "{{ .Values.autoPurgePurgeInterval}}"
- name: ZOOKEEPER_CLIENT_PORT
value: "{{ .Values.clientPort }}"
- name : ZOOKEEPER_SERVERS
value: {{ template "cp-zookeeper.serverlist" . }}
# ZOOKEEPER_SERVER_ID is required just to pass cp-zookeeper ensure script for env check,
# the value(metadata.mame) is not used and will be overwritten in command part
- name: ZOOKEEPER_SERVER_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
command:
- "bash"
- "-c"
- |
ZK_FIX_HOST_REGEX="s/${HOSTNAME}\.[^:]*:/0.0.0.0:/g"
ZOOKEEPER_SERVER_ID=$((${HOSTNAME##*-}+1)) \
ZOOKEEPER_SERVERS=`echo $ZOOKEEPER_SERVERS | sed -e "$ZK_FIX_HOST_REGEX"` \
/etc/confluent/docker/run
volumeMounts:
- name: datadir
mountPath: /var/lib/zookeeper/data
- name: datalogdir
mountPath: /var/lib/zookeeper/log
{{- if .Values.imagePullSecrets }}
imagePullSecrets:
{{ toYaml .Values.imagePullSecrets | indent 8 }}
{{- end }}
volumes:
{{ if not .Values.persistence.enabled }}
- name: datadir
emptyDir: {}
- name: datalogdir
emptyDir: {}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
- name: jmx-config
configMap:
name: {{ template "cp-zookeeper.fullname" . }}-jmx-configmap
{{- end }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
{{- end }}
{{- if .Values.persistence.enabled }}
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "{{ .Values.persistence.dataDirSize }}"
{{- if .Values.persistence.dataDirStorageClass }}
{{- if (eq "-" .Values.persistence.dataDirStorageClass) }}
storageClassName: ""
{{- else }}
storageClassName: "{{ .Values.persistence.dataDirStorageClass }}"
{{- end }}
{{- end }}
- metadata:
name: datalogdir
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: "{{ .Values.persistence.dataLogDirSize }}"
{{- if .Values.persistence.dataLogDirStorageClass }}
{{- if (eq "-" .Values.persistence.dataLogDirStorageClass) }}
storageClassName: ""
{{- else }}
storageClassName: "{{ .Values.persistence.dataLogDirStorageClass }}"
{{- end }}
{{- end }}
{{- end }}
## Default values for cp-zookeeper.
## This is a YAML-formatted file.
## Declare variables to be passed into your templates.
## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
## Number of zookeeper servers, should be odd number
servers: 3
## Minimum number of servers available to ensure the availability of zookeeper service
## minAvailable: 2
## Images Info
## ref: https://hub.docker.com/r/confluentinc/cp-zookeeper/
image: confluentinc/cp-zookeeper
imageTag: 5.3.0
## Specify a imagePullPolicy
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
imagePullPolicy: IfNotPresent
## Specify an array of imagePullSecrets.
## Secrets must be manually created in the namespace.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
## StatefulSet Config
## Start and stop pods in Parallel or OrderedReady (one-by-one.)
## ref: https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/#pod-management-policy
podManagementPolicy: OrderedReady
## The StatefulSet Update Strategy which Zookeeper will use when changes are applied: OnDelete or RollingUpdate
## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
updateStrategy: RollingUpdate
## Zookeeper Configuration
## ref: https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_configuration
## ref: https://docs.confluent.io/current/zookeeper/deployment.html#important-configuration-options
tickTime: 2000
syncLimit: 5
initLimit: 10
maxClientCnxns: 60
autoPurgeSnapRetainCount: 3
autoPurgePurgeInterval: 24
## Zookeeper JVM Heap Option
heapOptions: "-Xms512M -Xmx512M"
## Port
## ref: https://zookeeper.apache.org/doc/r3.4.10/zookeeperAdmin.html#sc_configuration
serverPort: 2888
leaderElectionPort: 3888
clientPort: 2181
## Persistent Volumes
## ref: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
persistence:
enabled: true
## The size of the PersistentVolume to allocate to each Zookeeper Pod in the StatefulSet. For
## production servers this number should likely be much larger.
##
## Size for Data dir, where ZooKeeper will store the in-memory database snapshots.
dataDirSize: 5Gi
# dataDirStorageClass: ""
## Size for data log dir, which is a dedicated log device to be used, and helps avoid competition between logging and snaphots.
dataLogDirSize: 5Gi
# dataLogDirStorageClass: ""
## Pod Compute Resources
## ref: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## Custom pod annotations
podAnnotations: {}
## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeSelector: {}
## Taints to tolerate on node assignment:
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: {}
## Monitoring
## Zookeeper JMX Settings
## ref: https://docs.confluent.io/current/installation/docker/docs/operations/monitoring.html
jmx:
port: 5555
## Prometheus Exporter Configuration
## ref: https://prometheus.io/docs/instrumenting/exporters/
prometheus:
## JMX Exporter Configuration
## ref: https://github.com/prometheus/jmx_exporter
jmx:
enabled: true
image: solsson/kafka-prometheus-jmx-exporter@sha256
imageTag: 6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
imagePullPolicy: IfNotPresent
port: 5556
## Resources configuration for the JMX exporter container.
## See the `resources` documentation above for details.
resources: {}
# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj
apiVersion: v1
description: A Helm chart for Kafka Topics UI
name: kafka-topics-ui
version: 0.1.0
sources:
- https://github.com/Landoop/kafka-topics-ui
Business Source License 1.0
Licensor: Landoop Ltd
Software: Landoop Apache Kafka tools. The Software is © 2016 Landoop
Use Limitation: Usage of the software is free when your application uses the Software with a total of less than five Kafka server instances for production purposes.
Change Date: 2019-01-01
Change License: Apache-2.0 license.
_____________________________________________
You are granted limited license to the Software under this Business Source License. Please read this Business Source License carefully, particularly the Use Limitation set forth above.
Subject to the Use Limitation, Licensor grants you a non-exclusive, worldwide (subject to applicable laws) license to copy, modify, display, use, create derivative works, and redistribute the Software until the Change Date. If your use of the Software exceeds, or will exceed, the foregoing limitations you MUST obtain alternative licensing terms for the Software directly from Licensor. For the avoidance of doubt, prior to the Change Date, there is no Use Limitations for non-production purposes.
After the Change Date, this Business Source License will convert to the Change License and your use of the Software, including modified versions of the Software, will be governed by such Change License.
All copies of original and modified Software, and derivative works of the Software, are subject to this Business Source License. This Business Source License applies separately for each version of the Software and the Change Date will vary for each version of the Software released by Licensor.
You must conspicuously display this Business Source License on each original or modified copy of the Software. If you receive the Software in original or modified form from a third party, the restrictions set forth in this Business Source License apply to your use of such Software.
Any use of the Software in violation of this Business Source License will automatically terminate your rights under this Business Source License for the current and all future versions of the Software.
You may not use the marks or logos of Licensor or its affiliates for commercial purposes without prior written consent from Licensor.
TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE SOFTWARE AND ALL SERVICES PROVIDED BY LICENSOR OR ITS AFFILIATES UNDER OR IN CONNECTION WITH WITH THIS BUSINESS SOURCE LICENSE ARE PROVIDED ON AN “AS IS” AND “AS AVAILABLE” BASIS. YOU EXPRESSLY WAIVE ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, TITLE, SYSTEM INTEGRATION, AND ACCURACY OF INFORMATIONAL CONTENT.
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range .Values.ingress.hosts }}
http://{{ . }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "kafka-topics-ui.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get svc -w {{ template "kafka-topics-ui.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "kafka-topics-ui.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo http://$SERVICE_IP:{{ .Values.service.externalPort }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "kafka-topics-ui.name" . }},release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl port-forward $POD_NAME 8080:{{ .Values.service.internalPort }}
{{- end }}
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "kafka-topics-ui.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "kafka-topics-ui.fullname" -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- define "kafka-rest-proxy.name" -}}
{{- $port := .Values.kafkaRest.port | toString -}}
{{- printf "http://%s-%s:%s" .Release.Name .Values.kafkaRest.name $port | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- if not .Values.global.enterprise }}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "kafka-topics-ui.fullname" . }}
labels:
app: {{ template "kafka-topics-ui.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
selector:
matchLabels:
app: {{ template "kafka-topics-ui.name" . }}
release: {{ .Release.Name }}
replicas: {{ .Values.replicas }}
template:
metadata:
labels:
app: {{ template "kafka-topics-ui.name" . }}
release: {{ .Release.Name }}
spec:
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: {{ .Values.service.internalPort }}
env:
- name: KAFKA_REST_PROXY_URL
value: {{ template "kafka-rest-proxy.name" . }}
- name: PROXY
value: "true"
resources:
{{ toYaml .Values.resources | indent 12 }}
{{- if .Values.nodeSelector }}
nodeSelector:
{{ toYaml .Values.nodeSelector | indent 8 }}
{{- end }}
{{- end }}
{{- if not .Values.global.enterprise -}}
{{- if .Values.ingress.enabled -}}
{{- $serviceName := include "kafka-topics-ui.fullname" . -}}
{{- $servicePort := .Values.service.externalPort -}}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "kafka-topics-ui.fullname" . }}
labels:
app: {{ template "kafka-topics-ui.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
annotations:
{{- range $key, $value := .Values.ingress.annotations }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
rules:
{{- range $host := .Values.ingress.hosts }}
- host: {{ $host }}
http:
paths:
- path:
backend:
serviceName: {{ $serviceName }}
servicePort: {{ $servicePort }}
{{- end -}}
{{- if .Values.ingress.tls }}
tls:
{{ toYaml .Values.ingress.tls | indent 4 }}
{{- end -}}
{{- end -}}
{{- end -}}
{{- if not .Values.global.enterprise }}
apiVersion: v1
kind: Service
metadata:
name: {{ template "kafka-topics-ui.fullname" . }}
labels:
app: {{ template "kafka-topics-ui.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.externalPort }}
targetPort: {{ .Values.service.internalPort }}
protocol: TCP
name: {{ .Values.service.name }}
{{- if .Values.service.nodePort }}
nodePort: {{ .Values.service.nodePort }}
{{- end }}
selector:
app: {{ template "kafka-topics-ui.name" . }}
release: {{ .Release.Name }}
{{- end }}
# Default values for kafka-topics-ui.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicas: 1
image:
repository: landoop/kafka-topics-ui
tag: latest
pullPolicy: IfNotPresent
service:
name: web
type: NodePort
externalPort: 8000
internalPort: 8000
ingress:
enabled: false
# Used to create an Ingress record.
hosts:
- xip.io
annotations:
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
tls:
# Secrets must be manually created in the namespace.
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
kafkaRest:
name: cp-kafka-rest
port: 8082
labels:
io.cattle.role: project # options are cluster/project
io.rancher.app_min_version: 0.8.0
questions:
- variable: global.enterprise
default: false
description: "Enable Confluent enterprise platform, free 30 days trail https://www.confluent.io/download/"
type: boolean
label: Enable Confluent Enterprise Platform
group: "Global Settings"
- variable: cp-kafka.replicas
default: 3
description: "Replicas of Kafka Brokers"
type: int
min: 3
max: 30
required: true
label: Kafka Brokers
group: "Kafka Settings"
- variable: cp-kafka.persistence.enabled
default: false
description: "Enable persistent volume for Kafka"
type: boolean
required: true
label: Kafka Persistent Volume Enabled
show_subquestion_if: true
group: "Kafka Settings"
subquestions:
- variable: cp-kafka.persistence.size
default: "5Gi"
description: "Kafka Persistent Volume Size"
type: string
label: Kafka Volume Size
- variable: cp-kafka.persistence.storageClass
default: ""
description: "If undefined or null, uses the default StorageClass. Default to null"
type: storageclass
label: Default StorageClass for Kafka
# Zookeeper Configurations
- variable: cp-zookeeper.persistence.enabled
default: false
description: "Enable persistent volume for Zookeeper"
type: boolean
required: true
label: Zookeeper Persistent Volume Enabled
show_subquestion_if: true
group: "Zookeeper Settings"
subquestions:
- variable: cp-zookeeper.persistence.dataDirSize
default: "10Gi"
description: "Zookeeper Persistent Volume Size"
type: string
label: Zookeeper Data Directory Volume Size
- variable: cp-zookeeper.persistence.dataDirStorageClass
default: ""
description: "If undefined or null, uses the default StorageClass. Default to null"
type: storageclass
label: Default StorageClass for Zookeeper Data Directory
- variable: cp-zookeeper.persistence.dataLogDirSize
default: "10Gi"
description: "Zookeeper Persistent Volume Size"
type: string
label: Zookeeper Data Log Volume Size
- variable: cp-zookeeper.persistence.dataLogDirStorageClass
default: ""
description: "If undefined or null, uses the default StorageClass. Default to null"
type: storageclass
label: Default StorageClass for Zookeeper Data Log
# kafka-Topics-UI Configurations
- variable: kafka-topics-ui.enabled
default: true
description: "Enable kafka topics ui dashboard"
type: boolean
label: Enable Kafka Topics UI Dashboard
group: "Kafka Topics UI"
show_if: "global.enterprise=false"
- variable: kafka-topics-ui.ingress.enabled
default: true
description: "Expose kafka topics UI using Layer 7 Load Balancer - ingress"
type: boolean
label: Expose Kafka Topics UI using Layer 7 Load Balancer
show_if: "kafka-topics-ui.enabled=true&&global.enterprise=false"
show_subquestion_if: true
group: "Kafka Topics UI"
subquestions:
- variable: kafka-topics-ui.ingress.hosts[0]
default: "xip.io"
description: "layer 7 Load Balancer hostname"
type: hostname
show_if: "kafka-topics-ui.enabled=true&&kafka-topics-ui.ingress.enabled=true&&global.enterprise=false"
required: true
label: Layer 7 Load Balancer Hostname
dependencies:
- name: cp-kafka
version: 0.1.0
repository: file://./charts/cp-kafka
condition: cp-kafka.enabled
- name: cp-zookeeper
version: 0.1.0
repository: file://./charts/cp-zookeeper
condition: cp-zookeeper.enabled
- name: cp-schema-registry
version: 0.1.0
repository: file://./charts/cp-schema-registry
condition: cp-schema-registry.enabled
- name: cp-kafka-rest
version: 0.1.0
repository: file://./charts/cp-kafka-rest
condition: cp-kafka-rest.enabled
- name: cp-kafka-connect
version: 0.1.0
repository: file://./charts/cp-kafka-connect
condition: cp-kafka-connect.enabled
- name: cp-ksql-server
version: 0.1.0
repository: file://./charts/cp-ksql-server
condition: cp-ksql-server.enabled
- name: cp-control-center
version: 0.1.0
repository: file://./charts/cp-control-center
condition: cp-control-center.enabled
- name: kafka-topics-ui
version: 0.1.0
repository: file://./charts/kafka-topics-ui
condition: kafka-topics-ui.enabled
{{- if (index .Values "cp-zookeeper" "enabled") -}}
## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
Connection string for Confluent Kafka:
{{ .Release.Name }}-cp-zookeeper-0.{{ .Release.Name }}-cp-zookeeper-headless:{{ default 2181 .Values.clientPort }},{{ .Release.Name }}-cp-zookeeper-1.{{ .Release.Name }}-cp-zookeeper-headless:{{ default 2181 .Values.clientPort }},...
To connect from a client pod:
1. Deploy a zookeeper client pod with configuration:
apiVersion: v1
kind: Pod
metadata:
name: zookeeper-client
namespace: {{ .Release.Namespace }}
spec:
containers:
- name: zookeeper-client
image: {{index .Values "cp-zookeeper" "image" }}:{{index .Values "cp-zookeeper" "imageTag" }}
command:
- sh
- -c
- "exec tail -f /dev/null"
2. Log into the Pod
kubectl exec -it zookeeper-client -- /bin/bash
3. Use zookeeper-shell to connect in the zookeeper-client Pod:
zookeeper-shell {{ .Release.Name }}-cp-zookeeper:{{ default 2181 .Values.clientPort }}
4. Explore with zookeeper commands, for example:
# Gives the list of active brokers
ls /brokers/ids
# Gives the list of topics
ls /brokers/topics
# Gives more detailed information of the broker id '0'
get /brokers/ids/0
{{- end -}}
{{- if (index .Values "cp-kafka" "enabled") -}}
## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
To connect from a client pod:
1. Deploy a kafka client pod with configuration:
apiVersion: v1
kind: Pod
metadata:
name: kafka-client
namespace: {{ .Release.Namespace }}
spec:
containers:
- name: kafka-client
image: {{ index .Values "cp-kafka" "image" }}:{{ index .Values "cp-kafka" "imageTag" }}
command:
- sh
- -c
- "exec tail -f /dev/null"
2. Log into the Pod
kubectl exec -it kafka-client -- /bin/bash
3. Explore with kafka commands:
# Create the topic
kafka-topics --zookeeper {{ .Release.Name }}-cp-zookeeper-headless:{{ default 2181 .Values.clientPort }} --topic {{ .Release.Name }}-topic --create --partitions 1 --replication-factor 1 --if-not-exists
# Create a message
MESSAGE="`date -u`"
# Produce a test message to the topic
echo "$MESSAGE" | kafka-console-producer --broker-list {{ .Release.Name }}-cp-kafka-headless:9092 --topic {{ .Release.Name }}-topic
# Consume a test message from the topic
kafka-console-consumer --bootstrap-server {{ .Release.Name }}-cp-kafka-headless:9092 --topic {{ .Release.Name }}-topic --from-beginning --timeout-ms 2000 --max-messages 1 | grep "$MESSAGE"
{{- end -}}
\ No newline at end of file
{{/* vim: set filetype=mustache: */}}
{{/*
Expand the name of the chart.
*/}}
{{- define "cp-helm-charts.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "cp-helm-charts.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "cp-helm-charts.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
\ No newline at end of file
global:
enterprise: false
## ------------------------------------------------------
## Zookeeper
## ------------------------------------------------------
cp-zookeeper:
enabled: true
servers: 3
image: ranchercharts/confluentinc-cp-zookeeper
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
persistence:
enabled: false
## The size of the PersistentVolume to allocate to each Zookeeper Pod in the StatefulSet. For
## production servers this number should likely be much larger.
##
## Size for Data dir, where ZooKeeper will store the in-memory database snapshots.
dataDirSize: 10Gi
# dataDirStorageClass: ""
## Size for data log dir, which is a dedicated log device to be used, and helps avoid competition between logging and snaphots.
dataLogDirSize: 10Gi
# dataLogDirStorageClass: ""
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## Kafka
## ------------------------------------------------------
cp-kafka:
enabled: true
brokers: 3
image: ranchercharts/confluentinc-cp-kafka
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
persistence:
enabled: false
# storageClass: ""
size: 5Gi
disksPerBroker: 1
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## Schema Registry
## ------------------------------------------------------
cp-schema-registry:
enabled: true
image: ranchercharts/confluentinc-cp-schema-registry
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## REST Proxy
## ------------------------------------------------------
cp-kafka-rest:
enabled: true
image: ranchercharts/confluentinc-cp-kafka-rest
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## Kafka Connect
## ------------------------------------------------------
cp-kafka-connect:
enabled: true
image: ranchercharts/confluentinc-cp-kafka-connect
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
## ------------------------------------------------------
## KSQL Server
## ------------------------------------------------------
cp-ksql-server:
enabled: true
image: ranchercharts/confluentinc-cp-ksql-server
imageTag: 5.3.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
ksql:
headless: false
## ------------------------------------------------------
## Control Center
## ------------------------------------------------------
cp-control-center:
enabled: true
image: ranchercharts/confluentinc-cp-enterprise-control-center
imageTag: 5.2.0
## Optionally specify an array of imagePullSecrets. Secrets must be manually created in the namespace.
## https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
imagePullSecrets:
# - name: "regcred"
heapOptions: "-Xms512M -Xmx512M"
resources: {}
## If you do want to specify resources, uncomment the following lines, adjust them as necessary,
## and remove the curly braces after 'resources:'
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
kafka-topics-ui:
enabled: true
replicas: 1
image:
repository: ranchercharts/landoop-kafka-topics-ui
tag: latest
pullPolicy: IfNotPresent
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment