Skip to content

C
charts
Commit 93a99436 by Guangbo Chen
Options

Bump hadoop to v1.1.1

parent 66555b1d
Showing with 271 additions and 469 deletions
charts/hadoop/v1.1.1/Chart.yaml
apiVersion: v1
description: The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
name: hadoop
version: 1.0.7
appVersion: 2.7.3
version: 1.1.1
appVersion: 2.9.0
home: https://hadoop.apache.org/
sources:
- https://github.com/apache/hadoop
icon: http://hadoop.apache.org/images/hadoop-logo.jpg
icon: file://../hadoop-logo.png
maintainers:
- name: danisla
email: disla@google.com
charts/hadoop/v1.1.1/README.md
......@@ -24,15 +24,17 @@ The following table lists the configurable parameters of the Hadoop chart and th
| Parameter | Description | Default |
| ------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------- |
| `image.repository` | Hadoop image ([source](https://github.com/Comcast/kube-yarn/tree/master/image)) | `danisla/hadoop` |
| `image.tag` | Version of hadoop libraries being used | `{VERSION}` |
| `image.pullPolicy` | Pull policy for the images | `IfNotPresent` |
| `image.repository` | Hadoop image ([source](https://github.com/Comcast/kube-yarn/tree/master/image)) | `danisla/hadoop` |
| `image.tag` | Hadoop image tag | `2.9.0` |
| `imagee.pullPolicy` | Pull policy for the images | `IfNotPresent` |
| `hadoopVersion` | Version of hadoop libraries being used | `2.9.0` |
| `antiAffinity` | Pod antiaffinity, `hard` or `soft` | `hard` |
| `hdfs.nameNode.pdbMinAvailable` | PDB for HDFS NameNode | `1` |
| `hdfs.nameNode.resources` | resources for the HDFS NameNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `hdfs.dataNode.replicas` | Number of HDFS DataNode replicas | `1` |
| `hdfs.dataNode.pdbMinAvailable` | PDB for HDFS DataNode | `1` |
| `hdfs.dataNode.resources` | resources for the HDFS DataNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `hdfs.webhdfs.enabled` | Enable WebHDFS REST API | `false`
| `yarn.resourceManager.pdbMinAvailable` | PDB for the YARN ResourceManager | `1` |
| `yarn.resourceManager.resources` | resources for the YARN ResourceManager | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `yarn.nodeManager.pdbMinAvailable` | PDB for the YARN NodeManager | `1` |
......
charts/hadoop/v1.1.1/charts/zeppelin/Chart.yaml
apiVersion: v1
description: Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
name: zeppelin
version: 1.0.1
version: 1.1.0
appVersion: 0.7.2
home: https://zeppelin.apache.org/
sources:
......
charts/hadoop/v1.1.1/charts/zeppelin/README.md
......@@ -18,6 +18,12 @@ The following table lists the configurable parameters of the Zeppelin chart and
| `hadoop.useConfigMap` | Use external Hadoop configuration for Spark executors | `false` |
| `hadoop.configMapName` | Name of the hadoop config map to use (must be in same namespace) | `hadoop-config` |
| `hadoop.configPath` | Path in the Zeppelin image where the Hadoop config is mounted | `/usr/hadoop-2.7.3/etc/hadoop` |
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.annotations` | Ingress annotations | `{}` |
| `ingress.hosts` | Ingress Hostnames | `["zeppelin.local"]` |
| `ingress.path` | Path within the URL structure | `/` |
| `ingress.tls` | Ingress TLS configuration | `[]` |
| `nodeSelecor` | Node selector for the Zeppelin deployment | `{}` |
## Related charts
......
charts/hadoop/v1.1.1/charts/zeppelin/questions.yml deleted 100644 → 0
questions:
- variable: defaultImage
default: "true"
description: "Use default Docker image"
label: Use Default Image
type: boolean
show_subquestion_if: false
group: "Container Images"
subquestions:
- variable: zeppelin.image.repository
default: "dylanmei/zeppelin"
description: "Zeppelin image name"
type: string
label: Zeppelin Image Name
- variable: zeppelin.image.tag
default: "0.7.2"
description: "Zeppelin image tag"
type: string
label: Zeppelin Image Tag
- variable: hadoop.useConfigMap
default: true
description: "Expose Zeppelin using Layer 7 Load Balancer - ingress"
type: boolean
label: Expose Zeppelin using Layer 7 Load Balancer
show_subquestion_if: true
group: "Hadoop Configuration"
required: true
- variable: ingress.enabled
default: "true"
description: "Expose Zeppelin using Layer 7 Load Balancer - ingress"
type: boolean
label: Expose Zeppelin using Layer 7 Load Balancer
show_subquestion_if: true
group: "Zeppelin"
required: true
subquestions:
- variable: ingress.hosts[0]
default: "xip.io"
description: "Hostname to your Zeppelin installation"
type: hostname
required: true
label: Hostname
- variable: server.type
default: "ClusterIP"
description: "yarn ui service type"
type: enum
group: "Zeppelin"
options:
- "ClusterIP"
- "NodePort"
required: true
label: Zeppelin Service Type
show_subquestion_if: "NodePort"
show_if: "ingress.enabled=false"
subquestions:
- variable: service.nodePort
default: ""
description: "NodePort http port(to set explicitly, choose port between 30000-32767)"
type: int
min: 30000
max: 32767
show_if: "ingress.enabled=false"
label: Zeppelin NodePort Number
charts/hadoop/v1.1.1/charts/zeppelin/templates/deployment.yaml
apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ template "zeppelin.fullname" . }}
......@@ -8,6 +8,10 @@ metadata:
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
selector:
matchLabels:
app: {{ template "zeppelin.name" . }}
release: {{ .Release.Name }}
strategy:
rollingUpdate:
maxUnavailable: 0
......
charts/hadoop/v1.1.1/charts/zeppelin/values.yaml
zeppelin:
zeppelin:
replicas: 1
image:
repository: dylanmei/zeppelin
tag: 0.7.2
resources:
limits:
memory: "4096Mi"
cpu: "2000m"
requests:
memory: "512Mi"
cpu: "200m"
hadoop:
useConfigMap: true
# configMapName: hadoop-hadoop
configPath: /usr/hadoop-2.7.3/etc/hadoop
spark:
driverMemory: 1g
executorMemory: 1g
numExecutors: 2
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
service:
## Annotations to be added to the Service
##
annotations: {}
## Cluster-internal IP address for Alertmanager Service
##
clusterIP: ""
## List of external IP addresses at which the Alertmanager Service will be available
##
externalIPs: []
## External IP address to assign to Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerIP: ""
## List of client IPs allowed to access Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerSourceRanges: []
## Port to expose on each node
## Only used if service.type is 'NodePort'
##
# nodePort: 30902
## Service type
##
type: ClusterIP
replicas: 1
image:
repository: ranchercharts/dylanmei-zeppelin
tag: 0.7.2
resources:
limits:
memory: "4096Mi"
cpu: "2000m"
requests:
memory: "512Mi"
cpu: "200m"
hadoop:
useConfigMap: true
# configMapName: hadoop-hadoop
configPath: /usr/hadoop-2.7.3/etc/hadoop
spark:
driverMemory: 1g
executorMemory: 1g
numExecutors: 2
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
service:
## Annotations to be added to the Service
##
annotations: {}
## Cluster-internal IP address for Alertmanager Service
##
clusterIP: ""
## List of external IP addresses at which the Alertmanager Service will be available
##
externalIPs: []
## External IP address to assign to Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerIP: ""
## List of client IPs allowed to access Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerSourceRanges: []
## Port to expose on each node
## Only used if service.type is 'NodePort'
##
# nodePort: 30902
## Service type
##
type: ClusterIP
charts/hadoop/v1.1.1/questions.yml
labels:
io.cattle.role: project # options are cluster/project
categories:
- hadoop
questions:
......@@ -10,17 +12,17 @@ questions:
group: "Container Images"
subquestions:
- variable: image.repository
default: "danisla/hadoop"
default: "ranchercharts/danisla-hadoop"
description: "Hadoop image name"
type: string
label: Hadoop Image Name
- variable: image.tag
default: "2.7.3"
default: "2.9.0"
description: "Hadoop image tag"
type: string
label: Hadoop Image Tag
- variable: zeppelin.zeppelin.image.repository
default: "dylanmei/zeppelin"
default: "ranchercharts/dylanmei-zeppelin"
description: "Zeppelin image name"
type: string
label: Zeppelin Image Name
......
charts/hadoop/v1.1.1/templates/NOTES.txt
1. You can check the status of HDFS by running this command:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-hdfs-nn-0 -- /usr/local/hadoop/bin/hdfs dfsadmin -report
kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-hdfs-nn-0 -- /usr/local/hadoop/bin/hdfs dfsadmin -report
2. You can list the yarn nodes by running this command:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/yarn node -list
kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/yarn node -list
3. Create a port-forward to the yarn resource manager UI:
kubectl port-forward -n {{ .Release.Namespace }} {{ template "hadoop.fullname" . }}-yarn-rm-0 8088:8088
kubectl port-forward -n {{ .Release.Namespace }} {{ include "hadoop.fullname" . }}-yarn-rm-0 8088:8088
Then open the ui in your browser:
open http://localhost:8088
4. You can run included hadoop tests like this:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-nm-0 -- /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-{{ .Values.image.tag }}-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt
kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-nm-0 -- /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-{{ .Values.hadoopVersion }}-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt
5. You can list the mapreduce jobs like this:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/mapred job -list
kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/mapred job -list
6. This chart can also be used with the zeppelin chart
helm install --namespace {{ .Release.Namespace }} --set hadoop.useConfigMap=true,hadoop.configMapName={{ template "hadoop.fullname" . }} stable/zeppelin
helm install --namespace {{ .Release.Namespace }} --set hadoop.useConfigMap=true,hadoop.configMapName={{ include "hadoop.fullname" . }} stable/zeppelin
7. You can scale the number of yarn nodes like this:
helm upgrade {{ .Release.Name }} --set yarn.nodeManager.replicas=4 stable/hadoop
......
charts/hadoop/v1.1.1/templates/_helpers.yaml → charts/hadoop/v1.1.1/templates/_helpers.tpl
......@@ -14,3 +14,10 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "hadoop.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
charts/hadoop/v1.1.1/templates/hadoop-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ template "hadoop.fullname" . }}
name: {{ include "hadoop.fullname" . }}
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data:
bootstrap.sh: |
#!/bin/bash
......@@ -41,8 +40,8 @@ data:
if [[ "${HOSTNAME}" =~ "hdfs-dn" ]]; then
mkdir -p /root/hdfs/datanode
# wait up to 30 seconds for namenode
(while [[ $count -lt 15 && -z `curl -sf http://{{ template "hadoop.fullname" . }}-hdfs-nn:50070` ]]; do ((count=count+1)) ; echo "Waiting for {{ template "hadoop.fullname" . }}-hdfs-nn" ; sleep 2; done && [[ $count -lt 15 ]])
# wait up to 30 seconds for namenode
(while [[ $count -lt 15 && -z `curl -sf http://{{ include "hadoop.fullname" . }}-hdfs-nn:50070` ]]; do ((count=count+1)) ; echo "Waiting for {{ include "hadoop.fullname" . }}-hdfs-nn" ; sleep 2; done && [[ $count -lt 15 ]])
[[ $? -ne 0 ]] && echo "Timeout waiting for hdfs-nn, exiting." && exit 1
$HADOOP_PREFIX/sbin/hadoop-daemon.sh start datanode
......@@ -74,7 +73,7 @@ data:
chmod +x start-yarn-nm.sh
# wait up to 30 seconds for resourcemanager
(while [[ $count -lt 15 && -z `curl -sf http://{{ template "hadoop.fullname" . }}-yarn-rm:8088/ws/v1/cluster/info` ]]; do ((count=count+1)) ; echo "Waiting for {{ template "hadoop.fullname" . }}-yarn-rm" ; sleep 2; done && [[ $count -lt 15 ]])
(while [[ $count -lt 15 && -z `curl -sf http://{{ include "hadoop.fullname" . }}-yarn-rm:8088/ws/v1/cluster/info` ]]; do ((count=count+1)) ; echo "Waiting for {{ include "hadoop.fullname" . }}-yarn-rm" ; sleep 2; done && [[ $count -lt 15 ]])
[[ $? -ne 0 ]] && echo "Timeout waiting for yarn-rm, exiting." && exit 1
./start-yarn-nm.sh
......@@ -96,7 +95,7 @@ data:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://{{ template "hadoop.fullname" . }}-hdfs-nn:9000/</value>
<value>hdfs://{{ include "hadoop.fullname" . }}-hdfs-nn:9000/</value>
<description>NameNode URI</description>
</property>
</configuration>
......@@ -105,6 +104,14 @@ data:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
{{- if .Values.hdfs.webhdfs.enabled -}}
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
{{- end -}}
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>false</value>
......@@ -147,7 +154,7 @@ data:
<value>0.0.0.0</value>
</property>
<!-- /Bind to all interfaces -->
</configuration>
mapred-site.xml: |
......@@ -161,11 +168,11 @@ data:
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm-0.{{ template "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:10020</value>
<value>{{ include "hadoop.fullname" . }}-yarn-rm-0.{{ include "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm-0.{{ template "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:19888</value>
<value>{{ include "hadoop.fullname" . }}-yarn-rm-0.{{ include "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:19888</value>
</property>
</configuration>
......@@ -253,7 +260,7 @@ data:
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm</value>
<value>{{ include "hadoop.fullname" . }}-yarn-rm</value>
</property>
<!-- Bind to all interfaces -->
......
charts/hadoop/v1.1.1/templates/hdfs-dn-pdb.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn
name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn
spec:
selector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-dn
minAvailable: {{ .Values.hdfs.dataNode.pdbMinAvailable }}
\ No newline at end of file
charts/hadoop/v1.1.1/templates/hdfs-dn-pvc.yaml
......@@ -2,12 +2,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn
name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn
spec:
accessModes:
......
charts/hadoop/v1.1.1/templates/hdfs-dn-statefulset.yaml
apiVersion: apps/v1beta1
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn
name: {{ include "hadoop.fullname" . }}-hdfs-dn
annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn
spec:
serviceName: {{ template "hadoop.fullname" . }}-hdfs-dn
selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-dn
serviceName: {{ include "hadoop.fullname" . }}-hdfs-dn
replicas: {{ .Values.hdfs.dataNode.replicas }}
template:
metadata:
labels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-dn
spec:
......@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: hdfs-dn
{{- else if eq .Values.antiAffinity "soft" }}
......@@ -37,20 +41,20 @@ spec:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: hdfs-dn
{{- end }}
terminationGracePeriodSeconds: 0
containers:
- name: hdfs-dn
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
- "-d"
resources:
resources:
{{ toYaml .Values.hdfs.dataNode.resources | indent 10 }}
readinessProbe:
httpGet:
......@@ -72,11 +76,11 @@ spec:
volumes:
- name: hadoop-config
configMap:
name: {{ template "hadoop.fullname" . }}
name: {{ include "hadoop.fullname" . }}
- name: dfs
{{- if .Values.persistence.dataNode.enabled }}
persistentVolumeClaim:
claimName: {{ template "hadoop.fullname" . }}-hdfs-dn
{{- else }}
claimName: {{ include "hadoop.fullname" . }}-hdfs-dn
{{- else }}
emptyDir: {}
{{- end }}
charts/hadoop/v1.1.1/templates/hdfs-dn-svc.yaml
......@@ -2,12 +2,11 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn
name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn
spec:
ports:
......@@ -18,6 +17,6 @@ spec:
port: 50075
clusterIP: None
selector:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-dn
\ No newline at end of file
charts/hadoop/v1.1.1/templates/hdfs-nn-pdb.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn
name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn
spec:
selector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-nn
minAvailable: {{ .Values.hdfs.nameNode.pdbMinAvailable }}
\ No newline at end of file
charts/hadoop/v1.1.1/templates/hdfs-nn-pvc.yaml
......@@ -2,12 +2,11 @@
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn
name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn
spec:
accessModes:
......
charts/hadoop/v1.1.1/templates/hdfs-nn-statefulset.yaml
apiVersion: apps/v1beta1
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn
name: {{ include "hadoop.fullname" . }}-hdfs-nn
annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn
spec:
serviceName: {{ template "hadoop.fullname" . }}-hdfs-nn
selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-nn
serviceName: {{ include "hadoop.fullname" . }}-hdfs-nn
replicas: 1
template:
metadata:
labels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-nn
spec:
......@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: hdfs-nn
{{- else if eq .Values.antiAffinity "soft" }}
......@@ -37,15 +41,15 @@ spec:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: hdfs-nn
{{- end }}
terminationGracePeriodSeconds: 0
containers:
- name: hdfs-nn
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
command:
- "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh"
......@@ -72,11 +76,11 @@ spec:
volumes:
- name: hadoop-config
configMap:
name: {{ template "hadoop.fullname" . }}
name: {{ include "hadoop.fullname" . }}
- name: dfs
{{- if .Values.persistence.nameNode.enabled }}
persistentVolumeClaim:
claimName: {{ template "hadoop.fullname" . }}-hdfs-nn
{{- else }}
claimName: {{ include "hadoop.fullname" . }}-hdfs-nn
{{- else }}
emptyDir: {}
{{- end }}
charts/hadoop/v1.1.1/templates/hdfs-nn-svc.yaml
......@@ -2,12 +2,11 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn
name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn
spec:
ports:
......@@ -18,6 +17,6 @@ spec:
port: 50070
clusterIP: None
selector:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-nn
\ No newline at end of file
charts/hadoop/v1.1.1/templates/yarn-nm-pdb.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm
name: {{ include "hadoop.fullname" . }}-yarn-nm
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm
spec:
selector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-nm
minAvailable: {{ .Values.yarn.nodeManager.pdbMinAvailable }}
\ No newline at end of file
charts/hadoop/v1.1.1/templates/yarn-nm-statefulset.yaml
apiVersion: apps/v1beta1
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm
name: {{ include "hadoop.fullname" . }}-yarn-nm
annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm
spec:
podManagementPolicy: Parallel
serviceName: {{ template "hadoop.fullname" . }}-yarn-nm
selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-nm
serviceName: {{ include "hadoop.fullname" . }}-yarn-nm
replicas: {{ .Values.yarn.nodeManager.replicas }}
{{- if .Values.yarn.nodeManager.parallelCreate }}
podManagementPolicy: Parallel
......@@ -20,7 +23,7 @@ spec:
template:
metadata:
labels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-nm
spec:
......@@ -31,7 +34,7 @@ spec:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: yarn-nm
{{- else if eq .Values.antiAffinity "soft" }}
......@@ -41,15 +44,15 @@ spec:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: yarn-nm
{{- end }}
terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }}
terminationGracePeriodSeconds: 0
containers:
- name: yarn-nm
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
ports:
- containerPort: 8088
name: web
......@@ -90,4 +93,4 @@ spec:
volumes:
- name: hadoop-config
configMap:
name: {{ template "hadoop.fullname" . }}
name: {{ include "hadoop.fullname" . }}
charts/hadoop/v1.1.1/templates/yarn-nm-svc.yaml
......@@ -2,12 +2,11 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm
name: {{ include "hadoop.fullname" . }}-yarn-nm
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm
spec:
ports:
......@@ -19,6 +18,6 @@ spec:
name: api
clusterIP: None
selector:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-nm
\ No newline at end of file
charts/hadoop/v1.1.1/templates/yarn-rm-pdb.yaml
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm
name: {{ include "hadoop.fullname" . }}-yarn-rm
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm
spec:
selector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm
minAvailable: {{ .Values.yarn.resourceManager.pdbMinAvailable }}
charts/hadoop/v1.1.1/templates/yarn-rm-statefulset.yaml
apiVersion: apps/v1beta1
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm
name: {{ include "hadoop.fullname" . }}-yarn-rm
annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm
spec:
serviceName: {{ template "hadoop.fullname" . }}-yarn-rm
selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm
serviceName: {{ include "hadoop.fullname" . }}-yarn-rm
replicas: 1
template:
metadata:
labels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm
spec:
......@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: yarn-rm
{{- else if eq .Values.antiAffinity "soft" }}
......@@ -37,15 +41,15 @@ spec:
topologyKey: "kubernetes.io/hostname"
labelSelector:
matchLabels:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }}
component: yarn-rm
{{- end }}
terminationGracePeriodSeconds: 0
containers:
- name: yarn-rm
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
ports:
- containerPort: 8088
name: web
......@@ -73,4 +77,4 @@ spec:
volumes:
- name: hadoop-config
configMap:
name: {{ template "hadoop.fullname" . }}
name: {{ include "hadoop.fullname" . }}
charts/hadoop/v1.1.1/templates/yarn-rm-svc.yaml
......@@ -2,12 +2,11 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm
name: {{ include "hadoop.fullname" . }}-yarn-rm
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm
spec:
ports:
......@@ -15,6 +14,6 @@ spec:
name: web
clusterIP: None
selector:
app: {{ template "hadoop.name" . }}
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm
\ No newline at end of file
charts/hadoop/v1.1.1/templates/yarn-ui-ingress.yaml deleted 100644 → 0
{{- if .Values.yarn.ingress.enabled -}}
{{- $serviceName := include "hadoop.fullname" . }}
{{- $routePrefix := .Values.yarn.routePrefix }}
{{- $releaseName := .Release.Name }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
{{- if .Values.yarn.ingress.annotations }}
annotations:
{{ toYaml .Values.yarn.ingress.annotations | indent 4 }}
{{- end }}
labels:
app: {{ template "hadoop.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
{{- if .Values.yarn.ingress.labels }}
{{ toYaml .Values.yarn.ingress.labels | indent 4 }}
{{- end }}
name: {{ template "hadoop.fullname" . }}
spec:
rules:
{{- range $host := .Values.yarn.ingress.hosts }}
- host: {{ $host }}
http:
paths:
- path: "{{ $routePrefix }}"
backend:
serviceName: {{ printf "%s-%s" $serviceName "yarn-ui" }}
servicePort: 8088
{{- end -}}
{{- if .Values.yarn.ingress.tls }}
tls:
{{ toYaml .Values.yarn.ingress.tls | indent 4 }}
{{- end }}
{{- end }}
charts/hadoop/v1.1.1/templates/yarn-ui-svc.yaml
......@@ -2,23 +2,16 @@
apiVersion: v1
kind: Service
metadata:
name: {{ template "hadoop.fullname" . }}-yarn-ui
name: {{ include "hadoop.fullname" . }}-yarn-ui
labels:
app: {{ template "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
app: {{ include "hadoop.name" . }}
chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-ui
spec:
type: {{ .Values.yarn.service.type }}
ports:
- name: web
port: 8088
targetPort: web
{{- if (and (eq .Values.yarn.service.type "NodePort") (not (empty .Values.yarn.service.nodePort)))}}
nodePort: {{ .Values.yarn.service.nodePort }}
{{- end }}
- port: 8088
name: web
selector:
app: {{ template "hadoop.name" . }}
release: {{ .Release.Name }}
app: {{ include "hadoop.name" . }}
component: yarn-rm
charts/hadoop/v1.1.1/tools/calc_resources.sh
......@@ -39,7 +39,7 @@ for NODE in ${NODES}; do
# Get available memory
AVAIL_MEM[$i]=$(jq '.node.memory.availableBytes' <<< "${NODE_STATS[$i]}")
AVAIL_MEM[$i]=$(bc -l <<< "scale=0; ${AVAIL_MEM[$i]}/1024/1024")
# Derive available CPU
USED_CPU=$(jq '.node.cpu.usageNanoCores' <<< "${NODE_STATS[$i]}")
AVAIL_CPU[$i]=$(bc -l <<< "scale=2; (${TOTAL_CPU} - ${USED_CPU})/1000000")
......
charts/hadoop/v1.1.1/values.yaml
# The base hadoop image to use for all components.
# See this repo for image build details: https://github.com/Comcast/kube-yarn/tree/master/image
image:
repository: danisla/hadoop
tag: 2.7.3
repository: ranchercharts/danisla-hadoop
tag: 2.9.0
pullPolicy: IfNotPresent
# Select antiAffinity as either hard or soft, default is hard
antiAffinity: "soft"
# The version of the hadoop libraries being used in the image.
hadoopVersion: 2.9.0
terminationGracePeriodSeconds: 30 # Duration in seconds a pod needs to terminate gracefully.
# Select antiAffinity as either hard or soft, default is soft
antiAffinity: "soft"
hdfs:
nameNode:
......@@ -35,6 +36,9 @@ hdfs:
memory: "2048Mi"
cpu: "1000m"
webhdfs:
enabled: false
yarn:
resourceManager:
pdbMinAvailable: 1
......@@ -66,40 +70,6 @@ yarn:
memory: "2048Mi"
cpu: "1000m"
service:
type: ClusterIP
# nodePort: 32000 #range form 32000-32767
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
persistence:
nameNode:
enabled: false
......@@ -112,94 +82,3 @@ persistence:
storageClass: "-"
accessMode: ReadWriteOnce
size: 200Gi
zeppelin:
enabled: true
#zeppelin configurations
zeppelin:
replicas: 1
image:
repository: dylanmei/zeppelin
tag: 0.7.2
resources:
limits:
memory: "4096Mi"
cpu: "2000m"
requests:
memory: "512Mi"
cpu: "200m"
hadoop:
useConfigMap: true
# configMapName: hadoop-hadoop
configPath: /usr/hadoop-2.7.3/etc/hadoop
spark:
driverMemory: 1g
executorMemory: 1g
numExecutors: 2
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
service:
## Annotations to be added to the Service
##
annotations: {}
## Cluster-internal IP address for Alertmanager Service
##
clusterIP: ""
## List of external IP addresses at which the Alertmanager Service will be available
##
externalIPs: []
## External IP address to assign to Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerIP: ""
## List of client IPs allowed to access Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerSourceRanges: []
## Port to expose on each node
## Only used if service.type is 'NodePort'
##
# nodePort: 30902
## Service type
##
type: ClusterIP
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment