Commit 93a99436 by Guangbo Chen

Bump hadoop to v1.1.1

parent 66555b1d
apiVersion: v1 apiVersion: v1
description: The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. description: The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
name: hadoop name: hadoop
version: 1.0.7 version: 1.1.1
appVersion: 2.7.3 appVersion: 2.9.0
home: https://hadoop.apache.org/ home: https://hadoop.apache.org/
sources: sources:
- https://github.com/apache/hadoop - https://github.com/apache/hadoop
icon: http://hadoop.apache.org/images/hadoop-logo.jpg icon: file://../hadoop-logo.png
maintainers: maintainers:
- name: danisla - name: danisla
email: disla@google.com email: disla@google.com
...@@ -24,15 +24,17 @@ The following table lists the configurable parameters of the Hadoop chart and th ...@@ -24,15 +24,17 @@ The following table lists the configurable parameters of the Hadoop chart and th
| Parameter | Description | Default | | Parameter | Description | Default |
| ------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------- | | ------------------------------------------------- | ------------------------------- | ---------------------------------------------------------------- |
| `image.repository` | Hadoop image ([source](https://github.com/Comcast/kube-yarn/tree/master/image)) | `danisla/hadoop` | | `image.repository` | Hadoop image ([source](https://github.com/Comcast/kube-yarn/tree/master/image)) | `danisla/hadoop` |
| `image.tag` | Version of hadoop libraries being used | `{VERSION}` | | `image.tag` | Hadoop image tag | `2.9.0` |
| `image.pullPolicy` | Pull policy for the images | `IfNotPresent` | | `imagee.pullPolicy` | Pull policy for the images | `IfNotPresent` |
| `hadoopVersion` | Version of hadoop libraries being used | `2.9.0` |
| `antiAffinity` | Pod antiaffinity, `hard` or `soft` | `hard` | | `antiAffinity` | Pod antiaffinity, `hard` or `soft` | `hard` |
| `hdfs.nameNode.pdbMinAvailable` | PDB for HDFS NameNode | `1` | | `hdfs.nameNode.pdbMinAvailable` | PDB for HDFS NameNode | `1` |
| `hdfs.nameNode.resources` | resources for the HDFS NameNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` | | `hdfs.nameNode.resources` | resources for the HDFS NameNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `hdfs.dataNode.replicas` | Number of HDFS DataNode replicas | `1` | | `hdfs.dataNode.replicas` | Number of HDFS DataNode replicas | `1` |
| `hdfs.dataNode.pdbMinAvailable` | PDB for HDFS DataNode | `1` | | `hdfs.dataNode.pdbMinAvailable` | PDB for HDFS DataNode | `1` |
| `hdfs.dataNode.resources` | resources for the HDFS DataNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` | | `hdfs.dataNode.resources` | resources for the HDFS DataNode | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `hdfs.webhdfs.enabled` | Enable WebHDFS REST API | `false`
| `yarn.resourceManager.pdbMinAvailable` | PDB for the YARN ResourceManager | `1` | | `yarn.resourceManager.pdbMinAvailable` | PDB for the YARN ResourceManager | `1` |
| `yarn.resourceManager.resources` | resources for the YARN ResourceManager | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` | | `yarn.resourceManager.resources` | resources for the YARN ResourceManager | `requests:memory=256Mi,cpu=10m,limits:memory=2048Mi,cpu=1000m` |
| `yarn.nodeManager.pdbMinAvailable` | PDB for the YARN NodeManager | `1` | | `yarn.nodeManager.pdbMinAvailable` | PDB for the YARN NodeManager | `1` |
......
apiVersion: v1 apiVersion: v1
description: Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more. description: Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.
name: zeppelin name: zeppelin
version: 1.0.1 version: 1.1.0
appVersion: 0.7.2 appVersion: 0.7.2
home: https://zeppelin.apache.org/ home: https://zeppelin.apache.org/
sources: sources:
......
...@@ -18,6 +18,12 @@ The following table lists the configurable parameters of the Zeppelin chart and ...@@ -18,6 +18,12 @@ The following table lists the configurable parameters of the Zeppelin chart and
| `hadoop.useConfigMap` | Use external Hadoop configuration for Spark executors | `false` | | `hadoop.useConfigMap` | Use external Hadoop configuration for Spark executors | `false` |
| `hadoop.configMapName` | Name of the hadoop config map to use (must be in same namespace) | `hadoop-config` | | `hadoop.configMapName` | Name of the hadoop config map to use (must be in same namespace) | `hadoop-config` |
| `hadoop.configPath` | Path in the Zeppelin image where the Hadoop config is mounted | `/usr/hadoop-2.7.3/etc/hadoop` | | `hadoop.configPath` | Path in the Zeppelin image where the Hadoop config is mounted | `/usr/hadoop-2.7.3/etc/hadoop` |
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.annotations` | Ingress annotations | `{}` |
| `ingress.hosts` | Ingress Hostnames | `["zeppelin.local"]` |
| `ingress.path` | Path within the URL structure | `/` |
| `ingress.tls` | Ingress TLS configuration | `[]` |
| `nodeSelecor` | Node selector for the Zeppelin deployment | `{}` |
## Related charts ## Related charts
......
questions:
- variable: defaultImage
default: "true"
description: "Use default Docker image"
label: Use Default Image
type: boolean
show_subquestion_if: false
group: "Container Images"
subquestions:
- variable: zeppelin.image.repository
default: "dylanmei/zeppelin"
description: "Zeppelin image name"
type: string
label: Zeppelin Image Name
- variable: zeppelin.image.tag
default: "0.7.2"
description: "Zeppelin image tag"
type: string
label: Zeppelin Image Tag
- variable: hadoop.useConfigMap
default: true
description: "Expose Zeppelin using Layer 7 Load Balancer - ingress"
type: boolean
label: Expose Zeppelin using Layer 7 Load Balancer
show_subquestion_if: true
group: "Hadoop Configuration"
required: true
- variable: ingress.enabled
default: "true"
description: "Expose Zeppelin using Layer 7 Load Balancer - ingress"
type: boolean
label: Expose Zeppelin using Layer 7 Load Balancer
show_subquestion_if: true
group: "Zeppelin"
required: true
subquestions:
- variable: ingress.hosts[0]
default: "xip.io"
description: "Hostname to your Zeppelin installation"
type: hostname
required: true
label: Hostname
- variable: server.type
default: "ClusterIP"
description: "yarn ui service type"
type: enum
group: "Zeppelin"
options:
- "ClusterIP"
- "NodePort"
required: true
label: Zeppelin Service Type
show_subquestion_if: "NodePort"
show_if: "ingress.enabled=false"
subquestions:
- variable: service.nodePort
default: ""
description: "NodePort http port(to set explicitly, choose port between 30000-32767)"
type: int
min: 30000
max: 32767
show_if: "ingress.enabled=false"
label: Zeppelin NodePort Number
apiVersion: extensions/v1beta1 apiVersion: apps/v1
kind: Deployment kind: Deployment
metadata: metadata:
name: {{ template "zeppelin.fullname" . }} name: {{ template "zeppelin.fullname" . }}
...@@ -8,6 +8,10 @@ metadata: ...@@ -8,6 +8,10 @@ metadata:
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }} heritage: {{ .Release.Service }}
spec: spec:
selector:
matchLabels:
app: {{ template "zeppelin.name" . }}
release: {{ .Release.Name }}
strategy: strategy:
rollingUpdate: rollingUpdate:
maxUnavailable: 0 maxUnavailable: 0
......
zeppelin: zeppelin:
zeppelin: replicas: 1
replicas: 1 image:
image: repository: ranchercharts/dylanmei-zeppelin
repository: dylanmei/zeppelin tag: 0.7.2
tag: 0.7.2 resources:
resources: limits:
limits: memory: "4096Mi"
memory: "4096Mi" cpu: "2000m"
cpu: "2000m" requests:
requests: memory: "512Mi"
memory: "512Mi" cpu: "200m"
cpu: "200m"
hadoop:
hadoop: useConfigMap: true
useConfigMap: true # configMapName: hadoop-hadoop
# configMapName: hadoop-hadoop configPath: /usr/hadoop-2.7.3/etc/hadoop
configPath: /usr/hadoop-2.7.3/etc/hadoop
spark:
spark: driverMemory: 1g
driverMemory: 1g executorMemory: 1g
executorMemory: 1g numExecutors: 2
numExecutors: 2
ingress:
ingress: ## If true, Grafana Ingress will be created
## If true, Grafana Ingress will be created ##
## enabled: false
enabled: false
## Annotations for Alertmanager Ingress
## Annotations for Alertmanager Ingress ##
## annotations: {}
annotations: {} # kubernetes.io/ingress.class: nginx
# kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true"
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
## Labels to be added to the Ingress ##
## labels: {}
labels: {}
## Hostnames.
## Hostnames. ## Must be provided if Ingress is enabled.
## Must be provided if Ingress is enabled. ##
## # hosts:
# hosts: # - grafana.domain.com
# - grafana.domain.com hosts: []
hosts: []
## TLS configuration for Alertmanager Ingress
## TLS configuration for Alertmanager Ingress ## Secret must be manually created in the namespace
## Secret must be manually created in the namespace ##
## tls: []
tls: [] # - secretName: alertmanager-general-tls
# - secretName: alertmanager-general-tls # hosts:
# hosts: # - alertmanager.example.com
# - alertmanager.example.com
service:
service:
## Annotations to be added to the Service
## Annotations to be added to the Service ##
## annotations: {}
annotations: {}
## Cluster-internal IP address for Alertmanager Service
## Cluster-internal IP address for Alertmanager Service ##
## clusterIP: ""
clusterIP: ""
## List of external IP addresses at which the Alertmanager Service will be available
## List of external IP addresses at which the Alertmanager Service will be available ##
## externalIPs: []
externalIPs: []
## External IP address to assign to Alertmanager Service
## External IP address to assign to Alertmanager Service ## Only used if service.type is 'LoadBalancer' and supported by cloud provider
## Only used if service.type is 'LoadBalancer' and supported by cloud provider ##
## loadBalancerIP: ""
loadBalancerIP: ""
## List of client IPs allowed to access Alertmanager Service
## List of client IPs allowed to access Alertmanager Service ## Only used if service.type is 'LoadBalancer' and supported by cloud provider
## Only used if service.type is 'LoadBalancer' and supported by cloud provider ##
## loadBalancerSourceRanges: []
loadBalancerSourceRanges: []
## Port to expose on each node
## Port to expose on each node ## Only used if service.type is 'NodePort'
## Only used if service.type is 'NodePort' ##
## # nodePort: 30902
# nodePort: 30902
## Service type
## Service type ##
## type: ClusterIP
type: ClusterIP
labels:
io.cattle.role: project # options are cluster/project
categories: categories:
- hadoop - hadoop
questions: questions:
...@@ -10,17 +12,17 @@ questions: ...@@ -10,17 +12,17 @@ questions:
group: "Container Images" group: "Container Images"
subquestions: subquestions:
- variable: image.repository - variable: image.repository
default: "danisla/hadoop" default: "ranchercharts/danisla-hadoop"
description: "Hadoop image name" description: "Hadoop image name"
type: string type: string
label: Hadoop Image Name label: Hadoop Image Name
- variable: image.tag - variable: image.tag
default: "2.7.3" default: "2.9.0"
description: "Hadoop image tag" description: "Hadoop image tag"
type: string type: string
label: Hadoop Image Tag label: Hadoop Image Tag
- variable: zeppelin.zeppelin.image.repository - variable: zeppelin.zeppelin.image.repository
default: "dylanmei/zeppelin" default: "ranchercharts/dylanmei-zeppelin"
description: "Zeppelin image name" description: "Zeppelin image name"
type: string type: string
label: Zeppelin Image Name label: Zeppelin Image Name
......
1. You can check the status of HDFS by running this command: 1. You can check the status of HDFS by running this command:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-hdfs-nn-0 -- /usr/local/hadoop/bin/hdfs dfsadmin -report kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-hdfs-nn-0 -- /usr/local/hadoop/bin/hdfs dfsadmin -report
2. You can list the yarn nodes by running this command: 2. You can list the yarn nodes by running this command:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/yarn node -list kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/yarn node -list
3. Create a port-forward to the yarn resource manager UI: 3. Create a port-forward to the yarn resource manager UI:
kubectl port-forward -n {{ .Release.Namespace }} {{ template "hadoop.fullname" . }}-yarn-rm-0 8088:8088 kubectl port-forward -n {{ .Release.Namespace }} {{ include "hadoop.fullname" . }}-yarn-rm-0 8088:8088
Then open the ui in your browser: Then open the ui in your browser:
open http://localhost:8088 open http://localhost:8088
4. You can run included hadoop tests like this: 4. You can run included hadoop tests like this:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-nm-0 -- /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-{{ .Values.image.tag }}-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-nm-0 -- /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-{{ .Values.hadoopVersion }}-tests.jar TestDFSIO -write -nrFiles 5 -fileSize 128MB -resFile /tmp/TestDFSIOwrite.txt
5. You can list the mapreduce jobs like this: 5. You can list the mapreduce jobs like this:
kubectl exec -n {{ .Release.Namespace }} -it {{ template "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/mapred job -list kubectl exec -n {{ .Release.Namespace }} -it {{ include "hadoop.fullname" . }}-yarn-rm-0 -- /usr/local/hadoop/bin/mapred job -list
6. This chart can also be used with the zeppelin chart 6. This chart can also be used with the zeppelin chart
helm install --namespace {{ .Release.Namespace }} --set hadoop.useConfigMap=true,hadoop.configMapName={{ template "hadoop.fullname" . }} stable/zeppelin helm install --namespace {{ .Release.Namespace }} --set hadoop.useConfigMap=true,hadoop.configMapName={{ include "hadoop.fullname" . }} stable/zeppelin
7. You can scale the number of yarn nodes like this: 7. You can scale the number of yarn nodes like this:
helm upgrade {{ .Release.Name }} --set yarn.nodeManager.replicas=4 stable/hadoop helm upgrade {{ .Release.Name }} --set yarn.nodeManager.replicas=4 stable/hadoop
......
...@@ -14,3 +14,10 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this ...@@ -14,3 +14,10 @@ We truncate at 63 chars because some Kubernetes name fields are limited to this
{{- $name := default .Chart.Name .Values.nameOverride -}} {{- $name := default .Chart.Name .Values.nameOverride -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}} {{- end -}}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "hadoop.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
{{- end -}}
apiVersion: v1 apiVersion: v1
kind: ConfigMap kind: ConfigMap
metadata: metadata:
name: {{ template "hadoop.fullname" . }} name: {{ include "hadoop.fullname" . }}
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
data: data:
bootstrap.sh: | bootstrap.sh: |
#!/bin/bash #!/bin/bash
...@@ -41,8 +40,8 @@ data: ...@@ -41,8 +40,8 @@ data:
if [[ "${HOSTNAME}" =~ "hdfs-dn" ]]; then if [[ "${HOSTNAME}" =~ "hdfs-dn" ]]; then
mkdir -p /root/hdfs/datanode mkdir -p /root/hdfs/datanode
# wait up to 30 seconds for namenode # wait up to 30 seconds for namenode
(while [[ $count -lt 15 && -z `curl -sf http://{{ template "hadoop.fullname" . }}-hdfs-nn:50070` ]]; do ((count=count+1)) ; echo "Waiting for {{ template "hadoop.fullname" . }}-hdfs-nn" ; sleep 2; done && [[ $count -lt 15 ]]) (while [[ $count -lt 15 && -z `curl -sf http://{{ include "hadoop.fullname" . }}-hdfs-nn:50070` ]]; do ((count=count+1)) ; echo "Waiting for {{ include "hadoop.fullname" . }}-hdfs-nn" ; sleep 2; done && [[ $count -lt 15 ]])
[[ $? -ne 0 ]] && echo "Timeout waiting for hdfs-nn, exiting." && exit 1 [[ $? -ne 0 ]] && echo "Timeout waiting for hdfs-nn, exiting." && exit 1
$HADOOP_PREFIX/sbin/hadoop-daemon.sh start datanode $HADOOP_PREFIX/sbin/hadoop-daemon.sh start datanode
...@@ -74,7 +73,7 @@ data: ...@@ -74,7 +73,7 @@ data:
chmod +x start-yarn-nm.sh chmod +x start-yarn-nm.sh
# wait up to 30 seconds for resourcemanager # wait up to 30 seconds for resourcemanager
(while [[ $count -lt 15 && -z `curl -sf http://{{ template "hadoop.fullname" . }}-yarn-rm:8088/ws/v1/cluster/info` ]]; do ((count=count+1)) ; echo "Waiting for {{ template "hadoop.fullname" . }}-yarn-rm" ; sleep 2; done && [[ $count -lt 15 ]]) (while [[ $count -lt 15 && -z `curl -sf http://{{ include "hadoop.fullname" . }}-yarn-rm:8088/ws/v1/cluster/info` ]]; do ((count=count+1)) ; echo "Waiting for {{ include "hadoop.fullname" . }}-yarn-rm" ; sleep 2; done && [[ $count -lt 15 ]])
[[ $? -ne 0 ]] && echo "Timeout waiting for yarn-rm, exiting." && exit 1 [[ $? -ne 0 ]] && echo "Timeout waiting for yarn-rm, exiting." && exit 1
./start-yarn-nm.sh ./start-yarn-nm.sh
...@@ -96,7 +95,7 @@ data: ...@@ -96,7 +95,7 @@ data:
<configuration> <configuration>
<property> <property>
<name>fs.defaultFS</name> <name>fs.defaultFS</name>
<value>hdfs://{{ template "hadoop.fullname" . }}-hdfs-nn:9000/</value> <value>hdfs://{{ include "hadoop.fullname" . }}-hdfs-nn:9000/</value>
<description>NameNode URI</description> <description>NameNode URI</description>
</property> </property>
</configuration> </configuration>
...@@ -105,6 +104,14 @@ data: ...@@ -105,6 +104,14 @@ data:
<?xml version="1.0"?> <?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration> <configuration>
{{- if .Values.hdfs.webhdfs.enabled -}}
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
{{- end -}}
<property> <property>
<name>dfs.datanode.use.datanode.hostname</name> <name>dfs.datanode.use.datanode.hostname</name>
<value>false</value> <value>false</value>
...@@ -147,7 +154,7 @@ data: ...@@ -147,7 +154,7 @@ data:
<value>0.0.0.0</value> <value>0.0.0.0</value>
</property> </property>
<!-- /Bind to all interfaces --> <!-- /Bind to all interfaces -->
</configuration> </configuration>
mapred-site.xml: | mapred-site.xml: |
...@@ -161,11 +168,11 @@ data: ...@@ -161,11 +168,11 @@ data:
</property> </property>
<property> <property>
<name>mapreduce.jobhistory.address</name> <name>mapreduce.jobhistory.address</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm-0.{{ template "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:10020</value> <value>{{ include "hadoop.fullname" . }}-yarn-rm-0.{{ include "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:10020</value>
</property> </property>
<property> <property>
<name>mapreduce.jobhistory.webapp.address</name> <name>mapreduce.jobhistory.webapp.address</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm-0.{{ template "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:19888</value> <value>{{ include "hadoop.fullname" . }}-yarn-rm-0.{{ include "hadoop.fullname" . }}-yarn-rm.{{ .Release.Namespace }}.svc.cluster.local:19888</value>
</property> </property>
</configuration> </configuration>
...@@ -253,7 +260,7 @@ data: ...@@ -253,7 +260,7 @@ data:
<configuration> <configuration>
<property> <property>
<name>yarn.resourcemanager.hostname</name> <name>yarn.resourcemanager.hostname</name>
<value>{{ template "hadoop.fullname" . }}-yarn-rm</value> <value>{{ include "hadoop.fullname" . }}-yarn-rm</value>
</property> </property>
<!-- Bind to all interfaces --> <!-- Bind to all interfaces -->
......
apiVersion: policy/v1beta1 apiVersion: policy/v1beta1
kind: PodDisruptionBudget kind: PodDisruptionBudget
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn component: hdfs-dn
spec: spec:
selector: selector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-dn component: hdfs-dn
minAvailable: {{ .Values.hdfs.dataNode.pdbMinAvailable }} minAvailable: {{ .Values.hdfs.dataNode.pdbMinAvailable }}
\ No newline at end of file
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
apiVersion: v1 apiVersion: v1
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn component: hdfs-dn
spec: spec:
accessModes: accessModes:
......
apiVersion: apps/v1beta1 apiVersion: apps/v1
kind: StatefulSet kind: StatefulSet
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn name: {{ include "hadoop.fullname" . }}-hdfs-dn
annotations: annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }} checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn component: hdfs-dn
spec: spec:
serviceName: {{ template "hadoop.fullname" . }}-hdfs-dn selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-dn
serviceName: {{ include "hadoop.fullname" . }}-hdfs-dn
replicas: {{ .Values.hdfs.dataNode.replicas }} replicas: {{ .Values.hdfs.dataNode.replicas }}
template: template:
metadata: metadata:
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-dn component: hdfs-dn
spec: spec:
...@@ -27,7 +31,7 @@ spec: ...@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname" - topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: hdfs-dn component: hdfs-dn
{{- else if eq .Values.antiAffinity "soft" }} {{- else if eq .Values.antiAffinity "soft" }}
...@@ -37,20 +41,20 @@ spec: ...@@ -37,20 +41,20 @@ spec:
topologyKey: "kubernetes.io/hostname" topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: hdfs-dn component: hdfs-dn
{{- end }} {{- end }}
terminationGracePeriodSeconds: 0 terminationGracePeriodSeconds: 0
containers: containers:
- name: hdfs-dn - name: hdfs-dn
image: {{ .Values.image.repository }}:{{ .Values.image.tag }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
command: command:
- "/bin/bash" - "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh" - "/tmp/hadoop-config/bootstrap.sh"
- "-d" - "-d"
resources: resources:
{{ toYaml .Values.hdfs.dataNode.resources | indent 10 }} {{ toYaml .Values.hdfs.dataNode.resources | indent 10 }}
readinessProbe: readinessProbe:
httpGet: httpGet:
...@@ -72,11 +76,11 @@ spec: ...@@ -72,11 +76,11 @@ spec:
volumes: volumes:
- name: hadoop-config - name: hadoop-config
configMap: configMap:
name: {{ template "hadoop.fullname" . }} name: {{ include "hadoop.fullname" . }}
- name: dfs - name: dfs
{{- if .Values.persistence.dataNode.enabled }} {{- if .Values.persistence.dataNode.enabled }}
persistentVolumeClaim: persistentVolumeClaim:
claimName: {{ template "hadoop.fullname" . }}-hdfs-dn claimName: {{ include "hadoop.fullname" . }}-hdfs-dn
{{- else }} {{- else }}
emptyDir: {} emptyDir: {}
{{- end }} {{- end }}
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-dn name: {{ include "hadoop.fullname" . }}-hdfs-dn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-dn component: hdfs-dn
spec: spec:
ports: ports:
...@@ -18,6 +17,6 @@ spec: ...@@ -18,6 +17,6 @@ spec:
port: 50075 port: 50075
clusterIP: None clusterIP: None
selector: selector:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-dn component: hdfs-dn
\ No newline at end of file
apiVersion: policy/v1beta1 apiVersion: policy/v1beta1
kind: PodDisruptionBudget kind: PodDisruptionBudget
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn component: hdfs-nn
spec: spec:
selector: selector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-nn component: hdfs-nn
minAvailable: {{ .Values.hdfs.nameNode.pdbMinAvailable }} minAvailable: {{ .Values.hdfs.nameNode.pdbMinAvailable }}
\ No newline at end of file
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
apiVersion: v1 apiVersion: v1
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn component: hdfs-nn
spec: spec:
accessModes: accessModes:
......
apiVersion: apps/v1beta1 apiVersion: apps/v1
kind: StatefulSet kind: StatefulSet
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn name: {{ include "hadoop.fullname" . }}-hdfs-nn
annotations: annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }} checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn component: hdfs-nn
spec: spec:
serviceName: {{ template "hadoop.fullname" . }}-hdfs-nn selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: hdfs-nn
serviceName: {{ include "hadoop.fullname" . }}-hdfs-nn
replicas: 1 replicas: 1
template: template:
metadata: metadata:
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-nn component: hdfs-nn
spec: spec:
...@@ -27,7 +31,7 @@ spec: ...@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname" - topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: hdfs-nn component: hdfs-nn
{{- else if eq .Values.antiAffinity "soft" }} {{- else if eq .Values.antiAffinity "soft" }}
...@@ -37,15 +41,15 @@ spec: ...@@ -37,15 +41,15 @@ spec:
topologyKey: "kubernetes.io/hostname" topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: hdfs-nn component: hdfs-nn
{{- end }} {{- end }}
terminationGracePeriodSeconds: 0 terminationGracePeriodSeconds: 0
containers: containers:
- name: hdfs-nn - name: hdfs-nn
image: {{ .Values.image.repository }}:{{ .Values.image.tag }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
command: command:
- "/bin/bash" - "/bin/bash"
- "/tmp/hadoop-config/bootstrap.sh" - "/tmp/hadoop-config/bootstrap.sh"
...@@ -72,11 +76,11 @@ spec: ...@@ -72,11 +76,11 @@ spec:
volumes: volumes:
- name: hadoop-config - name: hadoop-config
configMap: configMap:
name: {{ template "hadoop.fullname" . }} name: {{ include "hadoop.fullname" . }}
- name: dfs - name: dfs
{{- if .Values.persistence.nameNode.enabled }} {{- if .Values.persistence.nameNode.enabled }}
persistentVolumeClaim: persistentVolumeClaim:
claimName: {{ template "hadoop.fullname" . }}-hdfs-nn claimName: {{ include "hadoop.fullname" . }}-hdfs-nn
{{- else }} {{- else }}
emptyDir: {} emptyDir: {}
{{- end }} {{- end }}
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-hdfs-nn name: {{ include "hadoop.fullname" . }}-hdfs-nn
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: hdfs-nn component: hdfs-nn
spec: spec:
ports: ports:
...@@ -18,6 +17,6 @@ spec: ...@@ -18,6 +17,6 @@ spec:
port: 50070 port: 50070
clusterIP: None clusterIP: None
selector: selector:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: hdfs-nn component: hdfs-nn
\ No newline at end of file
apiVersion: policy/v1beta1 apiVersion: policy/v1beta1
kind: PodDisruptionBudget kind: PodDisruptionBudget
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm name: {{ include "hadoop.fullname" . }}-yarn-nm
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm component: yarn-nm
spec: spec:
selector: selector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-nm component: yarn-nm
minAvailable: {{ .Values.yarn.nodeManager.pdbMinAvailable }} minAvailable: {{ .Values.yarn.nodeManager.pdbMinAvailable }}
\ No newline at end of file
apiVersion: apps/v1beta1 apiVersion: apps/v1
kind: StatefulSet kind: StatefulSet
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm name: {{ include "hadoop.fullname" . }}-yarn-nm
annotations: annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }} checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm component: yarn-nm
spec: spec:
podManagementPolicy: Parallel selector:
serviceName: {{ template "hadoop.fullname" . }}-yarn-nm matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-nm
serviceName: {{ include "hadoop.fullname" . }}-yarn-nm
replicas: {{ .Values.yarn.nodeManager.replicas }} replicas: {{ .Values.yarn.nodeManager.replicas }}
{{- if .Values.yarn.nodeManager.parallelCreate }} {{- if .Values.yarn.nodeManager.parallelCreate }}
podManagementPolicy: Parallel podManagementPolicy: Parallel
...@@ -20,7 +23,7 @@ spec: ...@@ -20,7 +23,7 @@ spec:
template: template:
metadata: metadata:
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-nm component: yarn-nm
spec: spec:
...@@ -31,7 +34,7 @@ spec: ...@@ -31,7 +34,7 @@ spec:
- topologyKey: "kubernetes.io/hostname" - topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: yarn-nm component: yarn-nm
{{- else if eq .Values.antiAffinity "soft" }} {{- else if eq .Values.antiAffinity "soft" }}
...@@ -41,15 +44,15 @@ spec: ...@@ -41,15 +44,15 @@ spec:
topologyKey: "kubernetes.io/hostname" topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: yarn-nm component: yarn-nm
{{- end }} {{- end }}
terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }} terminationGracePeriodSeconds: 0
containers: containers:
- name: yarn-nm - name: yarn-nm
image: {{ .Values.image.repository }}:{{ .Values.image.tag }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
ports: ports:
- containerPort: 8088 - containerPort: 8088
name: web name: web
...@@ -90,4 +93,4 @@ spec: ...@@ -90,4 +93,4 @@ spec:
volumes: volumes:
- name: hadoop-config - name: hadoop-config
configMap: configMap:
name: {{ template "hadoop.fullname" . }} name: {{ include "hadoop.fullname" . }}
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-nm name: {{ include "hadoop.fullname" . }}-yarn-nm
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-nm component: yarn-nm
spec: spec:
ports: ports:
...@@ -19,6 +18,6 @@ spec: ...@@ -19,6 +18,6 @@ spec:
name: api name: api
clusterIP: None clusterIP: None
selector: selector:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-nm component: yarn-nm
\ No newline at end of file
apiVersion: policy/v1beta1 apiVersion: policy/v1beta1
kind: PodDisruptionBudget kind: PodDisruptionBudget
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm name: {{ include "hadoop.fullname" . }}-yarn-rm
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm component: yarn-rm
spec: spec:
selector: selector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-rm component: yarn-rm
minAvailable: {{ .Values.yarn.resourceManager.pdbMinAvailable }} minAvailable: {{ .Values.yarn.resourceManager.pdbMinAvailable }}
apiVersion: apps/v1beta1 apiVersion: apps/v1
kind: StatefulSet kind: StatefulSet
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm name: {{ include "hadoop.fullname" . }}-yarn-rm
annotations: annotations:
checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }} checksum/config: {{ include (print $.Template.BasePath "/hadoop-configmap.yaml") . | sha256sum }}
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm component: yarn-rm
spec: spec:
serviceName: {{ template "hadoop.fullname" . }}-yarn-rm selector:
matchLabels:
app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm
serviceName: {{ include "hadoop.fullname" . }}-yarn-rm
replicas: 1 replicas: 1
template: template:
metadata: metadata:
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-rm component: yarn-rm
spec: spec:
...@@ -27,7 +31,7 @@ spec: ...@@ -27,7 +31,7 @@ spec:
- topologyKey: "kubernetes.io/hostname" - topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: yarn-rm component: yarn-rm
{{- else if eq .Values.antiAffinity "soft" }} {{- else if eq .Values.antiAffinity "soft" }}
...@@ -37,15 +41,15 @@ spec: ...@@ -37,15 +41,15 @@ spec:
topologyKey: "kubernetes.io/hostname" topologyKey: "kubernetes.io/hostname"
labelSelector: labelSelector:
matchLabels: matchLabels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name | quote }} release: {{ .Release.Name | quote }}
component: yarn-rm component: yarn-rm
{{- end }} {{- end }}
terminationGracePeriodSeconds: 0 terminationGracePeriodSeconds: 0
containers: containers:
- name: yarn-rm - name: yarn-rm
image: {{ .Values.image.repository }}:{{ .Values.image.tag }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }} imagePullPolicy: {{ .Values.image.pullPolicy | quote }}
ports: ports:
- containerPort: 8088 - containerPort: 8088
name: web name: web
...@@ -73,4 +77,4 @@ spec: ...@@ -73,4 +77,4 @@ spec:
volumes: volumes:
- name: hadoop-config - name: hadoop-config
configMap: configMap:
name: {{ template "hadoop.fullname" . }} name: {{ include "hadoop.fullname" . }}
...@@ -2,12 +2,11 @@ ...@@ -2,12 +2,11 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-rm name: {{ include "hadoop.fullname" . }}-yarn-rm
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-rm component: yarn-rm
spec: spec:
ports: ports:
...@@ -15,6 +14,6 @@ spec: ...@@ -15,6 +14,6 @@ spec:
name: web name: web
clusterIP: None clusterIP: None
selector: selector:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
component: yarn-rm component: yarn-rm
\ No newline at end of file
{{- if .Values.yarn.ingress.enabled -}}
{{- $serviceName := include "hadoop.fullname" . }}
{{- $routePrefix := .Values.yarn.routePrefix }}
{{- $releaseName := .Release.Name }}
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
{{- if .Values.yarn.ingress.annotations }}
annotations:
{{ toYaml .Values.yarn.ingress.annotations | indent 4 }}
{{- end }}
labels:
app: {{ template "hadoop.fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
heritage: "{{ .Release.Service }}"
release: "{{ .Release.Name }}"
{{- if .Values.yarn.ingress.labels }}
{{ toYaml .Values.yarn.ingress.labels | indent 4 }}
{{- end }}
name: {{ template "hadoop.fullname" . }}
spec:
rules:
{{- range $host := .Values.yarn.ingress.hosts }}
- host: {{ $host }}
http:
paths:
- path: "{{ $routePrefix }}"
backend:
serviceName: {{ printf "%s-%s" $serviceName "yarn-ui" }}
servicePort: 8088
{{- end -}}
{{- if .Values.yarn.ingress.tls }}
tls:
{{ toYaml .Values.yarn.ingress.tls | indent 4 }}
{{- end }}
{{- end }}
...@@ -2,23 +2,16 @@ ...@@ -2,23 +2,16 @@
apiVersion: v1 apiVersion: v1
kind: Service kind: Service
metadata: metadata:
name: {{ template "hadoop.fullname" . }}-yarn-ui name: {{ include "hadoop.fullname" . }}-yarn-ui
labels: labels:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} chart: {{ include "hadoop.chart" . }}
release: {{ .Release.Name }} release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
component: yarn-ui component: yarn-ui
spec: spec:
type: {{ .Values.yarn.service.type }}
ports: ports:
- name: web - port: 8088
port: 8088 name: web
targetPort: web
{{- if (and (eq .Values.yarn.service.type "NodePort") (not (empty .Values.yarn.service.nodePort)))}}
nodePort: {{ .Values.yarn.service.nodePort }}
{{- end }}
selector: selector:
app: {{ template "hadoop.name" . }} app: {{ include "hadoop.name" . }}
release: {{ .Release.Name }}
component: yarn-rm component: yarn-rm
...@@ -39,7 +39,7 @@ for NODE in ${NODES}; do ...@@ -39,7 +39,7 @@ for NODE in ${NODES}; do
# Get available memory # Get available memory
AVAIL_MEM[$i]=$(jq '.node.memory.availableBytes' <<< "${NODE_STATS[$i]}") AVAIL_MEM[$i]=$(jq '.node.memory.availableBytes' <<< "${NODE_STATS[$i]}")
AVAIL_MEM[$i]=$(bc -l <<< "scale=0; ${AVAIL_MEM[$i]}/1024/1024") AVAIL_MEM[$i]=$(bc -l <<< "scale=0; ${AVAIL_MEM[$i]}/1024/1024")
# Derive available CPU # Derive available CPU
USED_CPU=$(jq '.node.cpu.usageNanoCores' <<< "${NODE_STATS[$i]}") USED_CPU=$(jq '.node.cpu.usageNanoCores' <<< "${NODE_STATS[$i]}")
AVAIL_CPU[$i]=$(bc -l <<< "scale=2; (${TOTAL_CPU} - ${USED_CPU})/1000000") AVAIL_CPU[$i]=$(bc -l <<< "scale=2; (${TOTAL_CPU} - ${USED_CPU})/1000000")
......
# The base hadoop image to use for all components. # The base hadoop image to use for all components.
# See this repo for image build details: https://github.com/Comcast/kube-yarn/tree/master/image # See this repo for image build details: https://github.com/Comcast/kube-yarn/tree/master/image
image: image:
repository: danisla/hadoop repository: ranchercharts/danisla-hadoop
tag: 2.7.3 tag: 2.9.0
pullPolicy: IfNotPresent pullPolicy: IfNotPresent
# Select antiAffinity as either hard or soft, default is hard # The version of the hadoop libraries being used in the image.
antiAffinity: "soft" hadoopVersion: 2.9.0
terminationGracePeriodSeconds: 30 # Duration in seconds a pod needs to terminate gracefully. # Select antiAffinity as either hard or soft, default is soft
antiAffinity: "soft"
hdfs: hdfs:
nameNode: nameNode:
...@@ -35,6 +36,9 @@ hdfs: ...@@ -35,6 +36,9 @@ hdfs:
memory: "2048Mi" memory: "2048Mi"
cpu: "1000m" cpu: "1000m"
webhdfs:
enabled: false
yarn: yarn:
resourceManager: resourceManager:
pdbMinAvailable: 1 pdbMinAvailable: 1
...@@ -66,40 +70,6 @@ yarn: ...@@ -66,40 +70,6 @@ yarn:
memory: "2048Mi" memory: "2048Mi"
cpu: "1000m" cpu: "1000m"
service:
type: ClusterIP
# nodePort: 32000 #range form 32000-32767
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
persistence: persistence:
nameNode: nameNode:
enabled: false enabled: false
...@@ -112,94 +82,3 @@ persistence: ...@@ -112,94 +82,3 @@ persistence:
storageClass: "-" storageClass: "-"
accessMode: ReadWriteOnce accessMode: ReadWriteOnce
size: 200Gi size: 200Gi
zeppelin:
enabled: true
#zeppelin configurations
zeppelin:
replicas: 1
image:
repository: dylanmei/zeppelin
tag: 0.7.2
resources:
limits:
memory: "4096Mi"
cpu: "2000m"
requests:
memory: "512Mi"
cpu: "200m"
hadoop:
useConfigMap: true
# configMapName: hadoop-hadoop
configPath: /usr/hadoop-2.7.3/etc/hadoop
spark:
driverMemory: 1g
executorMemory: 1g
numExecutors: 2
ingress:
## If true, Grafana Ingress will be created
##
enabled: false
## Annotations for Alertmanager Ingress
##
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
## Labels to be added to the Ingress
##
labels: {}
## Hostnames.
## Must be provided if Ingress is enabled.
##
# hosts:
# - grafana.domain.com
hosts: []
## TLS configuration for Alertmanager Ingress
## Secret must be manually created in the namespace
##
tls: []
# - secretName: alertmanager-general-tls
# hosts:
# - alertmanager.example.com
service:
## Annotations to be added to the Service
##
annotations: {}
## Cluster-internal IP address for Alertmanager Service
##
clusterIP: ""
## List of external IP addresses at which the Alertmanager Service will be available
##
externalIPs: []
## External IP address to assign to Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerIP: ""
## List of client IPs allowed to access Alertmanager Service
## Only used if service.type is 'LoadBalancer' and supported by cloud provider
##
loadBalancerSourceRanges: []
## Port to expose on each node
## Only used if service.type is 'NodePort'
##
# nodePort: 30902
## Service type
##
type: ClusterIP
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment