Install Hadoop HDFS cluster on Kubernetes running on Google Compute Engine (not GKE).
Environment:
Google Compute Engine (not GKE).
- Kubernetes-master : n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.5
- Kubernetes-worker1 : n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.6
- Kubernetes-worker2 : n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.7
- Kubernetes-worker3 : n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.8
- Centos 7 installed on all compute instance
Pre-requesties:
1. Install gcloud sdk on all compute instance.
2. Refer https://www.devopsbar.com/ for to Deploy Kubernetes cluster 1.11.3 on Centos 7 in Google Cloud Platform.
3. Create gce-user service account with Compute Instance Admin (v1) and Compute Storage Admin roles assigned.
4. Assign gce-user service account all worker compute instance and reboot. Master instance should have access to Allow full access to all Cloud APIswith default compute service account.
5. Add host entries on all compute instance:
vim /etc/hosts
10.138.0.5 kubernetes-master.us-west1-a.c.devopsbar20.internal kubernetes-master
10.138.0.6 kubernetes-worker1.us-west1-a.c.devopsbar20.internal kubernetes-worker1
10.138.0.7 kubernetes-worker2.us-west1-a.c.devopsbar20.internal kubernetes-worker2
10.138.0.8 kubernetes-worker3.us-west1-a.c.devopsbar20.internal kubernetes-worker3
10.138.0.5 kubernetes-master.us-west1-a.c.devopsbar20.internal kubernetes-master
10.138.0.6 kubernetes-worker1.us-west1-a.c.devopsbar20.internal kubernetes-worker1
10.138.0.7 kubernetes-worker2.us-west1-a.c.devopsbar20.internal kubernetes-worker2
10.138.0.8 kubernetes-worker3.us-west1-a.c.devopsbar20.internal kubernetes-worker3
6. On Master and worker node add below mentioned entries:
# vim /etc/sysconfig/kubelet
add at the end of DAEMON_ARGS string:
--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cloud-provider=gce
# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
add at the end of Environment="KUBELET_KUBECONFIG_ARGS:
--cloud-provider=gce
# vim /etc/kubernetes/cloud-config
[Global]
project-id = "<google-project-id>"
# systemctl restart kubelet
7. On Master node add cloud provider on kubeadm :
vim /etc/kubernetes/manifests/kube-controller-manager.yaml
add the following line:
- --cloud-provider=gce
# systemctl restart kubelet
8. Verify on Master node :
run the ps -ef |grep controller then you must see "gce" in controller output.
[root@kubernetes-master ~]# ps -ef |grep controller
root 1858 1840 3 01:25 ? 00:00:44 kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --node-cidr-mask-size=24 --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true --cloud-provider=gce
root 11401 11074 0 01:48 pts/0 00:00:00 grep --color=auto controller
9. Install helm package manager for Kubernetes on Master node:
Check cluster information
# kubectl cluster-info
# kubectl config get-contexts
Download helm installation script
# curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh
# chmod u+x install-helm.sh
Edit install.sh and change path:
#vim install-helm.sh
${HELM_INSTALL_DIR:="/usr//bin"}Run the installation script
# ./install-helm.sh
Create service account for helm/tiller
# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/rbac-config.yaml > rbac-config.yaml
# kubectl apply -f rbac-config.yaml
# kubectl create clusterrolebinding tiller-cluster-rule \
--clusterrole=cluster-admin --serviceaccount=kube-system:tiller
# helm init --service-account tiller --upgrade
Verify
# kubectl get pods --namespace kube-system
[root@kubernetes-master ~]# kubectl get pods --namespace kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-fkfvh 1/1 Running 2 1d
coredns-78fcdf6894-hpxg4 1/1 Running 2 1d
etcd-kubernetes-master 1/1 Running 2 1d
kube-apiserver-kubernetes-master 1/1 Running 2 1d
kube-controller-manager-kubernetes-master 1/1 Running 1 17h
kube-flannel-ds-8ss44 1/1 Running 4 22h
kube-flannel-ds-dm6fj 1/1 Running 3 1d
kube-flannel-ds-mdssr 1/1 Running 6 22h
kube-flannel-ds-mp67h 1/1 Running 1 17h
kube-proxy-dx94p 1/1 Running 4 22h
kube-proxy-mglsl 1/1 Running 1 17h
kube-proxy-n7hjx 1/1 Running 4 22h
kube-proxy-q4vth 1/1 Running 2 1d
kube-scheduler-kubernetes-master 1/1 Running 2 1d
tiller-deploy-6597c75fdd-tt8tw 1/1 Running 3 19h
10. Create storage class
# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/storageclass.yaml > storageclass.yaml
# kubectl apply -f storageclass.yaml
verify:
# kubectl get sc
NAME PROVISIONER AGE
hdfs kubernetes.io/gce-pd 6s
# kubectl describe sc
Name: hdfs
IsDefaultClass: No
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"hdfs","namespace":""},"parameters":{"replication-type":"none","type":"pd-standard"},"provisioner":"kubernetes.io/gce-pd"}
Provisioner: kubernetes.io/gce-pd
Parameters: replication-type=none,type=pd-standard
AllowVolumeExpansion: <unset>
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
11. Create disk on GCE
# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-0
# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-1
# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-2
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-0
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-1
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-2
# gcloud compute disks create --size=100GB --zone=us-west1-a hdfs-namenode-k8s-0
# gcloud compute disks create --size=100GB --zone=us-west1-a hdfs-namenode-k8s-1
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-krb5-k8s
# gcloud compute disks list
NAME LOCATION LOCATION_SCOPE SIZE_GB TYPE STATUS
ansible-client us-west1-a zone 10 pd-standard READY
ansible-controller us-west1-a zone 10 pd-standard READY
hdfs-journalnode-k8s-0 us-west1-a zone 20 pd-standard READY
hdfs-journalnode-k8s-1 us-west1-a zone 20 pd-standard READY
hdfs-journalnode-k8s-2 us-west1-a zone 20 pd-standard READY
hdfs-krb5-k8s us-west1-a zone 20 pd-standard READY
hdfs-namenode-k8s-0 us-west1-a zone 100 pd-standard READY
hdfs-namenode-k8s-1 us-west1-a zone 100 pd-standard READY
kubernetes-master us-west1-a zone 10 pd-standard READY
kubernetes-worker1 us-west1-a zone 10 pd-standard READY
kubernetes-worker2 us-west1-a zone 10 pd-standard READY
kubernetes-worker3 us-west1-a zone 10 pd-standard READY
zookeeper-0 us-west1-a zone 5 pd-standard READY
zookeeper-1 us-west1-a zone 5 pd-standard READY
zookeeper-2 us-west1-a zone 5 pd-standard READY
12. Create PersistentVolume
# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/pv.yaml > pv.yaml
# kubectl apply -f pv.yaml
verify:
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
hdfs-journalnode-k8s-0 20Gi RWO Retain Available hdfs 11s
hdfs-journalnode-k8s-1 20Gi RWO Retain Available hdfs 11s
hdfs-journalnode-k8s-2 20Gi RWO Retain Available hdfs 11s
hdfs-krb5-k8s 20Gi RWO Retain Available hdfs 11s
hdfs-namenode-k8s-0 100Gi RWO Retain Available hdfs 11s
hdfs-namenode-k8s-1 100Gi RWO Retain Available hdfs 11s
zookeeper-0 5Gi RWO Retain Available hdfs 11s
zookeeper-1 5Gi RWO Retain Available hdfs 11s
zookeeper-2 5Gi RWO Retain Available hdfs 11s
13. Clone Hadoop HDFS code from git repo
# git clone https://github.com/apache-spark-on-k8s/kubernetes-HDFS.git
14. Replace the vaules.yaml according to your environment
# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/values.yaml > values.yaml
15. copy values.yaml to kubernetes-HDFS/chars/hdfs-k8s/
# cp values.yaml kubernetes-HDFS/charts/hdfs-k8s/values.yaml
Install Hadoop HDFS on kubernetes:
# cd kubernetes-HDFS
# helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
# helm dependency build charts/hdfs-k8s
# helm install -n my-hdfs charts/hdfs-k8s
Verify installation (wait for few minutes):
# kubectl get pod -l release=my-hdfs
# kubectl get pod -l release=my-hdfs
NAME READY STATUS RESTARTS AGE
my-hdfs-client-598c44cdcb-xvvbn 1/1 Running 0 4m
my-hdfs-datanode-fvz4t 1/1 Running 1 4m
my-hdfs-datanode-m98r6 1/1 Running 1 4m
my-hdfs-datanode-w6z8m 1/1 Running 1 4m
my-hdfs-journalnode-0 1/1 Running 0 4m
my-hdfs-journalnode-1 1/1 Running 0 3m
my-hdfs-journalnode-2 1/1 Running 0 3m
my-hdfs-namenode-0 1/1 Running 5 4m
my-hdfs-namenode-1 1/1 Running 5 3m
my-hdfs-zookeeper-0 1/1 Running 0 4m
my-hdfs-zookeeper-1 1/1 Running 0 3m
my-hdfs-zookeeper-2 1/1 Running 0 2m
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-my-hdfs-zookeeper-0 Bound zookeeper-0 5Gi RWO hdfs 5m
data-my-hdfs-zookeeper-1 Bound zookeeper-1 5Gi RWO hdfs 4m
data-my-hdfs-zookeeper-2 Bound zookeeper-2 5Gi RWO hdfs 3m
editdir-my-hdfs-journalnode-0 Bound hdfs-journalnode-k8s-1 20Gi RWO hdfs 5m
editdir-my-hdfs-journalnode-1 Bound hdfs-journalnode-k8s-0 20Gi RWO hdfs 5m
editdir-my-hdfs-journalnode-2 Bound hdfs-journalnode-k8s-2 20Gi RWO hdfs 4m
metadatadir-my-hdfs-namenode-0 Bound hdfs-namenode-k8s-1 100Gi RWO hdfs 5m
metadatadir-my-hdfs-namenode-1 Bound hdfs-namenode-k8s-0 100Gi RWO hdfs 5m
# kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4h
my-hdfs-journalnode ClusterIP None <none> 8485/TCP,8480/TCP 2h
my-hdfs-namenode ClusterIP None <none> 8020/TCP,50070/TCP 2h
my-hdfs-zookeeper ClusterIP 10.102.189.41 <none> 2181/TCP 2h
my-hdfs-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP 2h
Hadoop HDFS GUI:
http://<external ip>:50070
