Install Hadoop HDFS cluster on Kubernetes running on Google Compute Engine

Summary:

Install Hadoop HDFS cluster on Kubernetes running on Google Compute Engine (not GKE).


Environment:
Google Compute Engine (not GKE).

• Kubernetes-master :  n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.5
• Kubernetes-worker1 :  n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.6
• Kubernetes-worker2 :  n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.7
• Kubernetes-worker3 :  n1-standard-2 (2 vCPUs, 7.5 GB memory)- IP: 10.138.0.8
• Centos 7 installed on all compute instance

Pre-requesties:

1. Install gcloud sdk on all compute instance.

2. Refer https://www.devopsbar.com/ for to Deploy Kubernetes cluster 1.11.3 on Centos 7 in Google Cloud Platform.

3. Create gce-user service account with Compute Instance Admin (v1) and Compute Storage Admin roles assigned.

4. Assign gce-user service account all worker compute instance and reboot. Master instance should have access to Allow full access to all Cloud APIswith default compute service account.

5. Add host entries on all compute instance:

vim /etc/hosts
10.138.0.5 kubernetes-master.us-west1-a.c.devopsbar20.internal kubernetes-master
10.138.0.6 kubernetes-worker1.us-west1-a.c.devopsbar20.internal kubernetes-worker1 
10.138.0.7 kubernetes-worker2.us-west1-a.c.devopsbar20.internal kubernetes-worker2
10.138.0.8 kubernetes-worker3.us-west1-a.c.devopsbar20.internal kubernetes-worker3

6. On Master and worker node add below mentioned entries:

# vim /etc/sysconfig/kubelet

add at the end of DAEMON_ARGS string:

 --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cloud-provider=gce

# vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

add at the end of Environment="KUBELET_KUBECONFIG_ARGS:

--cloud-provider=gce 

# vim /etc/kubernetes/cloud-config

[Global]
project-id = "<google-project-id>"

# systemctl restart kubelet


7. On Master node add cloud provider on kubeadm :

vim /etc/kubernetes/manifests/kube-controller-manager.yaml

add the following line:

- --cloud-provider=gce

# systemctl restart kubelet


8. Verify on Master node :

run the ps -ef |grep controller then you must see "gce" in controller output.

[root@kubernetes-master ~]# ps -ef |grep controller
root      1858  1840  3 01:25 ?        00:00:44 kube-controller-manager --address=127.0.0.1 --allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16 --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt --cluster-signing-key-file=/etc/kubernetes/pki/ca.key --controllers=*,bootstrapsigner,tokencleaner --kubeconfig=/etc/kubernetes/controller-manager.conf --leader-elect=true --node-cidr-mask-size=24 --root-ca-file=/etc/kubernetes/pki/ca.crt --service-account-private-key-file=/etc/kubernetes/pki/sa.key --use-service-account-credentials=true --cloud-provider=gce
root     11401 11074  0 01:48 pts/0    00:00:00 grep --color=auto controller


9. Install helm package manager for Kubernetes on Master node:

Check cluster information
 
# kubectl cluster-info 

# kubectl config get-contexts


Download helm installation script

# curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get > install-helm.sh 

# chmod u+x install-helm.sh

Edit install.sh and change path: 


#vim install-helm.sh
${HELM_INSTALL_DIR:="/usr//bin"}Run the installation script

# ./install-helm.sh 

Create service account for helm/tiller

# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/rbac-config.yaml > rbac-config.yaml

# kubectl apply -f rbac-config.yaml 


# kubectl create clusterrolebinding tiller-cluster-rule \
 --clusterrole=cluster-admin --serviceaccount=kube-system:tiller

# helm init --service-account tiller --upgrade

Verify
 
# kubectl get pods --namespace kube-system

[root@kubernetes-master ~]# kubectl get pods --namespace kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
coredns-78fcdf6894-fkfvh                    1/1     Running   2          1d
coredns-78fcdf6894-hpxg4                    1/1     Running   2          1d
etcd-kubernetes-master                      1/1     Running   2          1d
kube-apiserver-kubernetes-master            1/1     Running   2          1d
kube-controller-manager-kubernetes-master   1/1     Running   1          17h
kube-flannel-ds-8ss44                       1/1     Running   4          22h
kube-flannel-ds-dm6fj                       1/1     Running   3          1d
kube-flannel-ds-mdssr                       1/1     Running   6          22h
kube-flannel-ds-mp67h                       1/1     Running   1          17h
kube-proxy-dx94p                            1/1     Running   4          22h
kube-proxy-mglsl                            1/1     Running   1          17h
kube-proxy-n7hjx                            1/1     Running   4          22h
kube-proxy-q4vth                            1/1     Running   2          1d
kube-scheduler-kubernetes-master            1/1     Running   2          1d
tiller-deploy-6597c75fdd-tt8tw              1/1     Running   3          19h


10. Create storage class

# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/storageclass.yaml > storageclass.yaml
 
# kubectl apply -f storageclass.yaml

verify:

# kubectl get sc
NAME      PROVISIONER            AGE
hdfs      kubernetes.io/gce-pd   6s

# kubectl describe sc
Name:            hdfs
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"hdfs","namespace":""},"parameters":{"replication-type":"none","type":"pd-standard"},"provisioner":"kubernetes.io/gce-pd"}

Provisioner:           kubernetes.io/gce-pd
Parameters:            replication-type=none,type=pd-standard
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>


11. Create disk on GCE

# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-0
# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-1
# gcloud compute disks create --size=5GB --zone=us-west1-a zookeeper-2
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-0
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-1
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-journalnode-k8s-2
# gcloud compute disks create --size=100GB --zone=us-west1-a hdfs-namenode-k8s-0
# gcloud compute disks create --size=100GB --zone=us-west1-a hdfs-namenode-k8s-1
# gcloud compute disks create --size=20GB --zone=us-west1-a hdfs-krb5-k8s


# gcloud compute disks list
NAME                    LOCATION    LOCATION_SCOPE  SIZE_GB  TYPE         STATUS
ansible-client          us-west1-a  zone            10       pd-standard  READY
ansible-controller      us-west1-a  zone            10       pd-standard  READY
hdfs-journalnode-k8s-0  us-west1-a  zone            20       pd-standard  READY
hdfs-journalnode-k8s-1  us-west1-a  zone            20       pd-standard  READY
hdfs-journalnode-k8s-2  us-west1-a  zone            20       pd-standard  READY
hdfs-krb5-k8s           us-west1-a  zone            20       pd-standard  READY
hdfs-namenode-k8s-0     us-west1-a  zone            100      pd-standard  READY
hdfs-namenode-k8s-1     us-west1-a  zone            100      pd-standard  READY
kubernetes-master       us-west1-a  zone            10       pd-standard  READY
kubernetes-worker1      us-west1-a  zone            10       pd-standard  READY
kubernetes-worker2      us-west1-a  zone            10       pd-standard  READY
kubernetes-worker3      us-west1-a  zone            10       pd-standard  READY
zookeeper-0             us-west1-a  zone            5        pd-standard  READY
zookeeper-1             us-west1-a  zone            5        pd-standard  READY
zookeeper-2             us-west1-a  zone            5        pd-standard  READY



12. Create PersistentVolume

# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/pv.yaml > pv.yaml

# kubectl apply -f pv.yaml

verify:

# kubectl get pv
NAME                     CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM     STORAGECLASS   REASON    AGE
hdfs-journalnode-k8s-0   20Gi       RWO            Retain           Available             hdfs                     11s
hdfs-journalnode-k8s-1   20Gi       RWO            Retain           Available             hdfs                     11s
hdfs-journalnode-k8s-2   20Gi       RWO            Retain           Available             hdfs                     11s
hdfs-krb5-k8s            20Gi       RWO            Retain           Available             hdfs                     11s
hdfs-namenode-k8s-0      100Gi      RWO            Retain           Available             hdfs                     11s
hdfs-namenode-k8s-1      100Gi      RWO            Retain           Available             hdfs                     11s
zookeeper-0              5Gi        RWO            Retain           Available             hdfs                     11s
zookeeper-1              5Gi        RWO            Retain           Available             hdfs                     11s
zookeeper-2              5Gi        RWO            Retain           Available             hdfs                     11s



13. Clone Hadoop HDFS code from git repo


# git clone https://github.com/apache-spark-on-k8s/kubernetes-HDFS.git


14. Replace the vaules.yaml according to your environment

# curl https://raw.githubusercontent.com/vdsridevops/hdfs-kubernetes/master/values.yaml > values.yaml

15. copy values.yaml to kubernetes-HDFS/chars/hdfs-k8s/

# cp values.yaml kubernetes-HDFS/charts/hdfs-k8s/values.yaml 


Install Hadoop HDFS on kubernetes:

# cd kubernetes-HDFS

# helm repo add incubator        https://kubernetes-charts-incubator.storage.googleapis.com/

# helm dependency build charts/hdfs-k8s

# helm install -n my-hdfs charts/hdfs-k8s

Verify installation (wait for few minutes):

# kubectl get pod -l release=my-hdfs


# kubectl get pod -l release=my-hdfs

NAME                              READY     STATUS    RESTARTS   AGE
my-hdfs-client-598c44cdcb-xvvbn   1/1       Running   0          4m
my-hdfs-datanode-fvz4t            1/1       Running   1          4m
my-hdfs-datanode-m98r6            1/1       Running   1          4m
my-hdfs-datanode-w6z8m            1/1       Running   1          4m
my-hdfs-journalnode-0             1/1       Running   0          4m
my-hdfs-journalnode-1             1/1       Running   0          3m
my-hdfs-journalnode-2             1/1       Running   0          3m
my-hdfs-namenode-0                1/1       Running   5          4m
my-hdfs-namenode-1                1/1       Running   5          3m
my-hdfs-zookeeper-0               1/1       Running   0          4m
my-hdfs-zookeeper-1               1/1       Running   0          3m
my-hdfs-zookeeper-2               1/1       Running   0          2m 

# kubectl get pvc
NAME                             STATUS    VOLUME                   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-my-hdfs-zookeeper-0         Bound     zookeeper-0              5Gi        RWO            hdfs           5m
data-my-hdfs-zookeeper-1         Bound     zookeeper-1              5Gi        RWO            hdfs           4m
data-my-hdfs-zookeeper-2         Bound     zookeeper-2              5Gi        RWO            hdfs           3m
editdir-my-hdfs-journalnode-0    Bound     hdfs-journalnode-k8s-1   20Gi       RWO            hdfs           5m
editdir-my-hdfs-journalnode-1    Bound     hdfs-journalnode-k8s-0   20Gi       RWO            hdfs           5m
editdir-my-hdfs-journalnode-2    Bound     hdfs-journalnode-k8s-2   20Gi       RWO            hdfs           4m
metadatadir-my-hdfs-namenode-0   Bound     hdfs-namenode-k8s-1      100Gi      RWO            hdfs           5m
metadatadir-my-hdfs-namenode-1   Bound     hdfs-namenode-k8s-0      100Gi      RWO            hdfs           5m 

# kubectl get services
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
kubernetes                   ClusterIP   10.96.0.1       <none>        443/TCP                      4h
my-hdfs-journalnode          ClusterIP   None            <none>        8485/TCP,8480/TCP            2h
my-hdfs-namenode             ClusterIP   None            <none>        8020/TCP,50070/TCP           2h
my-hdfs-zookeeper            ClusterIP   10.102.189.41   <none>        2181/TCP                     2h
my-hdfs-zookeeper-headless   ClusterIP   None            <none>        2181/TCP,3888/TCP,2888/TCP   2h

Hadoop HDFS GUI:

http://<external ip>:50070