Kubernetes 部署 Jenkins

一: 环境准备

1:系统环境

# uname -a

Linux node-16 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release

CentOS Linux release 7.7.1908 (Core)

# kubectl get nodes

NAME STATUS ROLES AGE VERSION

node-1 Ready master 49d v1.16.0

node-1 Ready worker 49d v1.16.0

# docker version

Client: Docker Engine - Community

Version: 18.09.6

API version: 1.39

Go version: go1.10.8

Git commit: 481bc77

Built: Sat May 4 02:33:34 2019

OS/Arch: linux/amd64

Experimental: false

Server: Docker Engine - Community

Engine:

Version: 18.09.6

API version: 1.39 (minimum version 1.12)

Go version: go1.10.8

Git commit: 481bc77

Built: Sat May 4 02:41:08 2019

OS/Arch: linux/amd64

Experimental: false

# docker images|grep jenkins

jenkins/jenkins lts 5d1103b93f92 12 days ago 656MB

2:下载源代码

# git clone https://github.com/my-dlq/blog-example.git

Cloning into 'blog-example'...

remote: Enumerating objects: 752, done.

remote: Counting objects: 100% (752/752), done.

remote: Compressing objects: 100% (478/478), done.

remote: Total 1954 (delta 285), reused 618 (delta 171), pack-reused 1202

Receiving objects: 100% (1954/1954), 390.83 KiB | 4.00 KiB/s, done.

Resolving deltas: 100% (729/729), done.

# pwd

/root/software/blog-example/jenkins

# ls

jenkins-api-demo jenkins-ci&cd jenkins-deploy

二：存储pv/pvc准备

在 Kubenetes 环境下所起的应用都是一个个 Docker 镜像，为了保证应用重启的情况数据安全，所以需要将 Jenkins 持久化到存储中。这里用的是 NFS 网路存储，方便在 Kubernetes 环境下应用启动节点转义数据一致。当然也可以选择存储到本地，cent 等存储，但是为了保证应用数据一致，需要设置 Jenkins 固定到某一 Kubernetes 节点。

准备一台NFS服务器

先准备一台NFS服务器为K8S提供存储支持。

# yum install nfs-utils

# rpm -qa|grep nfs

libnfsidmap-0.25-19.el7.x86_64

nfs-utils-1.3.0-0.65.el7.x86_64

# cat /etc/exports

/ifs/kubernetes *(rw,sync,no_root_squash)

/ifs/pv001 *(rw,no_root_squash)

/ifs/pv002 *(rw,no_root_squash)

/ifs/pv003 *(rw,no_root_squash)

/ifs/pv004 *(rw,no_root_squash)

/ifs/pv005 *(rw,no_root_squash)

# systemctl start nfs

# systemctl enable nfs

nfs-client-provisioner插件

并且要在每个Node上安装nfs-utils包，用于mount挂载时用。

由于K8S不支持NFS动态供给，还需要先安装上图中的nfs-client-provisioner插件：

这个插件主要的功能是为nfs 创建PV目录和pv关链起来;

# cat deployment.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

---

kind: Deployment

apiVersion: apps/v1

metadata:

spec:

replicas: 1

strategy:

type: Recreate

selector:

matchLabels:

app: nfs-client-provisioner

template:

metadata:

labels:

app: nfs-client-provisioner

spec:

serviceAccountName: nfs-client-provisioner

containers:

- name: nfs-client-provisioner

image: quay.io/external_storage/nfs-client-provisioner:latest

volumeMounts:

- name: nfs-client-root

mountPath: /persistentvolumes

env:

- name: PROVISIONER_NAME

value: fuseim.pri/ifs

- name: NFS_SERVER

value: 100.100.100.16

- name: NFS_PATH

value: /ifs/kubernetes

volumes:

- name: nfs-client-root

nfs:

server: 100.100.100.16

path: /ifs/kubernetes

# cat class.yaml

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata:

provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'

parameters:

archiveOnDelete: "true"

# kubectl apply -f .

# kubectl get deployment|grep nfs-client-provisioner

nfs-client-provisioner 1/1 1 1 42d

# kubectl get sc|grep managed-nfs-storage

managed-nfs-storage fuseim.pri/ifs 42d

# kubectl get pods |grep nfs-client

nfs-client-provisioner-657b74d847-9k7zz 1/1 Running 5 42d

登录nfs pod 检查nfs挂载状态

# kubectl exec -it nfs-client-provisioner-657b74d847-9k7zz sh

/ #

/ # df -Th

Filesystem Type Size Used Available Use% Mounted on

overlay overlay 48.9G 18.7G 30.2G 38% /

tmpfs tmpfs 64.0M 0 64.0M 0% /dev

tmpfs tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup

100.100.100.16:/ifs/kubernetes

nfs4 48.9G 39.4G 9.5G 81% /persistentvolumes

创建PV/PVC

创建 PV 绑定 NFS 创建的 Jenkins 目录，然后创建 PVC 绑定这个 PV，将此 PVC 用于后面创建 Jenkins 服务时挂载的存储。

# cat jenkins-pv-pvc.yaml

apiVersion: v1

kind: PersistentVolume

metadata:

labels:

app: jenkins

spec:

capacity:

storage: 1Gi

accessModes:

- ReadWriteOnce

persistentVolumeReclaimPolicy: Retain

mountOptions: #NFS挂在选项

- hard

- nfsvers=4.1

nfs: #NFS设置

path: /nfs/data/jenkins

server: 100.100.100.16

---

kind: PersistentVolumeClaim

apiVersion: v1

metadata:

spec:

# storageClassName: "managed-nfs-storage"

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 1Gi #生产环境空间一定要设置比较大点

selector:

matchLabels:

app: jenkins

# kubectl apply -f jenkins-pv-pvc.yaml

创建 ServiceAccount & ClusterRoleBinding

此 kubernetes 集群用的是 RBAC 安全插件，必须创建权限给一个 ServiceAccount，然后将此 ServiceAccount 绑定到 Jenkins 服务，这样赋予 Jenkins 服务一定权限执行一些操作，为了方便，这里将 cluster-admin 绑定到 ServiceAccount 以保证 Jenkins 能拥有一定的权限。

注意：请提前修改 yaml 中的 namespace

# cat jenkins-rbac.yaml

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: default #指定namespace，一定要修改成你自己的namespace

labels:

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

labels:

subjects:

- kind: ServiceAccount

namespace: default

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

# kubectl create -f jenkins-rbac.yaml

三: jenkins部署

拉取jenkins源

node1# docker pull jenkins/jenkins:lts

node2# docker pull jenkins/jenkins:lts

node3# docker pull jenkins/jenkins:lts

[[email protected] ~]## docker images|grep jenkins

jenkins/jenkins lts 5d1103b93f92 10 days ago 656MB

# docker images|grep jenkins

jenkins/jenkins lts 5d1103b93f92 10 days ago 656MB

[[email protected] ~]#

[[email protected] ~]# docker images|grep jenkins

jenkins/jenkins lts 5d1103b93f92 10 days ago 656MB

[[email protected] ~]#

创建 Service & Deployment

这里开始部署 Jenkins 服务，创建 Service 与 Deployment，其中 Service 暴露两个接口 80880 与 50000。而 Deployment 里面要注意的是要设置上面创建的 ServiceAccount ，并且设置容器安全策略为“runAsUser: 0”以 Root 权限运行容器，而且暴露8080、50000两个端口。

创建 Service & Deployment 部署文件

# cat jenkins-deployment.yaml

apiVersion: v1

kind: Service

metadata:

labels:

app: jenkins

spec:

type: NodePort

ports:

- name: http

port: 8080 #服务端口

targetPort: 8080

nodePort: 32001 #NodePort方式暴露 Jenkins 端口

- name: jnlp

port: 50000 #代理端口

targetPort: 50000

nodePort: 32002

selector:

app: jenkins

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: jenkins

spec:

selector:

matchLabels:

app: jenkins

replicas: 1

template:

metadata:

labels:

app: jenkins

spec:

serviceAccountName: jenkins-admin

containers:

- name: jenkins

image: jenkins/jenkins:lts

securityContext:

runAsUser: 0 #设置以ROOT用户运行容器

privileged: true #拥有特权

ports:

- name: http

containerPort: 8080

- name: jnlp

containerPort: 50000

resources:

limits:

memory: 2Gi

cpu: "1000m"

requests:

memory: 1Gi

cpu: "500m"

env:

- name: LIMITS_MEMORY

valueFrom:

resourceFieldRef:

resource: limits.memory

divisor: 1Mi

- name: "JAVA_TOOL_OPTIONS"

value: "

-Dfile.encoding=UTF-8

-Dsun.jnu.encoding=UTF-8

- name: "JAVA_OPTS" #设置变量，指定时区和 jenkins slave 执行者设置

value: "

-Xmx$(LIMITS_MEMORY)m

-XshowSettings:vm

-Dhudson.slaves.NodeProvisioner.initialDelay=0

-Dhudson.slaves.NodeProvisioner.MARGIN=50

-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85

-Duser.timezone=Asia/Shanghai

- name: "JENKINS_OPTS"

value: "--prefix=/jenkins" #设置路径前缀加上 Jenkins

volumeMounts: #设置要挂在的目录

- name: data

mountPath: /var/jenkins_home

volumes:

- name: data

persistentVolumeClaim:

claimName: jenkins #设置PVC

# kubectl apply -f jenkins-deployment.yaml

service/jenkins created

deployment.apps/jenkins created

persistentvolumeclaim/jenkins created

检查运行结果

# kubectl get service|grep jenkins

jenkins NodePort 10.0.0.130 <none> 8080:32001/TCP,50000:32002/TCP 2m24s

# kubectl get deployment|grep jenkins

jenkins 1/1 1 1 2m38s

# kubectl get pod|grep jenkins

jenkins-677485858c-gsv8q 1/1 Running 0 2m47s

# kubectl get ep|grep jenkins

jenkins 10.244.2.84:8080,10.244.2.84:50000 3m45s

要查pvc/pv

# kubectl get pv|grep jenkins

pvc-249c5e67-2711-4bb5-a625-39030efe77d1 2Gi RWO Delete Bound default/jenkins managed-nfs-storage 18s

# kubectl get pvc|grep jenkins

jenkins Bound pvc-249c5e67-2711-4bb5-a625-39030efe77d1 2Gi RWO managed-nfs-storage 28s

# kubectl get sc

NAME PROVISIONER AGE

dynamic-cephfs ceph.com/cephfs 14d

managed-nfs-storage fuseim.pri/ifs 45d

nginx.storageclass ceph.com/rbd 47d

检查jenkins pod的日志

# kubectl logs jenkins-578dfd9b55-dqd58 -f

查看日志中生成的 Token 字符串

查看日志，默认给的token为：

*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.

Please use the following password to proceed to installation:

c76db77497a843eeabedc8af565d2304

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************

2020-06-08 07:46:57.877+0000 [id=39] INFO h.m.DownloadService$Downloadable#load: Obtained the updated data file for hudson.tasks.Maven.MavenInstaller

2020-06-08 07:46:57.877+0000 [id=39] INFO hudson.util.Retrier#start: Performed the action check updates server successfully at the attempt #1

2020-06-08 07:46:57.885+0000 [id=39] INFO hudson.model.AsyncPeriodicWork#lambda$doRun$0: Finished Download metadata. 148,705 ms

2020-06-08 07:47:01.066+0000 [id=26] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization

2020-06-08 07:47:01.080+0000 [id=19] INFO hudson.WebAppMain$3#run: Jenkins is fully up and running

# kubectl get pod

NAME READY STATUS RESTARTS AGE

cephfs01-deployment-78b9d99fcf-pzghb 1/1 Running 0 13d

jenkins-578dfd9b55-nbffb 1/1 Running 0 3m19s

参数说明：

JAVA_TOOL_OPTIONS：定义 JVM 环境变量，这里主要是设置编码参数，防止中文乱码
JAVA_OPTS： JVM 参数设置
JENKINS_OPTS： Jenkins 参数设置
设置执行任务时候不等待：

默认情况下，Jenkins生成代理是保守的。例如，如果队列中有两个构建，它不会立即生成两个执行器。它将生成一个执行器，并等待某个时间释放第一个执行器，然后再决定生成第二个执行器。Jenkins确保它生成的每个执行器都得到了最大限度的利用。如果你想覆盖这个行为，并生成一个执行器为每个构建队列立即不等待，所以在Jenkins启动时候添加这些参数:

-Dhudson.slaves.NodeProvisioner.initialDelay=0
-Dhudson.slaves.NodeProvisioner.MARGIN=50
-Dhudson.slaves.NodeProvisioner.MARGIN0=0.85