因为工作需要,需要在一台新集群上部署 jenkins + k8s agent 并结合 gitlab 做一下自动流水线,最后也发部到 k8s 中。
直接使用官方 helm chart 即可,首先添加 helm 仓库
shellhelm repo add jenkinsci https://charts.jenkins.io helm repo update
然后创建好 ns
shellkubectl create ns jenkins
接下来提前准备好 pvc :
yamlapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-pv-claim
namespace: jenkins
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jenkins-workspace-pv-claim
namespace: jenkins
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
RBAC 定义(用来读取 k8s 的 secret,给动态生成的 runner pod 挂载):
yaml# role-jenkins-all-secrets-reader.yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-all-secrets-reader
namespace: jenkins
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"] # 授予 get 和 list 权限
---
# rolebinding-jenkins-all-secrets-reader.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: jenkins-default-read-all-secrets
namespace: jenkins
subjects:
- kind: ServiceAccount
name: default
namespace: jenkins
roleRef:
kind: Role
name: jenkins-all-secrets-reader # 引用上面的 Role
apiGroup: rbac.authorization.k8s.io
然后准备好 buildkit 的本地缓存 pvc:
yamapiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-buildkit-pv-claim namespace: jenkins spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi
最后创建一个 docker 认证密钥
yamlapiVersion: v1
data:
.dockerconfigjson: xxxxxxx
kind: Secret
metadata:
name: docker-hub-mereith
namespace: jenkins
type: kubernetes.io/dockerconfigjson
.dockerconfigjson
是 base64 后的,其具体格式为:
json{
"auths": {
"https://index.docker.io/v1/": {
"auth": "xxxxxx"
}
}
}
其中 auth
也是一个 base64 编码的,解码后是 username:password
格式的
然后再准备一个 k8s 集群的 config 的 secret,CD 的时候用:
yamlapiVersion: v1
data:
config: <base64 的 kubeconfig,可以是多上下文的>
kind: Secret
metadata:
name: kubeconfig
namespace: jenkins
type: Opaque
这些都准备好后,都 apply 一下让他们生效。
然后开始正式安装 jenkins
shellhelm install jenkins -n jenkins --set persistence.enabled=true --set persistence.existingClaim=jenkins-pv-claim jenkinsci/jenkins
安装完之后需要创建一个 ingress
,就不再赘述了。
安装完成后,需要进行一些配置
chinese
: 中文插件
gitlab
: 和 gitlab 集成,接收 webhook 来触发流水线。
Stage View
: 展示一个直观的流水线视图。
AnsiColor:
展示流水线 console 输出的颜色。
podTmeplate 是 jenkins 启动任务时,会启动一个 pod 作为任务执行的载体,我们需要对他进行一些设置。
在系统管理
-> cloud
-> k8s
-> podTemplate
-> default
中:
工作空间卷
设置,选择 pvc,然后输出我们最开始创建的 jenkins-workspace-pv-claim
Raw YAML for the Pod
,输入:yamlapiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kubectl
image: lachlanevenson/k8s-kubectl:v1.25.4
imagePullPolicy: "IfNotPresent"
command:
- cat
tty: true
volumeMounts:
- name: kube-config
mountPath: /root/.kube
defaultMode: 0600
- name: buildkit
image: moby/buildkit:master
imagePullPolicy: "IfNotPresent"
command: ["cat"]
tty: true
securityContext:
privileged: true
volumeMounts:
- name: buildkit-secret
mountPath: /.docker
- name: buildkit-cache
mountPath: /cache
env:
- name: DOCKER_CONFIG
value: /.docker
restartPolicy: Never
volumes:
- name: buildkit-secret
secret:
secretName: docker-hub-mereith
items:
- key: .dockerconfigjson
path: config.json
- name: kaniko-cache
persistentVolumeClaim:
claimName: jenkins-buildkit-pv-claim
- name: kube-config
secret:
secretName: kubeconfig
items:
- key: config
path: config
这个
Raw YAML for the Pod
会合并到默认的模版中,我们添加的文件,会增加两个容器,一个打包用的,一个是发不到k8s
中。
找到 系统管理
-> 凭据管理
-> System
-> 全局凭据
-> 添加
:
gitlab api token
,去 gitlab
个人账号创建就行。username and password
,用来认证 git
仓库,当然你用 ssh
也行。找到 系统管理
-> 系统配置
-> GitLab
:
找到 系统管理
-> 脚本命令行
:
hudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION = true
关闭同理。
不关也行,按需调整。新增一个 pipeline
类型的流水线,设置中的 触发器
选 Gitlab
,然后按需设置,最后展开 Advanced
,生成一个 secret
,最上面有一个 webhook
地址。
去 gitlab
项目中的 webhook
设置里添加一个,地址和 secret
都用这个就行。
另外 gitlab
插件默认注入了一些环境变量可以用:
shellgitlabBranch gitlabSourceBranch gitlabActionType gitlabUserName gitlabUserUsername gitlabUserEmail gitlabSourceRepoHomepage gitlabSourceRepoName gitlabSourceNamespace gitlabSourceRepoURL gitlabSourceRepoSshUrl gitlabSourceRepoHttpUrl gitlabMergeCommitSha gitlabMergeRequestTitle gitlabMergeRequestDescription gitlabMergeRequestId gitlabMergeRequestIid gitlabMergeRequestState gitlabMergedByUser gitlabMergeRequestAssignee gitlabMergeRequestLastCommit gitlabMergeRequestTargetProjectId gitlabMergeRequestLabels gitlabTargetBranch gitlabTargetRepoName gitlabTargetNamespace gitlabTargetRepoSshUrl gitlabTargetRepoHttpUrl gitlabBefore gitlabAfter gitlabTriggerPhrase
最后提供一个 pipeline
模版,集成了飞书通知:
jenkinspipeline { agent { kubernetes { inheritFrom 'default' // 确保 'default' Pod 模板存在且包含 'kaniko' (和 'kubectl') 容器 } } options { ansiColor('xterm') timeout(time: 20, unit: 'MINUTES') // 全局超时设置 } // --- 参数优化 --- // 只定义一个手动触发所需的分支参数 parameters { string(name: 'BRANCH_TO_BUILD', defaultValue: '0414', description: '要构建和部署的 Git 分支') } environment { REGISTRY = "mereith/test" IMAGE_NAME = "test" DEPLOYMENT_NAME = "test" CONTAINER_NAME = "test" K8S_NAMESPACE = "app" K8S_CONTEXT = "test" GIT_REPO_URL = "https://gitlab.example.com/test" GIT_CREDENTIALS_ID = "test" LARK_WEBHOOK_URL = "xxxx" // 替换为你的飞书 Webhook URL BUILD_START_TIME = "${System.currentTimeMillis()}" // 记录构建开始时间 // GitLab 触发时使用的分支名 BRANCH_NAME = "${env.gitlabBranch ?: params.BRANCH_TO_BUILD}" // --- IMAGE_TAG 计算调整 --- // 分支名部分,基于输入参数进行清理 SAFE_BRANCH_NAME = "${env.BRANCH_NAME ?: 'unknown-branch'}".replaceAll('[^a-zA-Z0-9.-]', '-') // Commit SHA 部分将在 checkout 后获取,完整的 IMAGE_TAG 将在 Build 阶段构造 // 默认设置初始阶段 CURRENT_STAGE_NAME = "初始化阶段" } stages { stage('Send Build Triggered Notification') { steps { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName echo "当前阶段: ${stageName}" echo "发送构建触发通知" // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'} **目标分支:** ${env.gitlabTargetBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 对特殊字符进行处理,避免JSON格式问题 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') // 发送构建已触发的通知到飞书 def triggeredMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "🚀 ${jobName} - 构建进行中" }, "template": "blue" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**状态:** 构建进行中...\\n**分支:** ${branchName}\\n**构建编号:** #${env.BUILD_NUMBER}\\n**触发时间:** ${shanghaiTime}${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "hr" }, { "tag": "div", "text": { "tag": "lark_md", "content": "**构建详情:** [查看Jenkins任务](${buildUrl}/console)" } } ] } }""" try { // 使用临时文件发送消息,避免命令行参数长度和转义问题 writeFile file: 'lark-message.json', text: triggeredMessage sh "curl -s -X POST -H 'Content-Type: application/json' -d @lark-message.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-message.json" } catch (Exception e) { env.FAILURE_REASON = "发送构建触发通知失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } stage('Checkout Source Code') { steps { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName echo "当前阶段: ${stageName}" try { // 设置较大的 buffer 以解决大型代码库的传输问题 def bufferSize = 524288000 echo "Setting git config --global http.postBuffer to ${bufferSize}" sh "git config --global http.postBuffer ${bufferSize}" // 使用浅克隆加速检出过程 checkout scm: [$class: 'GitSCM', userRemoteConfigs: [[url: env.GIT_REPO_URL, credentialsId: env.GIT_CREDENTIALS_ID]], branches: [[name: env.BRANCH_NAME]], extensions: [ [$class: 'CloneOption', shallow: true, noTags: true, depth: 1 ] ] ] // 获取实际检出的 Commit SHA env.CHECKED_OUT_COMMIT = sh(script: 'git rev-parse HEAD', returnStdout: true).trim() if (env.CHECKED_OUT_COMMIT.isEmpty()) { env.FAILURE_REASON = "检出代码后无法获取 Commit SHA。" env.FAILED_STAGE = stageName error(env.FAILURE_REASON) } echo "成功检出分支 '${env.BRANCH_NAME}',Commit SHA: ${env.CHECKED_OUT_COMMIT}" // 获取最新提交者和提交信息 env.LATEST_COMMIT_BY = sh(script: 'git log -1 --pretty=format:"%an"', returnStdout: true).trim() env.LATEST_COMMIT_MESSAGE = sh(script: 'git log -1 --pretty=format:"%s"', returnStdout: true).trim() echo "最新提交者: ${env.LATEST_COMMIT_BY}" echo "最新提交信息: ${env.LATEST_COMMIT_MESSAGE}" // 构造最终的 IMAGE_TAG def commitShaShort = env.CHECKED_OUT_COMMIT.take(8) env.FINAL_IMAGE_TAG = "${env.SAFE_BRANCH_NAME}-${commitShaShort}" echo "计算出的镜像 Tag (FINAL_IMAGE_TAG): ${env.FINAL_IMAGE_TAG}" } catch (Exception e) { env.FAILURE_REASON = "代码检出失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } stage('Build and Push Docker Image') { steps { container('buildkit') { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName env.LAST_ACTIVE_STAGE = stageName echo "当前阶段: ${stageName}" try { // 检查必要的环境变量是否已设置 if (env.FINAL_IMAGE_TAG == null || env.FINAL_IMAGE_TAG.isEmpty()) { env.FAILURE_REASON = "FINAL_IMAGE_TAG 未设置,无法构建镜像。" env.FAILED_STAGE = stageName error(env.FAILURE_REASON) } def workspace = pwd() echo "当前目录为 ${workspace}" echo "开始使用 buildkit 构建镜像: ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG}" // 启用缓存使用以加速构建 sh """ buildctl-daemonless.sh build --local context=${workspace} \\ --local dockerfile=${workspace} \\ --frontend dockerfile.v0 \\ --output type=image,name=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG},push=true \\ --import-cache type=local,src=/cache \\ --export-cache type=local,dest=/cache """ echo "镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 构建并推送成功。" } catch (Exception e) { env.FAILURE_REASON = "Docker镜像构建或推送失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } } stage('Deploy to Kubernetes And Wait Done') { steps { container('kubectl') { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName env.LAST_ACTIVE_STAGE = stageName echo "当前阶段: ${stageName}" try { // 添加诊断信息 try { sh "kubectl version --context ${env.K8S_CONTEXT} --short" } catch (Exception e) { echo "kubectl 版本获取失败,可能是连接或权限问题: ${e.message}" } echo "开始部署镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 到 Deployment ${env.DEPLOYMENT_NAME}" sh """ kubectl set image deployment/${env.DEPLOYMENT_NAME} \\ ${env.CONTAINER_NAME}=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} \\ -n ${env.K8S_NAMESPACE} \\ --context ${env.K8S_CONTEXT} """ // 等待部署完成 sh "kubectl --context ${env.K8S_CONTEXT} rollout status deployment/${env.DEPLOYMENT_NAME} -n ${env.K8S_NAMESPACE} --timeout=4m" echo "Deployment ${env.DEPLOYMENT_NAME} 在命名空间 ${env.K8S_NAMESPACE} 中更新成功。" } catch (Exception e) { env.FAILURE_REASON = "Kubernetes 部署超时或失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } } } post { always { echo "Pipeline finished." } success { script { echo "Pipeline successfully completed for branch ${env.BRANCH_NAME} at commit ${env.CHECKED_OUT_COMMIT}." // 计算构建+部署耗时(秒) def endTimeMillis = System.currentTimeMillis() def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000 // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 转义特殊字符 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def commitMsg = env.LATEST_COMMIT_MESSAGE.replaceAll('"', '\\\\"') def commitBy = env.LATEST_COMMIT_BY.replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') def imageTag = env.FINAL_IMAGE_TAG.replaceAll('"', '\\\\"') // 直接发送成功消息 def successMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "✅ ${jobName} - 构建 + 部署成功" }, "template": "green" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT.take(8)}\\n**构建镜像:** ${env.REGISTRY}/${env.IMAGE_NAME}:${imageTag}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "note", "elements": [ { "tag": "plain_text", "content": "构建时间: ${shanghaiTime}" } ] } ] } }""" try { // 使用临时文件发送消息 writeFile file: 'lark-success.json', text: successMessage sh "curl -X POST -H 'Content-Type: application/json' -d @lark-success.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-success.json" } catch (Exception e) { echo "发送成功通知失败: ${e.message}" } } } failure { script { echo "Pipeline failed for branch ${env.BRANCH_NAME}." // 计算构建+部署耗时(秒) def endTimeMillis = System.currentTimeMillis() def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000 // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 获取失败的阶段信息,使用环境变量而不是通过不安全的API def failedStage = env.FAILED_STAGE ?: env.LAST_ACTIVE_STAGE ?: env.CURRENT_STAGE_NAME ?: "未知阶段" def failureReason = env.FAILURE_REASON ?: "未知原因" // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 转义特殊字符 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def commitMsg = (env.LATEST_COMMIT_MESSAGE ?: 'N/A').replaceAll('"', '\\\\"') def commitBy = (env.LATEST_COMMIT_BY ?: 'N/A').replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') // 在消息中记录调试信息 echo "调试信息 - FAILED_STAGE: ${env.FAILED_STAGE}, CURRENT_STAGE_NAME: ${env.CURRENT_STAGE_NAME}, LAST_ACTIVE_STAGE: ${env.LAST_ACTIVE_STAGE}" // 直接发送失败消息 def failureMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "❌ ${jobName} - 构建失败" }, "template": "red" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT ? env.CHECKED_OUT_COMMIT.take(8) : 'N/A'}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**失败阶段:** ${failedStage}\\n**失败原因:** ${failureReason}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "div", "text": { "tag": "lark_md", "content": "**查看详情:** [Jenkins构建日志](${buildUrl}/console)" } }, { "tag": "note", "elements": [ { "tag": "plain_text", "content": "构建时间: ${shanghaiTime}" } ] } ] } }""" try { // 使用临时文件发送消息 writeFile file: 'lark-failure.json', text: failureMessage sh "curl -X POST -H 'Content-Type: application/json' -d @lark-failure.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-failure.json" } catch (Exception e) { echo "发送失败通知失败: ${e.message}" } } } } }
本文作者:mereith
本文链接:
版权声明:本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!