编辑
2025-04-11
DevOps
00

目录

背景
jenkins 部署
helm 仓库
资源准备
安装
jenkins 配置
安装必要插件
设置 podTemplate
添加 git 认证密钥
设置 gitlab 插件
关闭 Crumb 保护
流水线配置

背景

因为工作需要,需要在一台新集群上部署 jenkins + k8s agent 并结合 gitlab 做一下自动流水线,最后也发部到 k8s 中。

jenkins 部署

helm 仓库

直接使用官方 helm chart 即可,首先添加 helm 仓库

shell
helm repo add jenkinsci https://charts.jenkins.io helm repo update

资源准备

然后创建好 ns

shell
kubectl create ns jenkins

接下来提前准备好 pvc :

yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-pv-claim namespace: jenkins spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-workspace-pv-claim namespace: jenkins spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi

RBAC 定义(用来读取 k8s 的 secret,给动态生成的 runner pod 挂载):

yaml
# role-jenkins-all-secrets-reader.yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins-all-secrets-reader namespace: jenkins rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] # 授予 get 和 list 权限 --- # rolebinding-jenkins-all-secrets-reader.yaml kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: jenkins-default-read-all-secrets namespace: jenkins subjects: - kind: ServiceAccount name: default namespace: jenkins roleRef: kind: Role name: jenkins-all-secrets-reader # 引用上面的 Role apiGroup: rbac.authorization.k8s.io

然后准备好 buildkit 的本地缓存 pvc:

yam
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: jenkins-buildkit-pv-claim namespace: jenkins spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi

最后创建一个 docker 认证密钥

yaml
apiVersion: v1 data: .dockerconfigjson: xxxxxxx kind: Secret metadata: name: docker-hub-mereith namespace: jenkins type: kubernetes.io/dockerconfigjson

.dockerconfigjson 是 base64 后的,其具体格式为:

json
{ "auths": { "https://index.docker.io/v1/": { "auth": "xxxxxx" } } }

其中 auth 也是一个 base64 编码的,解码后是 username:password 格式的

然后再准备一个 k8s 集群的 config 的 secret,CD 的时候用:

yaml
apiVersion: v1 data: config: <base64 kubeconfig,可以是多上下文的> kind: Secret metadata: name: kubeconfig namespace: jenkins type: Opaque

这些都准备好后,都 apply 一下让他们生效。

安装

然后开始正式安装 jenkins

shell
helm install jenkins -n jenkins --set persistence.enabled=true --set persistence.existingClaim=jenkins-pv-claim jenkinsci/jenkins

安装完之后需要创建一个 ingress,就不再赘述了。

jenkins 配置

安装完成后,需要进行一些配置

安装必要插件

chinese: 中文插件 gitlab: 和 gitlab 集成,接收 webhook 来触发流水线。 Stage View: 展示一个直观的流水线视图。 AnsiColor: 展示流水线 console 输出的颜色。

设置 podTemplate

podTmeplate 是 jenkins 启动任务时,会启动一个 pod 作为任务执行的载体,我们需要对他进行一些设置。

系统管理 -> cloud -> k8s -> podTemplate -> default 中:

  • 最下面的 工作空间卷 设置,选择 pvc,然后输出我们最开始创建的 jenkins-workspace-pv-claim
  • 找到 Raw YAML for the Pod,输入:
yaml
apiVersion: v1 kind: Pod metadata: name: kaniko spec: containers: - name: kubectl image: lachlanevenson/k8s-kubectl:v1.25.4 imagePullPolicy: "IfNotPresent" command: - cat tty: true volumeMounts: - name: kube-config mountPath: /root/.kube defaultMode: 0600 - name: buildkit image: moby/buildkit:master imagePullPolicy: "IfNotPresent" command: ["cat"] tty: true securityContext: privileged: true volumeMounts: - name: buildkit-secret mountPath: /.docker - name: buildkit-cache mountPath: /cache env: - name: DOCKER_CONFIG value: /.docker restartPolicy: Never volumes: - name: buildkit-secret secret: secretName: docker-hub-mereith items: - key: .dockerconfigjson path: config.json - name: kaniko-cache persistentVolumeClaim: claimName: jenkins-buildkit-pv-claim - name: kube-config secret: secretName: kubeconfig items: - key: config path: config

这个 Raw YAML for the Pod 会合并到默认的模版中,我们添加的文件,会增加两个容器,一个打包用的,一个是发不到 k8s 中。

添加 git 认证密钥

找到 系统管理 -> 凭据管理 -> System -> 全局凭据 -> 添加

  • 加一个 gitlab api token,去 gitlab 个人账号创建就行。
  • 加一个 username and password,用来认证 git 仓库,当然你用 ssh 也行。

设置 gitlab 插件

找到 系统管理 -> 系统配置 -> GitLab

  • 添加一个 Connection,密钥就用刚才创建的凭据。

关闭 Crumb 保护

找到 系统管理 -> 脚本命令行:

  • 运行 hudson.security.csrf.GlobalCrumbIssuerConfiguration.DISABLE_CSRF_PROTECTION = true 关闭同理。 不关也行,按需调整。

流水线配置

新增一个 pipeline 类型的流水线,设置中的 触发器Gitlab,然后按需设置,最后展开 Advanced,生成一个 secret,最上面有一个 webhook 地址。 去 gitlab 项目中的 webhook 设置里添加一个,地址和 secret 都用这个就行。

另外 gitlab 插件默认注入了一些环境变量可以用:

shell
gitlabBranch gitlabSourceBranch gitlabActionType gitlabUserName gitlabUserUsername gitlabUserEmail gitlabSourceRepoHomepage gitlabSourceRepoName gitlabSourceNamespace gitlabSourceRepoURL gitlabSourceRepoSshUrl gitlabSourceRepoHttpUrl gitlabMergeCommitSha gitlabMergeRequestTitle gitlabMergeRequestDescription gitlabMergeRequestId gitlabMergeRequestIid gitlabMergeRequestState gitlabMergedByUser gitlabMergeRequestAssignee gitlabMergeRequestLastCommit gitlabMergeRequestTargetProjectId gitlabMergeRequestLabels gitlabTargetBranch gitlabTargetRepoName gitlabTargetNamespace gitlabTargetRepoSshUrl gitlabTargetRepoHttpUrl gitlabBefore gitlabAfter gitlabTriggerPhrase

最后提供一个 pipeline 模版,集成了飞书通知:

jenkins
pipeline { agent { kubernetes { inheritFrom 'default' // 确保 'default' Pod 模板存在且包含 'kaniko' (和 'kubectl') 容器 } } options { ansiColor('xterm') timeout(time: 20, unit: 'MINUTES') // 全局超时设置 } // --- 参数优化 --- // 只定义一个手动触发所需的分支参数 parameters { string(name: 'BRANCH_TO_BUILD', defaultValue: '0414', description: '要构建和部署的 Git 分支') } environment { REGISTRY = "mereith/test" IMAGE_NAME = "test" DEPLOYMENT_NAME = "test" CONTAINER_NAME = "test" K8S_NAMESPACE = "app" K8S_CONTEXT = "test" GIT_REPO_URL = "https://gitlab.example.com/test" GIT_CREDENTIALS_ID = "test" LARK_WEBHOOK_URL = "xxxx" // 替换为你的飞书 Webhook URL BUILD_START_TIME = "${System.currentTimeMillis()}" // 记录构建开始时间 // GitLab 触发时使用的分支名 BRANCH_NAME = "${env.gitlabBranch ?: params.BRANCH_TO_BUILD}" // --- IMAGE_TAG 计算调整 --- // 分支名部分,基于输入参数进行清理 SAFE_BRANCH_NAME = "${env.BRANCH_NAME ?: 'unknown-branch'}".replaceAll('[^a-zA-Z0-9.-]', '-') // Commit SHA 部分将在 checkout 后获取,完整的 IMAGE_TAG 将在 Build 阶段构造 // 默认设置初始阶段 CURRENT_STAGE_NAME = "初始化阶段" } stages { stage('Send Build Triggered Notification') { steps { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName echo "当前阶段: ${stageName}" echo "发送构建触发通知" // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'} **目标分支:** ${env.gitlabTargetBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 对特殊字符进行处理,避免JSON格式问题 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') // 发送构建已触发的通知到飞书 def triggeredMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "🚀 ${jobName} - 构建进行中" }, "template": "blue" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**状态:** 构建进行中...\\n**分支:** ${branchName}\\n**构建编号:** #${env.BUILD_NUMBER}\\n**触发时间:** ${shanghaiTime}${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "hr" }, { "tag": "div", "text": { "tag": "lark_md", "content": "**构建详情:** [查看Jenkins任务](${buildUrl}/console)" } } ] } }""" try { // 使用临时文件发送消息,避免命令行参数长度和转义问题 writeFile file: 'lark-message.json', text: triggeredMessage sh "curl -s -X POST -H 'Content-Type: application/json' -d @lark-message.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-message.json" } catch (Exception e) { env.FAILURE_REASON = "发送构建触发通知失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } stage('Checkout Source Code') { steps { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName echo "当前阶段: ${stageName}" try { // 设置较大的 buffer 以解决大型代码库的传输问题 def bufferSize = 524288000 echo "Setting git config --global http.postBuffer to ${bufferSize}" sh "git config --global http.postBuffer ${bufferSize}" // 使用浅克隆加速检出过程 checkout scm: [$class: 'GitSCM', userRemoteConfigs: [[url: env.GIT_REPO_URL, credentialsId: env.GIT_CREDENTIALS_ID]], branches: [[name: env.BRANCH_NAME]], extensions: [ [$class: 'CloneOption', shallow: true, noTags: true, depth: 1 ] ] ] // 获取实际检出的 Commit SHA env.CHECKED_OUT_COMMIT = sh(script: 'git rev-parse HEAD', returnStdout: true).trim() if (env.CHECKED_OUT_COMMIT.isEmpty()) { env.FAILURE_REASON = "检出代码后无法获取 Commit SHA。" env.FAILED_STAGE = stageName error(env.FAILURE_REASON) } echo "成功检出分支 '${env.BRANCH_NAME}',Commit SHA: ${env.CHECKED_OUT_COMMIT}" // 获取最新提交者和提交信息 env.LATEST_COMMIT_BY = sh(script: 'git log -1 --pretty=format:"%an"', returnStdout: true).trim() env.LATEST_COMMIT_MESSAGE = sh(script: 'git log -1 --pretty=format:"%s"', returnStdout: true).trim() echo "最新提交者: ${env.LATEST_COMMIT_BY}" echo "最新提交信息: ${env.LATEST_COMMIT_MESSAGE}" // 构造最终的 IMAGE_TAG def commitShaShort = env.CHECKED_OUT_COMMIT.take(8) env.FINAL_IMAGE_TAG = "${env.SAFE_BRANCH_NAME}-${commitShaShort}" echo "计算出的镜像 Tag (FINAL_IMAGE_TAG): ${env.FINAL_IMAGE_TAG}" } catch (Exception e) { env.FAILURE_REASON = "代码检出失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } stage('Build and Push Docker Image') { steps { container('buildkit') { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName env.LAST_ACTIVE_STAGE = stageName echo "当前阶段: ${stageName}" try { // 检查必要的环境变量是否已设置 if (env.FINAL_IMAGE_TAG == null || env.FINAL_IMAGE_TAG.isEmpty()) { env.FAILURE_REASON = "FINAL_IMAGE_TAG 未设置,无法构建镜像。" env.FAILED_STAGE = stageName error(env.FAILURE_REASON) } def workspace = pwd() echo "当前目录为 ${workspace}" echo "开始使用 buildkit 构建镜像: ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG}" // 启用缓存使用以加速构建 sh """ buildctl-daemonless.sh build --local context=${workspace} \\ --local dockerfile=${workspace} \\ --frontend dockerfile.v0 \\ --output type=image,name=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG},push=true \\ --import-cache type=local,src=/cache \\ --export-cache type=local,dest=/cache """ echo "镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 构建并推送成功。" } catch (Exception e) { env.FAILURE_REASON = "Docker镜像构建或推送失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } } stage('Deploy to Kubernetes And Wait Done') { steps { container('kubectl') { script { // 记录当前阶段 def stageName = STAGE_NAME env.CURRENT_STAGE_NAME = stageName env.LAST_ACTIVE_STAGE = stageName echo "当前阶段: ${stageName}" try { // 添加诊断信息 try { sh "kubectl version --context ${env.K8S_CONTEXT} --short" } catch (Exception e) { echo "kubectl 版本获取失败,可能是连接或权限问题: ${e.message}" } echo "开始部署镜像 ${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} 到 Deployment ${env.DEPLOYMENT_NAME}" sh """ kubectl set image deployment/${env.DEPLOYMENT_NAME} \\ ${env.CONTAINER_NAME}=${env.REGISTRY}/${env.IMAGE_NAME}:${env.FINAL_IMAGE_TAG} \\ -n ${env.K8S_NAMESPACE} \\ --context ${env.K8S_CONTEXT} """ // 等待部署完成 sh "kubectl --context ${env.K8S_CONTEXT} rollout status deployment/${env.DEPLOYMENT_NAME} -n ${env.K8S_NAMESPACE} --timeout=4m" echo "Deployment ${env.DEPLOYMENT_NAME} 在命名空间 ${env.K8S_NAMESPACE} 中更新成功。" } catch (Exception e) { env.FAILURE_REASON = "Kubernetes 部署超时或失败: ${e.message}" env.FAILED_STAGE = stageName throw e } } } } } } post { always { echo "Pipeline finished." } success { script { echo "Pipeline successfully completed for branch ${env.BRANCH_NAME} at commit ${env.CHECKED_OUT_COMMIT}." // 计算构建+部署耗时(秒) def endTimeMillis = System.currentTimeMillis() def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000 // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 转义特殊字符 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def commitMsg = env.LATEST_COMMIT_MESSAGE.replaceAll('"', '\\\\"') def commitBy = env.LATEST_COMMIT_BY.replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') def imageTag = env.FINAL_IMAGE_TAG.replaceAll('"', '\\\\"') // 直接发送成功消息 def successMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "✅ ${jobName} - 构建 + 部署成功" }, "template": "green" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT.take(8)}\\n**构建镜像:** ${env.REGISTRY}/${env.IMAGE_NAME}:${imageTag}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "note", "elements": [ { "tag": "plain_text", "content": "构建时间: ${shanghaiTime}" } ] } ] } }""" try { // 使用临时文件发送消息 writeFile file: 'lark-success.json', text: successMessage sh "curl -X POST -H 'Content-Type: application/json' -d @lark-success.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-success.json" } catch (Exception e) { echo "发送成功通知失败: ${e.message}" } } } failure { script { echo "Pipeline failed for branch ${env.BRANCH_NAME}." // 计算构建+部署耗时(秒) def endTimeMillis = System.currentTimeMillis() def durationSeconds = (endTimeMillis - env.BUILD_START_TIME.toLong()) / 1000 // 使用上海时区获取当前时间 def shanghaiTime = new Date().format('yyyy-MM-dd HH:mm:ss', TimeZone.getTimeZone('Asia/Shanghai')) // 获取失败的阶段信息,使用环境变量而不是通过不安全的API def failedStage = env.FAILED_STAGE ?: env.LAST_ACTIVE_STAGE ?: env.CURRENT_STAGE_NAME ?: "未知阶段" def failureReason = env.FAILURE_REASON ?: "未知原因" // 获取 GitLab 相关信息 def gitlabInfo = "" if (env.gitlabActionType) { gitlabInfo = """**GitLab 触发:** ${env.gitlabActionType} **触发用户:** ${env.gitlabUserName ?: 'N/A'} (${env.gitlabUserUsername ?: 'N/A'}) **源分支:** ${env.gitlabSourceBranch ?: 'N/A'}""" // 如果是合并请求,添加合并请求信息 if (env.gitlabMergeRequestIid) { gitlabInfo += """ **合并请求:** #${env.gitlabMergeRequestIid} - ${env.gitlabMergeRequestTitle ?: 'N/A'}""" } } // 转义特殊字符 def gitlabInfoEscaped = gitlabInfo.replaceAll('\n', '\\\\n').replaceAll('"', '\\\\"') def jobName = env.JOB_NAME.replaceAll('"', '\\\\"') def branchName = env.BRANCH_NAME.replaceAll('"', '\\\\"') def commitMsg = (env.LATEST_COMMIT_MESSAGE ?: 'N/A').replaceAll('"', '\\\\"') def commitBy = (env.LATEST_COMMIT_BY ?: 'N/A').replaceAll('"', '\\\\"') def buildUrl = env.BUILD_URL.replaceAll('"', '\\\\"') // 在消息中记录调试信息 echo "调试信息 - FAILED_STAGE: ${env.FAILED_STAGE}, CURRENT_STAGE_NAME: ${env.CURRENT_STAGE_NAME}, LAST_ACTIVE_STAGE: ${env.LAST_ACTIVE_STAGE}" // 直接发送失败消息 def failureMessage = """{ "msg_type": "interactive", "card": { "config": { "wide_screen_mode": true }, "header": { "title": { "tag": "plain_text", "content": "❌ ${jobName} - 构建失败" }, "template": "red" }, "elements": [ { "tag": "div", "text": { "tag": "lark_md", "content": "**分支:** ${branchName}\\n**提交:** ${env.CHECKED_OUT_COMMIT ? env.CHECKED_OUT_COMMIT.take(8) : 'N/A'}\\n**提交者:** ${commitBy}\\n**提交信息:** ${commitMsg}\\n**失败阶段:** ${failedStage}\\n**失败原因:** ${failureReason}\\n**构建+部署耗时:** ${durationSeconds.intValue()} 秒${gitlabInfoEscaped ? '\\n\\n' + gitlabInfoEscaped : ''}" } }, { "tag": "div", "text": { "tag": "lark_md", "content": "**查看详情:** [Jenkins构建日志](${buildUrl}/console)" } }, { "tag": "note", "elements": [ { "tag": "plain_text", "content": "构建时间: ${shanghaiTime}" } ] } ] } }""" try { // 使用临时文件发送消息 writeFile file: 'lark-failure.json', text: failureMessage sh "curl -X POST -H 'Content-Type: application/json' -d @lark-failure.json ${env.LARK_WEBHOOK_URL}" sh "rm -f lark-failure.json" } catch (Exception e) { echo "发送失败通知失败: ${e.message}" } } } } }
如果对你有用的话,可以打赏哦
打赏
ali pay
wechat pay

本文作者:mereith

本文链接:

版权声明:本博客所有文章除特别声明外,均采用 BY-NC-SA 许可协议。转载请注明出处!