0 介紹

因為非常好奇通常怎麼在Kubernetes環境中建置CI/CD環境,因此我決定自己動手試試看,這篇文章將會介紹如何在Kubernetes中建置CI/CD環境,我們將會使用GitLab CI來建置CI環境,並且使用Argo CD來建置CD環境。

前提概要

在開始本篇文章前,我們假設你已經間至少以下環境:

  1. Kubernetes 集群中的多個節點
  2. 已經安裝好 GitLab
  3. 已經在 Kubernetes 中建置好GitLKab Runner,詳細可以參考GitLab Runner Helm Chart

GitLab, Harbor 建置

如果 2 還沒建置好,可以參考以下連結,會教你如何建置Harbor、GitLab:https://github.com/ShannonHung/VMWare-CICD/tree/main/CICD

GitLab Runner 建置

如果 3 還沒建置好,可以參考以下連結,會教你如何建置GitLab Runner:https://docs.gitlab.com/runner/install/kubernetes.html

1
2
3
helm repo add gitlab https://charts.gitlab.io
# 之前有遇到小坑 --name已經註銷,因此要把name放在前面喔
helm install gitlab-runner --namespace gitlab -f values.yaml gitlab/gitlab-runner

values.yaml檔案如下,需要修改以下內容 (values.yaml來源)

  • gitlabeUrl: 指定要註冊的GitLab
  • runnerRegistrationToken: 指定要註冊的GitLab Runner Token,這邊需要注意的是需要使用admin的帳號進入gitlab,找到ci runner去產生一個新的token提供runner來註冊。
  • rbac.create: 要調整成true否則無法run ci
  • config: 所有想要額外的設定都會在這裡設定,可以參考
`values.yaml` 內容說明
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
## GitLab Runner Image
##
## By default it's using registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## DEPRECATED: Setting `image: registry.gitlab.com/gitlab-org/gitlab-runner:alpine-v11.6.0` is deprecated
##
## ref: https://gitlab.com/gitlab-org/gitlab-runner/container_registry/29383?orderBy=NAME&sort=asc&search[]=alpine-v&search[]=
##
## Note: If you change the image to the ubuntu release
## don't forget to change the securityContext;
## these images run on different user IDs.
##
image:
registry: registry.gitlab.com
image: gitlab-org/gitlab-runner
# tag: alpine-v11.6.0

## Specify a imagePullPolicy for the main runner deployment
## 'Always' if imageTag is 'latest', else set to 'IfNotPresent'
##
## Note: it does not apply to job containers launched by this executor.
## Use `pull_policy` in [runners.kubernetes] to change it.
##
## ref: https://kubernetes.io/docs/concepts/containers/images/#pre-pulled-images
##
imagePullPolicy: IfNotPresent

## Specifying ImagePullSecrets on a Pod
## Kubernetes supports specifying container image registry keys on a Pod.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
##
# imagePullSecrets:
# - name: "image-pull-secret"

## Timeout, in seconds, for liveness and readiness probes of a runner pod.
# probeTimeoutSeconds: 1

## How many runner pods to launch.
##
## Note: Using more than one replica is not supported with a runnerToken. Use a runnerRegistrationToken
## to create multiple runner replicas.
# replicas: 1

## How many old ReplicaSets for this Deployment you want to retain
# revisionHistoryLimit: 10

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-register
##
gitlabUrl: https://git.prlab.io/

## The Registration Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/index.html
##
runnerRegistrationToken: "hu-89CHyJuYT5VzK-4Rx"

## The Runner Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance. It is token of already registered runner.
## ref: (we don't yet have docs for that, but we want to use existing token)
##
# runnerToken: ""
#

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/index.html#gitlab-runner-unregister
##
# unregisterRunners: true

## When stopping the runner, give it time to wait for its jobs to terminate.
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated with a graceful stop request. terminationGracePeriodSeconds
## instructs Kubernetes to wait long enough for the runner pod to terminate gracefully.
## ref: https://docs.gitlab.com/runner/commands/#signals
terminationGracePeriodSeconds: 3600

## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /home/gitlab-runner/.gitlab-runner/certs/ directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates-targeting-the-gitlab-server
##
# certsSecretName:

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 20

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 30

## Configure GitLab Runner's logging level. Available values are: debug, info, warn, error, fatal, panic
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logLevel:

## Configure GitLab Runner's logging format. Available values are: runner, text, json
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logFormat:

## Configure GitLab Runner's Sentry DSN.
## ref https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# sentryDsn:

## A custom bash script that will be executed prior to the invocation
## gitlab-runner process
#
#preEntrypointScript: |
# echo "hello"

## Specify whether the runner should start the session server.
## Defaults to false
## ref:
##
## When sessionServer is enabled, the user can either provide a public publicIP
## or either rely on the external IP auto discovery
## When a serviceAccountName is used with the automounting to the pod disable,
## we recommend the usage of the publicIP
sessionServer:
enabled: false
# annotations: {}
# timeout: 1800
# internalPort: 8093
# externalPort: 9000
# publicIP: ""
# loadBalancerSourceRanges:
# - 1.2.3.4/32

## For RBAC support:
rbac:
create: true

## Define specific rbac permissions.
## DEPRECATED: see .Values.rbac.rules
# resources: ["pods", "pods/exec", "secrets"]
# verbs: ["get", "list", "watch", "create", "patch", "delete"]

## Define list of rules to be added to the rbac role permissions.
## Each rule supports the keys:
## - apiGroups: default "" (indicates the core API group) if missing or empty.
## - resources: default "*" if missing or empty.
## - verbs: default "*" if missing or empty.
##
## Read more about the recommended rules on the following link
##
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#configuring-executor-service-account
##
rules: []
# - resources: ["configmaps", "pods", "pods/attach", "secrets", "services"]
# verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
# - apiGroups: [""]
# resources: ["pods/exec"]
# verbs: ["create", "patch", "delete"]

## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs
## cluster-wide or only within namespace
clusterWideAccess: false

## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
##
# serviceAccountName: default

## Specify annotations for Service Accounts, useful for annotations such as eks.amazonaws.com/role-arn
##
## ref: https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
##
# serviceAccountAnnotations: {}

## Use podSecurity Policy
## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
podSecurityPolicy:
enabled: false
resourceNames:
- gitlab-runner

## Specify one or more imagePullSecrets used for pulling the runner image
##
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account
##
## imagePullSecrets: [prlab-harbor]

## Configure integrated Prometheus metrics exporter
##
## ref: https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server
##
metrics:
enabled: false

## Define a name for the metrics port
##
portName: metrics

## Provide a port number for the integrated Prometheus metrics exporter
##
port: 9252

## Configure a prometheus-operator serviceMonitor to allow autodetection of
## the scraping target. Requires enabling the service resource below.
##
serviceMonitor:
enabled: false

## Provide additional labels to the service monitor ressource
##
## labels: {}

## Define a scrape interval (otherwise prometheus default is used)
##
## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
##
# interval: ""

## Specify the scrape protocol scheme e.g., https or http
##
# scheme: "http"

## Supply a tls configuration for the service monitor
##
## ref: https://github.com/helm/charts/blob/master/stable/prometheus-operator/crds/crd-servicemonitor.yaml
##
# tlsConfig: {}

## The URI path where prometheus metrics can be scraped from
##
# path: "/metrics"

## A list of MetricRelabelConfigs to apply to samples before ingestion
##
## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs
##
# metricRelabelings: []

## A list of RelabelConfigs to apply to samples before scraping
##
## ref: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
##
## relabelings: []

## Configure a service resource e.g., to allow scraping metrics via
## prometheus-operator serviceMonitor
service:
enabled: false

## Provide additonal labels for the service
##
# labels: {}

## Provide additonal annotations for the service
##
# annotations: {}

## Define a specific ClusterIP if you do not want a dynamic one
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#choosing-your-own-ip-address
##
# clusterIP: ""

## Define a list of one or more external IPs for this service
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#external-ips
##
# externalIPs: []

## Provide a specific loadbalancerIP e.g., of an external Loadbalancer
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
##
# loadBalancerIP: ""

## Provide a list of source IP ranges to have access to this service
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#aws-nlb-support
##
# loadBalancerSourceRanges: []

## Specify the service type e.g., ClusterIP, NodePort, Loadbalancer or ExternalName
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
##
type: ClusterIP

## Specify the services metrics nodeport if you use a service of type nodePort
##
# metrics:

## Specify the node port under which the prometheus metrics of the runner are made
## available.
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#nodeport
##
# nodePort: ""

## Provide a list of additional ports to be exposed by this service
##
## ref: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
##
# additionalPorts: []

## Configuration for the Pods that the runner launches for each new job
##
runners:
# runner configuration, where the multi line strings is evaluated as
# template so you can specify helm values inside of it.
#
# tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
# runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
config: |
[[runners]]
privileged = true
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:20.04"
## Which executor should be used
##
# executor: kubernetes

## Default container image to use for builds when none is specified
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# image: ubuntu:16.04

## Specify one or more imagePullSecrets
##
## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
imagePullSecrets: [prlab-harbor]

## Specify the image pull policy: never, if-not-present, always. The cluster default will be used if not set.
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# imagePullPolicy: ""

## Defines number of concurrent requests for new job from GitLab
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# requestConcurrency: 1

## Specify whether the runner should be locked to a specific project: true, false. Defaults to true.
##
## locked: true

## Specify the tags associated with the runner. Comma-separated list of tags.
##
## ref: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-control-which-jobs-a-runner-can-run
##
# tags: ""

## Specify the name for the runner.
##
# name: ""

## Specify the maximum timeout (in seconds) that will be set for job when using this Runner
##
# maximumTimeout: ""

## Specify if jobs without tags should be run.
## If not specified, Runner will default to true if no tags were specified. In other case it will
## default to false.
##
## ref: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#set-a-runner-to-run-untagged-jobs
##
# runUntagged: true

## Specify whether the runner should only run protected branches.
## Defaults to false.
##
## ref: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#prevent-runners-from-revealing-sensitive-information
##
# protected: true

## Run all containers with the privileged flag enabled
## This will allow the docker:dind image to run if you need to run Docker
## commands. Please read the docs before turning this on:
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# privileged: false

## The name of the secret containing runner-token and runner-registration-token
# secret: gitlab-runner

## Namespace to run Kubernetes jobs in (defaults to the same namespace of this release)
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# namespace:

## The amount of time, in seconds, that needs to pass before the runner will
## timeout attempting to connect to the container it has just created.
## ref: https://docs.gitlab.com/runner/executors/kubernetes.html
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# pollTimeout: 180

## Set maximum build log size in kilobytes, by default set to 4096 (4MB)
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# outputLimit: 4096

## Distributed runners caching
## ref: https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching
##
## If you want to use s3 based distributing caching:
## First of all you need to uncomment General settings and S3 settings sections.
##
## Create a secret 's3access' containing 'accesskey' & 'secretkey'
## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
##
## $ kubectl create secret generic s3access \
## --from-literal=accesskey="YourAccessKey" \
## --from-literal=secretkey="YourSecretKey"
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
##
## If you want to use gcs based distributing caching:
## First of all you need to uncomment General settings and GCS settings sections.
##
## Access using credentials file:
## Create a secret 'google-application-credentials' containing your application credentials file.
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section
## You could configure
## $ kubectl create secret generic google-application-credentials \
## --from-file=gcs-application-credentials-file=./path-to-your-google-application-credentials-file.json
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
##
## Access using access-id and private-key:
## Create a secret 'gcsaccess' containing 'gcs-access-id' & 'gcs-private-key'.
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscachegcs-section
## You could configure
## $ kubectl create secret generic gcsaccess \
## --from-literal=gcs-access-id="YourAccessID" \
## --from-literal=gcs-private-key="YourPrivateKey"
## ref: https://kubernetes.io/docs/concepts/configuration/secret/
##
## If you want to use Azure-based distributed caching:
## First, uncomment General settings.
##
## Create a secret 'azureaccess' containing 'azure-account-name' & 'azure-account-key'
## ref: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction
##
## $ kubectl create secret generic azureaccess \
## --from-literal=azure-account-name="YourAccountName" \
## --from-literal=azure-account-key="YourAccountKey"
## ref: https://kubernetes.io/docs/concepts/configuration/secret/

cache: {}
## General settings
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
# cacheType: s3
# cachePath: "gitlab_runner"
# cacheShared: true

## S3 settings
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
# s3ServerAddress: s3.amazonaws.com
# s3BucketName:
# s3BucketLocation:
# s3CacheInsecure: false

## GCS settings
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template
# gcsBucketName:

## S3 the name of the secret.
# secretName: s3access
## Use this line for access using gcs-access-id and gcs-private-key
# secretName: gcsaccess
## Use this line for access using google-application-credentials file
# secretName: google-application-credentials
## Use this line for access using Azure with azure-account-name and azure-account-key
# secretName: azureaccess


## Build Container specific configuration
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
builds: {}
# cpuLimit: 200m
# cpuLimitOverwriteMaxAllowed: 400m
# memoryLimit: 256Mi
# memoryLimitOverwriteMaxAllowed: 512Mi
# cpuRequests: 100m
# cpuRequestsOverwriteMaxAllowed: 200m
# memoryRequests: 128Mi
# memoryRequestsOverwriteMaxAllowed: 256Mi

## Service Container specific configuration
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
services: {}
# cpuLimit: 200m
# memoryLimit: 256Mi
# cpuRequests: 100m
# memoryRequests: 128Mi

## Helper Container specific configuration
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
helpers: {}
# cpuLimit: 200m
# memoryLimit: 256Mi
# cpuRequests: 100m
# memoryRequests: 128Mi
# image: "registry.gitlab.com/gitlab-org/gitlab-runner-helper:x86_64-${CI_RUNNER_REVISION}"

## Helper container security context configuration
## Refer to https://docs.gitlab.com/runner/executors/kubernetes.html#using-security-context
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# pod_security_context:
# run_as_non_root: true
# run_as_user: 100
# run_as_group: 100
# fs_group: 65533
# supplemental_groups: [101, 102]

## Service Account to be used for runners
##
# serviceAccountName:

## If Gitlab is not reachable through $CI_SERVER_URL
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# cloneUrl:

## Specify node labels for CI job pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# nodeSelector: {}

## Specify node tolerations for CI job pods assignment
## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# nodeTolerations: {}

## Specify pod labels for CI job pods
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# podLabels: {}

## Specify annotations for job pods, useful for annotations such as iam.amazonaws.com/role
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# podAnnotations: {}

## Configure environment variables that will be injected to the pods that are created while
## the build is running. These variables are passed as parameters, i.e. `--env "NAME=VALUE"`,
## to `gitlab-runner register` command.
##
## Note that `envVars` (see below) are only present in the runner pod, not the pods that are
## created for each build.
##
## ref: https://docs.gitlab.com/runner/commands/#gitlab-runner-register
##
## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration
# env:
# NAME: VALUE


## Specify the name of the scheduler which used to schedule runner pods.
## Kubernetes supports multiple scheduler configurations.
## ref: https://kubernetes.io/docs/reference/scheduling
# schedulerName: "my-custom-scheduler"

## Configure securitycontext for the main container
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
runAsNonRoot: true
privileged: false
capabilities:
drop: ["ALL"]

## Configure securitycontext valid for the whole pod
## ref: http://kubernetes.io/docs/user-guide/security-context/
##
podSecurityContext:
runAsUser: 100
# runAsGroup: 65533
fsGroup: 65533
# supplementalGroups: [65533]

## Note: values for the ubuntu image:
# runAsUser: 999
# fsGroup: 999

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}
# limits:
# memory: 256Mi
# cpu: 200m
# requests:
# memory: 128Mi
# cpu: 100m

## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
# Example: The gitlab runner manager should not run on spot instances so you can assign
# them to the regular worker nodes only.
# node-role.kubernetes.io/worker: "true"

## List of node taints to tolerate (requires Kubernetes >= 1.6)
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
# Example: Regular worker nodes may have a taint, thus you need to tolerate the taint
# when you assign the gitlab runner manager with nodeSelector or affinity to the nodes.
# - key: "node-role.kubernetes.io/worker"
# operator: "Exists"

## Configure environment variables that will be present when the registration command runs
## This provides further control over the registration process and the config.toml file
## ref: `gitlab-runner register --help`
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
##
# envVars:
# - name: RUNNER_EXECUTOR
# value: kubernetes

## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases: []
# Example:
# - ip: "127.0.0.1"
# hostnames:
# - "foo.local"
# - "bar.local"
# - ip: "10.1.2.3"
# hostnames:
# - "foo.remote"
# - "bar.remote"

## Annotations to be added to manager pod
##
podAnnotations: {}
# Example:
# iam.amazonaws.com/role: <my_role_arn>

## Labels to be added to manager pod
##
podLabels: {}
# Example:
# owner.team: <my_cool_team>

## HPA support for custom metrics:
## This section enables runners to autoscale based on defined custom metrics.
## In order to use this functionality, Need to enable a custom metrics API server by
## implementing "custom.metrics.k8s.io" using supported third party adapter
## Example: https://github.com/directxman12/k8s-prometheus-adapter
##
#hpa: {}
# minReplicas: 1
# maxReplicas: 10
# metrics:
# - type: Pods
# pods:
# metricName: gitlab_runner_jobs
# targetAverageValue: 400m

## Configure priorityClassName for manager pod. See k8s docs for more info on how pod priority works:
## https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/
priorityClassName: ""

## Secrets to be additionally mounted to the containers.
## All secrets are mounted through init-runner-secrets volume
## and placed as readonly at /init-secrets in the init container
## and finally copied to an in-memory volume runner-secrets that is
## mounted at /secrets.
secrets: []
# Example:
# - name: my-secret
# - name: myOtherSecret
# items:
# - key: key_one
# path: path_one

## Additional config files to mount in the containers in `/configmaps`.
##
## Please note that a number of keys are reserved by the runner.
## See https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/main/templates/configmap.yaml
## for a current list.
configMaps: {}

## Additional volumeMounts to add to the runner container
##
volumeMounts: []
# Example:
# - name: my-volume
# mountPath: /mount/path

## Additional volumes to add to the runner deployment
##
volumes: []
# Example:
# - name: my-volume
# persistentVolumeClaim:
# claimName: my-pvc

1 安裝ArgoCD

Argo CD是一個持續交付工具可以自動化部署應用程式到Kubernetes集群中。它遵循GitOps模式,當發現Git Repo 中所監控的資料夾有變更時,Argo CD會自動部署應用程式到Kubernetes集群中。因為CD透過拉取的方式,所以我們又稱之為Pull Based。

1.1 基本設定

1
2
3
4
5
6
7
8
9
10
# Getting Start 
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

# 設定nodeport 根據argo clusterIP的service調整成 nodePort,因為我們要讓外部可以連線到argo
kubectl get svc -n argocd argocd-server -o yaml > argocd-nodeport.yaml

# 也可以這樣 去更新service的type:NodePort
kubectl patch svc argocd-server -n argocd
-p '{"spec": {"type": "NodePort"}}'

argocd-nodeport.yaml 轉寫Nodeport的yaml 這樣才可以從外面取得

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/component: server
app.kubernetes.io/name: argocd-server
app.kubernetes.io/part-of: argocd
name: argocd-server-nodeport
namespace: argocd
spec:
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
nodePort: 30090
- name: https
port: 443
protocol: TCP
targetPort: 8080
nodePort: 30091
selector:
app.kubernetes.io/name: argocd-server
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
1
$ kubectl create -f argocd-nodeport.yaml

1.2 取得初始化密碼

1
2
3
4
5
6
7
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d; echo

<output>
5BX3ZVroqjsR1J05
# using username
username: admin
password: <output>

2 Gitlab Manifest架構

  • 這邊所使用的helm chart來進行config manager
  • 可以參考如下圖的架構:
    • templates: 裡面放所有deployment所需使用的yaml檔案
    • values: 當你templates裡面的yaml檔案有挖空的部分會根據values.yaml來填上
    • kioxia-helm.yaml: 下一步step3要執行的CRD Application yaml
    • Chart.yaml: 定義Chart的yaml檔案
    • values.yaml: 如果找不到相對應的values.yaml就會使用的初始值

2.1 templates - 建立k8s部署用yaml

  • 那我們先從templates開始,基本上這裡就是放所有的deployment, service那些在k8s部屬用的yaml檔案
  • 你可以發現很多被挖空的地方,這些地方會由values.yaml所填補上

deployment.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Values.metadata.name }}
namespace: {{ .Values.namespace.name }}
spec:
selector:
matchLabels:
kioxia.share.station/server: {{ .Values.metadata.name }}
replicas: 1
template:
metadata:
labels:
kioxia.share.station/server: {{ .Values.metadata.name }}
spec:
containers:
- image: {{.Values.image.repository}}:{{.Values.image.tag}}
lifecycle:
preStop:
exec:
command: [ "sh", "-c", "sleep 10" ]
imagePullPolicy: Always
name: {{ .Values.metadata.name }}
ports:
- containerPort: {{ .Values.service.port }}
protocol: TCP

2.2 values.yaml - 建立部署用所填入的值

  • 因為我們這裡使用微服務架構,因此我希望可以一次性透過helm來產所有的app的template
  • 我們有3個values.yaml和一個資料夾customize-template
    • customize-template: 之後gitlab cd的部分會把env變數塞進去然後來覆蓋掉外面的values-app.yaml來達到觸發argo cd
    • values-file/idp/pdp.yaml: 給特定server使用的yaml檔案,但是要怎麼知道argo要部屬哪一個values呢? 下一步2.3會告訴你

values-file.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# # values.yaml - default input configuration file for the chart. If no other values file is supplied, the parameters in this file will be used.

metadata:
name: app-name # 對應到 {{Values.metadata.name}}

namespace:
name: logging # 對應到 {{Values.namespace.name}}

image:
repository: <harbor-url/image-name> # harbor 的 image repo
tag: 1a015d32 # 這裡之後會被customizie-template所替換掉,我們每次commit到特定branch會觸發ci,ci會幫我建立image使用這個commit tag當作image tag,而我這個目的在於要拉這個image tag來部屬於k8s

service:
port: 8086 # pod 中服務的 port
nodePort: 30086 # 對外開放的port

customize-template/vlaues-file.yaml

  • 放在cutomize-template裡面的yaml檔案有點像是template的存在供customize工具來將env變數塞進去然後產生一個新的values-file.yaml放在values/底下ArgoCD會使用的檔案
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# # values.yaml - default input configuration file for the chart. If no other values file is supplied, the parameters in this file will be used.

metadata:
name: app-name

namespace:
name: logging

image:
repository: <harbor-url/image-name>
tag: ${IMAGE_TAG} # gitlab cd裡面會貼上 環境變數

service:
port: 8086
nodePort: 30086

2.3 crd.yaml - 建立argocd CRD

  • 當我們準備好
    • template: 在k8s部屬用的yaml檔案
    • values.yaml: 部屬用的value,會根據value-ooo.yaml來填補到template裡面
  • 接下來就是要執行在k8s環境用來監視gitlab的manifest是否有變動的application crd,所謂的CRD(Custom Resource Definition) 是 Kubernetes 提供的功能,允許你擴展 Kubernetes API,定義你自己的資源類型。通過 CRD,你可以將自定義的配置對象整合到 Kubernetes 中,像管理原生資源(如 Pods、Services)一樣管理這些自定義資源。

Argo CD 的核心理念是 GitOps,它通過監視 Git 存儲庫的配置,將 Kubernetes 集群中的應用程序狀態與 Git 中的定義保持一致。Application CRD 讓 Argo CD 能夠了解應用程序的來源、目標和配置。

argocd-crd.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-argocd # <== 自己取名
namespace: argocd # <== 一定要跟argo在同一個namespace
spec:
# which cluster our resource should be deployed基本上不用改動
destination:
namespace: default
server: https://kubernetes.default.svc
project: default
# reference our manifest
source:
helm:
valueFiles:
- values/values-file.yaml # <== 指定你要使用的values.yaml
path: manifests # <== 指定gitlab要監控的manifests,通常就是git repo裡面有個名為manifests的資料夾,這個資料夾裡面任何內容如果有變動argocd就會知道
repoURL: https://<gitlab-url>/<app>.git # <== 指定gitlab repo
targetRevision: master # <== 指定要監控哪一個branch
# empty mean we manually sync the app
syncPolicy: {} # <== 可以設定要手動還是自動同步

執行

1
2
3
# 啟動 看一下你執行的application是否有執行
$ kubectl create -f argocd-crd.yaml
$ kubectl get application -n argocd

2.4 argo 先登入gitlab

  • Ref: https://ppfocus.com/0/di2b6fdf4.html

  • 你可能會好奇argo要怎麼知道gitlab的帳號密碼?

  • 由於我們這裡的代碼倉庫是私有的 GitLab,所以我們還需要配置對應的倉庫地址,在頁面上 Settings → Repositories,點擊 Connect Repo using HTTPS 按鈕:

    • 需要注意的是這裡默認使用的是 HTTPS,所以我們需要勾選下方的 Skip server verification
    • 然後點擊上方的 CONNECT 按鈕添加即可。然後重新同步上面的兩個 Application,就可以看到正常的狀態了。

3 Gitlab CICD 設定撰寫

3.1 基本參數設定

  • 我們要先於gitlab ci variable設定以下參數:
  • 目的是為了push image到harbor
    • CI_REGISTER_IMAGE_<NAME>: image名稱,這裡的例子如果使用private repo則是可以填寫/dev/kioxia-file
    • CI_REGISTER: harbor位置,如果是使用private repo可以填寫192.168.0.100:30004
    • CI_REGISTER_USER: harbor robot account,所以要記得去Harbor建立robot的token和username,可以參考官網
    • CI_REGISTER_PASSWORD: harbor robot token
  • 目的是為了在cd pipeline做commit的動作,可以參考官網:
    • CI_USERNAME_GIT: project access token’s account
    • CI_PASSWORD_GIT: project access token


3.2 設定gitlab runner的適用對象

  • gitlab runner有設定可以使用的對象是誰,如果要使用gitlab runner的服務就要使用root權限去設定
  • 使用root@password帳號登入進入runners
  • 編輯runner設定
  • 把想要使用cicd的project加入

3.3 撰寫.gitlab-ci.yml

:::warning
請注意,.gitlab-ci.yml不要寫成.gitlab-ci.yaml否則無法觸發
:::

  • 原生vairable
  • ci pipeline
  • cd pipeline
    • 原本有兩種方式kustomzie來修改deployment的image tag但是argo裡面好像無法使用helm chart的同時又使用kustomize,我猜App of Apps之類的問題
    • 我測試過如果拿掉helm就可以正常使用kustomize,想要玩玩看可以參考
    • 在 Argo CD 中,如果你在使用 App of Apps 模式時,嘗試在同一個應用程序中同時使用 Helm 和 Kustomize,可能會遇到以下問題:

工具衝突:Argo CD 可能無法正確解析來自 Kustomize 和 Helm 的配置,因為這兩者有不同的處理邏輯和數據格式。
配置管理困難:在同一個應用程序中混合使用這兩種工具可能會導致配置管理變得複雜,增加了維護的難度。

.gitlab-ci.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

# 我想要先組成 push 用的 harbor-url/<image-name>:<sha>
variables:
IMAGE_TAG: $CI_COMMIT_SHORT_SHA # commit sha 作為 image tag\
CI_IMAGE_FILE: $CI_REGISTRY$CI_REGISTRY_IMAGE_FILE:$CI_COMMIT_SHORT_SHA # 目標是組成 push image 用的路徑 harbor-url/<image-name>:<sha>

stages:
- build-file
- deploy

# 這邊我使用kaniko進行build image and push image動作
# Every job we done, GitLab Runner will delete the container. So we need to push the image into registry
build-file: # <= can change the name you want
stage: build-file #<= should as same as stages below
image:
name: gcr.io/kaniko-project/executor:v1.9.0-debug
entrypoint: [""]
script:
- mkdir -p /kaniko/.docker
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTER_USER\",\"password\":\"$CI_REGISTER_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor
--context "${CI_PROJECT_DIR}"
--dockerfile "${CI_PROJECT_DIR}/File-Server/Dockerfile"
--destination "${CI_IMAGE_FILE}"
only:
- master

deploy:
stage: deploy
image: daniyalj/alpine-envsub:v1 # cnych/kustomize:v1.0 (if using kustomize)
before_script:
- git remote set-url origin https://${CI_USERNAME_GIT}:${CI_PASSWORD_GIT}@git.prlab.io/kioxia/kioxia-project.git
- git config --global user.email "gitlab-ci@git.k8s.local"
- git config --global user.name "GitLab CI/CD"
- git config http.sslVerify false
script:
# Works: use envsub change the values files.
## work1: logging-helm, edit kustomize file
- git checkout -B master
- cd ${CI_PROJECT_DIR}/manifests
- export IMAGE_TAG=$IMAGE_TAG
- envsubst < ./customize-template/deployment.yaml > ./deployment.yaml
## work2: logging-helm, edit kustomize file
- cd ${CI_PROJECT_DIR}/manifests
- envsubst < ./values/customize-template/values-file.yaml > ./values/values-file.yaml
# If your commit message contains [ci skip] or [skip ci], using any capitalization,
# the commit will be created but the pipeline will be skipped.
- git commit -am '[skip ci] K8s Deploy CICD Done'
- git push origin master
only:
- master

補充 ArgoCD CLI 操作

https://argo-cd.readthedocs.io/en/release-1.8/cli_installation/#linux

安裝CLI

1
2
3
4
5
6
7
# linus 
VERSION=$(curl --silent "https://api.github.com/repos/argoproj/argo-cd/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/download/$VERSION/argocd-linux-amd64

chmod +x /usr/local/bin/argocd

登入

1
2
3
4
5
6
# find pod
kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2

# login 要使用ip位置登入 192.168.0.100:30090
argocd login <ARGOCD_SERVER_IP>

補充 ArgoCD Auto Sync

https://argo-cd.readthedocs.io/en/release-1.8/user-guide/auto_sync/

Automated Sync

  • 目前git manifest有更新的話並不會自動做更新動作
  • 因此如果希望manifest有所改變的時候就自動部屬可以有以下設定
  • An automated sync will only be performed if the application is OutOfSync.
  • Applications in a Synced or error state will not attempt automated sync.
1
2
3
spec:
syncPolicy:
automated: {}

Automatic Pruning

  • 當argo cd 檢測到資源已經不再git中的定義時,並不會自動刪除原本資源
  • 如果沒有額外設定,要自己去手動刪除原本的資源
  • 但是可以通過以下命令,啟用prun作為自動同步的一部分自動發生

CRD Application設定

1
2
3
4
spec:
syncPolicy:
automated:
prune: true

Automatic Self-Healing

補充 ArgoCD 一些best practice

https://argo-cd.readthedocs.io/en/release-1.8/user-guide/best_practices/

  1. argo manifest config code最好和source code分開
    • 通常為服務是built 來自不同的Git repo因此應該要單獨deploy不要把所有微服務們的config放在一起
    • 因為有時候微服務應用有很多不同版本組成
    • 因為有時候commit會導致無限trigger部屬工作在這種情況可以透過分開的repo來放config changes避免這種事情發生
  2. 為指令留空間
    • 如果想要讓HPA留有設定空間就不要設定relicas in Git

補充 ArgoCD 重新設定密碼

1
2
3
4
# 先登入然後輸入使用者名稱與密碼
$ argocd login 192.168.0.100:30090
# 然後就可以執行重新設定密碼的指令
$ argocd account update-password