Using Skaffold for CI/CD with GitLab
Contributed by: @tvvignesh
Skaffold is a tool which is non-opinionated about the CI/CD tool you should use and acts as a platform agnostic solution for both development and deploying to production.
To facilitate building and deploying images in CI/CD, skaffold offers docker images which you can build on top of. If that doesn’t work for your use-case, you can make your own Dockerfile and pull in skaffold and its dependencies using curl.
Let us have a look at how we can use Skaffold with GitLab CI.
Step 1: Getting the project and Dockerfile ready
The first step is to obviously have your project which you want to deploy ready with a Dockerfile setup for the same.
Skaffold supports multiple builders like Docker, Kaniko, Bazel and more and if the builder you are looking for is not supported out of the box, you can script it out as well. We will use Docker in this demonstration.
For instance, if you would like to deploy nginx, your Dockerfile would look something like this:
FROM nginxinc/nginx-unprivileged:1.19.3-alpine
WORKDIR /app/server
COPY ./web ./web
EXPOSE 8080
CMD [ "/bin/bash", "-c", "nginx -g 'daemon off;'" ]
Skaffold can use your dockerfile to automatically build, tag and push images to the registry when needed.
Step 2: Choosing the deployment method
Skaffold supports deployment through kubectl, helm and a lot of other mechanisms. You can have a look at the complete list of deployers available here.
If you choose to deploy through kubectl, you must have all your yaml files ready, or if you would like to deploy through helm charts, you must have all your charts ready.
Step 3: Choosing the image and tag strategy
Skaffold automatically tags images based on different strategies as documented here: https://skaffold.dev/docs/pipeline-stages/taggers/. The tagging strategy used is configurable, so choose the mechanism which is right for you. By default, skaffold uses the git sha tagger.
If you are using helm as your deployer, you might want to use the helm image strategy if you would like to follow helm specific conventions, as skaffold will pass in all the details for you.
Step 4: Plugins (optional)
If you want to do some kind of processing before deployment like decrypting the secrets in the CI/CD pipeline, skaffold also supports plugins like Helm Secrets. So, if you would like to deploy secrets that have been encrypted using KMS or any other mechanism supported by SOPS you can actually set useHelmSecrets option to true and skaffold will handle everything automatically for you.
Step 5: The skaffold.yaml file
Your skaffold.yaml file will change depending on the way you would like to build and deploy your project. It is recommended that you set up skaffold and run it manually before you setup your CI pipeline since the CI pipeline can just call skaffold to do all the builds and deployments which can be tested out locally via skaffold.
A sample skaffold file can look something like this:
apiVersion: skaffold/v2beta8
kind: Config
profiles:
# Specify the profile name which you can run later using skaffold run -p <profilename>
- name: my-profile-name
build:
artifacts:
# Skaffold will use this as your image name and push it here after building
- image: asia.gcr.io/my-project/my-image
# We are using Docker as our builder here
docker:
# Pass the args we want to Docker during build
buildArgs:
NPM_REGISTRY: '{{.NPM_REGISTRY}}'
NPM_TOKEN: '{{.NPM_TOKEN}}'
deploy:
# Using Helm as the deployment strategy
helm:
# Pass the parameters according to https://skaffold.dev/docs/references/yaml/
releases:
- name: my-release
namespace: default
# Using Helm secrets plugin to process secrets before deploying
useHelmSecrets: true
# Location of the chart - here, we use a local chart
chartPath: ./charts/my-chart
# Path to the values file
valuesFiles:
- ./deployment/dev/values.yaml
- ./deployment/dev/secrets.yaml
skipBuildDependencies: true
flags:
upgrade:
- --install
Please have a look at the comments above to understand what the yaml does.
Rather, if you want to deploy via kubectl, your skaffold file can look something like this:
apiVersion: skaffold/v2beta8
kind: Config
profiles:
# Specify the profile name which you can run later using skaffold run -p <profilename>
- name: dev-svc
build:
artifacts:
# Skaffold will use this as your image name and push it here after building
- image: asia.gcr.io/my-project/my-image
# We are using Docker as our builder here
docker:
# Pass the args you want to Docker during build
buildArgs:
# Pass the args we want to Docker during build
NPM_REGISTRY: '{{.NPM_REGISTRY}}'
NPM_TOKEN: '{{.NPM_TOKEN}}'
deploy:
# In case we enable status check, this will be the timeout till which skaffold will wait
statusCheckDeadlineSeconds: 600
# Using kubectl as our deployer
kubectl:
# Location to our yaml files
# Refer https://skaffold.dev/docs/references/yaml/ for more options
manifests:
- k8/dev/*.yml
Step 6: Development & Testing locally
Before we move on to setting up the CI pipeline, we should test it out locally. Use skaffold run if you are deploying it to the cluster, skaffold build to just build and push the artifacts, skaffold dev if you are developing (this will enable auto reload on changes) or debug using skaffold debug
You can find docs about all these workflows here
Also, note that you can also set up file synchronization to enable faster development workflows. You can read more about that here.
Enabling file synchronization would avoid the need to rebuild the images repeatedly during development and testing and this can accelerate the inner dev loop.
Step 7: Setting up authentication with the registry
As you may already know, there are a lot of places where you can host images namely Docker Hub, GCR, GitLab Registry, Quay and so on.
The way you authenticate is specific to the registry provider of your choice. For eg. you may choose to do a username-password authentication with DockerHub or authenticate using service accounts in GCR and so on.
NOTE: Username-Password authentication is not recommended for CI/CD pipelines since they can be easily compromised and can sometimes be logged as well.
We will be using GCR in our example. You can refer this page to understand how to authenticate with it.
Step 8: Using DIND Service (Optional)
Docker images are typically not cached if you are running your builds/pipeline within Docker containers(i.e. DIND or Docker-in-Docker) and this can be very slow since your CI runner has to download the images again and again for every build even if some layers are already available.
NOTE: If you are not looking to run your pipeline within Docker containers or are fine with slower pipelines/pipelines without caching, you can completely skip this step and proceed to Step 9.
To avoid this, we can set up a DIND service in our cluster to enable caching and storage of image layers for us. A sample dind deployment file as deployed in Kubernetes can look something like this:
# Setup a DIND deployment within the Kubernetes cluster to cache images as we build images in our pipelines
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-dind
spec:
selector:
matchLabels:
app: my-dind
strategy:
type: Recreate
template:
metadata:
labels:
app: my-dind
spec:
containers:
- image: docker:dind
imagePullPolicy: Always
name: my-dind
ports:
- containerPort: 2375
name: my-dind
env:
- name: DOCKER_HOST
value: tcp://localhost:2375
- name: DOCKER_TLS_CERTDIR
value: ''
securityContext:
privileged: true
volumeMounts:
- name: my-dind-storage
mountPath: /var/lib/docker
resources:
limits:
memory: '2Gi'
cpu: '1000m'
requests:
memory: '1Gi'
cpu: '250m'
volumes:
- name: my-dind-storage
persistentVolumeClaim:
claimName: my-dind-pv-claim
Service File:
# Expose DIND as a service so that all the pipelines can access it
apiVersion: v1
kind: Service
metadata:
name: my-dind-svc
spec:
selector:
app: my-dind
type: LoadBalancer
ports:
- port: 2375
targetPort: 2375
protocol: TCP
name: http
PV Claim:
# Provision a Persistent Volume Claim to be used to store the cached artifacts by DIND
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-dind-pv-claim
labels:
app: my-dind
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 200Gi
storageClassName: standard
And we would need to pass the URL to the DIND service when we do builds in the pipeline and all the caching will happen in the DIND storage speeding up the pipeline a lot. Make sure that you clean up the DIND storage periodically or setup cleanup scripts for the same else it might fill up soon.
Step 9: Getting the CI pipeline setup
Now that we have Skaffold ready and working locally, we look at the CI pipeline.
Here is a sample pipeline using GitLab CI and if you are new to it, you can refer the docs here. Also, you might want to look at how to define GitLab CI/CD environment variables here
This pipeline might look complicated at first glance, but most of it related to configuring access for deploying to GCP and configure GitLab’s Docker-in-Docker support to cache artifacts between builds. What is important to note is that we use Skaffold to do all the builds and deployments.
stages:
# The name of the pipeline stage
- my-stage
# The name of the job
my-job:
# The image to be used as the base for this job
image:
name: gcr.io/k8s-skaffold/skaffold:v1.15.0
# Pipeline tags (if you have specified tags to your GitLab Runner)
tags:
- development
stage: my-stage
retry: 2
script:
# Logging in to our gcp account using the service account key (specified in GITLAB variables)
- echo "$GCP_DEV_SERVICE_KEY" > gcloud-service-key.json
- gcloud auth activate-service-account --key-file gcloud-service-key.json
# Specifying the project, zone, cluster to deploy our application in.
- gcloud config set project $PROJECT_DEV_NAME
- gcloud config set compute/zone $PROJECT_DEV_REGION
- gcloud container clusters get-credentials $CLUSTER_DEV_NAME
- kubectl config get-contexts
# Pass in all the environment variables to be passed to skaffold during build and deployment with all the args as documented in https://skaffold.dev/docs/references/cli/ (in our case our cluster context, env vars, namespace and some labels)
- DOCKER_HOST="$DIND_DEV" NPM_REGISTRY="$NPM_REGISTRY_DEV" NPM_TOKEN="$NPM_TOKEN_DEV" skaffold run --kube-context $CLUSTER_DEV_CONTEXT -n default -l skaffold.dev/run-id=deploydep -p dev-svc --status-check
only:
# Run this only for changes in the main branch
refs:
- main
So, ultimately our GitLab CI pipeline does not know anything about the helm charts or build strategy or any deployment methods and relies completely on skaffold to do the job for us.
This works equally well irrespective of whether you choose the Docker Executor or the Kubernetes Executor and would require very little changes between the two.
While GitLab supports many executors (including Docker, Kubernetes, etc.) as documented here, we are working with Kubernetes executor here since it is vendor-agnostic, has the ability to auto-scale up/down, private (if you deploy the Runner in your Kubernetes cluster) and also is well supported for the forseeable future.
NOTE: If you are running Skaffold using the Kubernetes executor, make sure that you are running the runner with appropriate permissions or pod security policies. If you are running using DIND, it requires access to the Docker socket and being in privileged mode which might actually tend to be insecure. An alternative can be to use kaniko, buildkit or other custom builders like Buildah for building the image.
Step 10: That’s all folks
If all went well so far, you can try pushing a commit and see your pipeline running, picking up the changes, building and pushing the image and also deploying the same.
Hope this post was informative. Good luck with your journey with Skaffold.