Run Codefresh pipelines on your private Kubernetes cluster
The Codefresh runner is a helper application that can be installed on your own Kubernetes cluster (behind a company firewall). It can then build Codefresh pipelines, with full access to secure internal services, without actually compromising the requirements of the on-premise installation.
Codefresh Runner installation
The Codefresh runner installer is available at https://github.com/codefresh-io/venona. You can use Venona to create, upgrade and remove runner installations on any internal Kubernetes cluster.
Notice that a runner installation is needed for each cluster that will run Codefresh pipelines. A runner is not needed in clusters that are used for deployment. It is possible to deploy applications on different clusters other than the ones the runner is running on.
The installation process takes care of all the components of the runner as well as the other resources (config-maps, secrets, volumes) needed by them.
In order to use the Codefresh runner you need the following
- A Kubernetes cluster with outgoing Internet access (preferably with version 1.10). Each node should have 50GB disk size
- A Codefresh account with the Hybrid feature enabled.
- A Codefresh CLI token that will be used to authenticate to your Codefresh account.
Installation can happen from any workstation or laptop that has access (i.e. via
kubectl) to the Kubernetes cluster that will run Codefresh builds. The Codefresh runner will authenticate to your Codefresh account by using the Codefresh CLI token.
Command line installation
First setup Codefresh CLI access first, in the machine where you want to install the runner from.
You can see if the CLI works correctly by running any command such as:
codefresh get pipelines
This should list the pipelines of your Codefresh account. Notice that access to the Codefresh CLI is only needed once during the Runner installation. After that, the Runner with authenticate on it own using the details provided. You do NOT need to install the Codefresh CLI on the cluster that is running Codefresh pipelines.
Download the Runner installer (called Venona) from the releases page or by using homebrew if you are on a Mac:
brew tap codefresh-io/venona brew install venona
Create a namespace in your cluster where you want the Codefresh runner to be installed:
kubectl create namespace codefresh-runtime
And finally run the installer passing as argument the namespace you just created:
venona install --kube-namespace codefresh-runtime
After a while you should see a message that the installation process has finished with success. You can run
venona --help to get additional installation options.
As an example, you can define your own location for kubeconfig and/or CLI config:
venona install --kube-namespace my-codefresh-runtime --verbose --kube-config-path c:/users/kostis/.kube/config --cfconfig c:/Users/Kostis/.cfconfig
To check the installation result type
venona status --verbose and you will get a list of all installations.
Installing on Kubernetes clusters with version earlier than 1.10
If your Kubernetes cluster is using a version earlier than 1.10 you also need to do the following:
Make sure the
PersistentLocalVolumes feature gate is turned on
The runner will try to load available apis using the
Add this endpoint to the ClusterRole
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: system:discovery rules: - nonResourceURLs: - ...other_resources - /openapi - /openapi/* verbs: - get
kubectl or any other management tool to perform this change to the role.
Installing on Google Kubernetes Engine
If you are installing Codefresh runner on the Kubernetes cluster on GKE:
- Make sure your user has
Kubernetes Engine Cluster Adminrole in google console
- Bind your user with
cluster-adminKubernetes cluster role
kubectl create clusterrolebinding NAME --clusterrole cluster-admin --user <YOUR_USER>
Installation of the Codefresh runner on a Kubernetes cluster is also setting up 2 groups of objects. Each one has own RBAC needs and therefore, expected created roles(and cluster-roles)
You can see the exact resource descriptors on Github.
Here is a list of the resources that are created during a Runner installation:
- Agent (grouped by
service-account.venona.yaml- The service account that the agent’s pod will use at the end
cluster-role-binding.venona.yaml- The agent discovering K8S apis by calling to
openapi/v2, this ClusterRoleBinding binds bootstrapped ClusterRole by Kubernetes
service-account.venona.yaml. This role has only permissions to make a GET calls to non-resources URLs
role.venona.yaml- Allow to
DELETEpods and persistent volume claims
role-binding.venona.yaml- The agent is spinning up pods and pvc, this binding binds
- Runtime-environment (grouped by
/.*.re.yaml/) - Kubernetes controller that spins up all required resources to provide a good caching experience during pipeline execution
service-account.dind-volume-provisioner.re.yaml- The service account that the controller will use
cluster-role.dind-volume-provisioner.re.yamlDefines all the permission needed for the controller to operate correctly
cluster-role-binding.dind-volume-provisioner.yaml- Binds the ClusterRole to
Using the Codefresh Runner
Once installed the Runner is fully automated. It polls on its own the Codefresh SAAS (by default every ten seconds) and creates automatically all resources needed for running pipelines.
Once installation is complete, you should see the cluster of the runner as a new Runtime environment in Codefresh at your Account Settings, in the respective tab.
If you have multiple environments available, you can change the default one (shown with a thin blue border) by clicking on the 3 dot menu on the right of each environment. The Codefresh runner installer comes with an option
set-default that will automatically set as default the new runtime environment.
You can even override the runtime environment for a specific pipeline by specifying in the respective section in the pipeline settings.
Monitoring the Runner
Once installed, the runner is a normal Kubernetes application like all other applications. You can use your existing tools to monitor it.
Only the runner pod is long living inside your cluster. All other components (such as the engine) are short lived and exist only during pipeline builds. You can always see what the Runner is doing by listing the resources inside the namespace you chose during installation:
$ kubectl get pods -n codefresh-runtime NAME READY STATUS RESTARTS AGE dind-5c5afbb02e9fd02917b33f06 1/1 Running 0 1m dind-lv-monitor-venona-kkkwr 1/1 Running 1 7d dind-volume-provisioner-venona-646cdcdc9-dqh8k 1/1 Running 2 7d engine-5c5afbb02e9fd02917b33f06 1/1 Running 0 1m venona-8b5f787c5-ftbnd 1/1 Running 2 7d
In the same manner you can list secrets, config-maps, logs, volumes etc. for the Codefresh builds.
Upgrading the Runner
To update the runner to a new version you use the
venona upgrade command
To delete a runner installation use
venona delete <installation_name>. Make sure that you don’t have any pipelines assigned to that runtime environment before the removal takes place.
Deploying applications to the same cluster of the runner
By default the Codefresh runner allows you to build pipelines in your private cluster. If you also want to deploy Docker images on your private cluster you need to use the Codefresh API or CLI to add the cluster to Codefresh.
Here is the CLI command
codefresh create cluster --kube-context <CONTEXT_NAME> --namespace <NAMESPACE> --serviceaccount <SERVICE_ACCOUNT> --behind-firewall
See the connecting a cluster page for more details on adding cluster.