Ginkgo Testing for Operator SDK

Stay up to date

Get notified when I publish something new, and unsubscribe at any time.

So you've gone through the toil of building a shiny new Kubernetes Operator. You might not have been fully versed in Go when you began, but thanks to a bit of blood, sweat, and tears, you've been able to get it to run against your sample Custom Resource (Congrats!).

But here's the hard part.

Your operator, bless its heart, is complicated. It's mostly generated code, save for a few custom types, a reconcile function, and some client code. How the heck are you supposed to test this thing? I mean, you want to test it, right? Okay, maybe that's not the right question. Your manager probably wants some tests, right?

Yeah, that's what I figured.

No worries, friend. I'll help you out. This guide is going to help you go from no controller tests to full, Behavior-Driven Development (BDD) style testing using Ginkgo and envtest. This is really just a jumping-off point, so by the time you are done with this tutorial, you should have the pattern to fill in the gaps — and hopefully make your manager proud.

Assumptions
  • You already have a working operator that you can run in a cluster, and watches for requests of a certain kind.
  • This tutorial will use operator-sdk as the assumed framework, but the general process can be extrapolated to other frameworks, like kubebuilder.
  • This tutorial is optimized for OSX and Linux, sorry Windows friends (but maybe it's still useful to you).

Pre-Basics

I encourage you to get at least a little familiar with Ginkgo using their official docs, as I won't go nearly as in-depth here. I just want to give you enough to get you started.

Also, you'll see functions like Expect(), Eventually(), and Fail(). These are from Gomega, which is Ginkgo's preferred Matcher library. You may also want to patronize their docs too.

The Basics

If you haven't already, install Ginkgo and Gomega.

$ go get github.com/onsi/ginkgo/ginkgo
$ go get github.com/onsi/gomega/...

Navigate into your controller's package folder. In this tutorial, we'll assume it's called mycontroller

$ cd pkg/controller/mycontroller/

Now we need to bootstrap our Ginkgo test suite. Do that easily by running the bootstrap command

$ ginkgo bootstrap
Generating ginkgo test suite bootstrap for mycontroller in:
        mycontroller_suite_test.go

Ginkgo just built you a test suite. How polite! This file will be used to configure your test framework, such as building or attaching controllers to managers, starting clusters and API servers, and more.

Configuring the Test Suite Controller

Our actual tests will be forthcoming. But for now, we need to get the test environment up and running. Let's start by opening the file that Ginkgo made for us.

package mycontroller_test
import (
    "testing"
    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"
)
func TestMycontroller(t *testing.T) {
    RegisterFailHandler(Fail)
    RunSpecs(t, "Mycontroller Suite")
}

Not much, is there? Well, that's alright. This is where the magic will happen.

First, let's import a few more packages that we will need.

import (
    // ... other packages from earlier ...
    "github.com/myuser/super-awesome-operator/pkg/apis" // Or whatever it is for you
    "k8s.io/client-go/kubernetes/scheme"
    "k8s.io/client-go/rest"
    "sigs.k8s.io/controller-runtime/pkg/client"
    "sigs.k8s.io/controller-runtime/pkg/envtest"
    logf "sigs.k8s.io/controller-runtime/pkg/log"
    "sigs.k8s.io/controller-runtime/pkg/log/zap"
    ctrl "sigs.k8s.io/controller-runtime"
)

We need to create a BeforeSuite() block underneath (not inside) of that TestMycontroller function, which will build our test API server using envtest, and start our controllers. So let's do that quick.

var _ = BeforeSuite(func(done Done) {
    var err error
    logf.SetLogger(zap.LoggerTo(GinkgoWriter, true))
    By("bootstrapping test environment")
    err = apis.AddToScheme(scheme.Scheme)
    Expect(err).ToNot(HaveOccurred())
    testEnv = &envtest.Environment{
        CRDDirectoryPaths: []string{
            path.Join(getFileDir(), "../../../deploy/crds"),
        },
    }
    cfg, err = testEnv.Start()
    Expect(err).ToNot(HaveOccurred())
    Expect(cfg).ToNot(BeNil())
    k8sManager, err = ctrl.NewManager(cfg, ctrl.Options{
        Scheme:             scheme.Scheme,
        MetricsBindAddress: "0",
    })
    Expect(err).ToNot(HaveOccurred())
    go func() {
        defer GinkgoRecover()
        err = k8sManager.Start(ctrl.SetupSignalHandler())
        Expect(err).ToNot(HaveOccurred())
    }()
    k8sClient = k8sManager.GetClient()
    Expect(k8sClient).ToNot(BeNil())
    // Add controllers required for testing
    // Add your controller to the manager
    // Use whatever params your controller needs here
    // This example only needs the manager
    err = Add(k8sManager)
    Expect(err).ToNot(HaveOccurred())
    close(done)
}, 60)

Okay, that was a lot. Let's break it down quick.

We start by registering our APIs (the different Kinds that our operator defines) with scheme.Scheme using the following.

Then we configure a new test environment using envtest, which ingests our CRDs to allow an API server to know about our custom resources. When we run testEnv.Start(), we are given a *rest.Client which we call cfg, this is used to create a manager that has a client to talk to the API server we just spun up. Next, we spin off a go routine to start the manager.

In the remaining steps, we run Add(), passing in our manager. This initializes a new reconcile object that starts watching for requests with the registered Kinds, just like a real operation would. The only difference is that this cluster is really only backed by envtest, and nothing more.

At this point, whether you know it or not, we have everything we need to start hitting our “cluster” and have our reconcile loop start processing requests. From hereon out, wee can actually write some tests!

Write a test

Alright, so where do we do that? Oh, you know what, Ginkgo helps you here too. Just run:

$ ginkgo generate mycontroller

This will produce a new file called mycontroller_test.go, which you can use to build your actual tests, like the following:

package mycontroller_test
import (
    "context"
    "fmt"
    . "github.com/onsi/ginkgo"
    . "github.com/onsi/gomega"
    myresourcev1alpha1 "github.com/myuser/super-awesome-operator/pkg/apis/myresource/v1alpha1"
    "sigs.k8s.io/controller-runtime/pkg/reconcile"
    "k8s.io/apimachinery/pkg/types"
)
var _ = Describe("MyController", func() {
    var (
        name        string
        namespace   string
        request     reconcile.Request
    )
BeforeEach(func() {
        name = "test-resource"
        namespace = "test-namespace"
        request = reconcile.Request{
            NamespacedName: types.NamespacedName{
                Name:      name,
                Namespace: namespace,
            },
        }
    })
    Describe("S3V1 ResourceAccess CRD", func() {
        var (
            instance *myresourcev1alpha1.MyResource
        )
        Context("with one base S3V1 resource", func() {
            BeforeEach(func() {
                // Create a new resource using k8sClient.Create()
                // I'm just going to assume you've done this in
                // a method called createInstanceInCluster()
                instance = createInstanceInCluster(name, namespace, instance)
            })
            AfterEach(func() {
                // Remember to clean up the cluster after each test
                deleteInstanceInCluster(instance)
            })
            It("should update the status of the CR", func() {
                // Some method where you've implemented Gomega Expect()'s
                assertResourceUpdated(instance)
            })
        })
    })
})

I'm not going to dive deep into this part, because the official Ginkgo and Gomega docs do a much better job of describing this flow. The big takeaway here is that, behind the scenes, there is an API server (cluster) which you can apply, get, and delete resources from and run Gomega assertions on. How you want to structure that is totally up to you and your use case.

One last thing

Remember that, because you are calling an API to apply your resources against, keep in mind that any reconcile process will be asynchronous. This means it may take upwards of a couple seconds (depending on your controller's complexity) to complete the request and update the CR. As such, you should wrap almost all of your client calls within Gomega's Eventually() block.

// This polls the interior function for a result
// and checks it against a condition until either
// it is true, or the timeout has been reached
Eventually(func() error {
    defer GinkgoRecover()
    err := k8sClient.Get(context.TODO(), request.NamespacedName, instance)
    return err
}, "10s", "1s").ShouldNot(HaveOccurred())

This will allow you to set a timeout for the condition to be true, and give the controller time to actually process the request. Remember to consider what your reconcile function's refresh and requeue intervals are. And give your Eventually blocks enough of a timeout for at least 2 of these intervals to pass.

For example, if your reconcile requeues requests on 5 second intervals, you should set your Eventually timeout to something like 12s, with a check interval of 1s. That way, your controller has time to requeue if needed, but your tests will still return within at least 1 second of the controller's completion.

That's all for now

Sorry, I know it's a lot. I just hope I've put you on the right path to getting this up and running. And if not, and I've made this harder for you — complain in the comments below. I probably deserve it.

I'm always willing to update my article with new suggestions.

More to come…

In a future post, I'll to touch on how to completely, and generically, mock a controller in this framework to allow for injecting results from external CRs into your test conditions.

This article originally appeared on Medium.