We recently had the need to write a Mutating Admissions Webhook in k8s to monitor and mutate
jobs being submitted to the cluster. The reason why is for another post – but the first thing I wanted to do was get to the point where I could hit a breakpoint and see what was going on.
This was trickier than I first thought, given the certificates involved and the way that webhooks run in the cluster. We used the very helpful Kubebuilder project to scaffold out our webhooks, but there’s no straightforward way of running one locally to see what k8s passes to it.
My colleague Bart Jansen and I eventually found a way to ‘attach’ a locally running webhook to a running cluster to step through our webhook code, via ngrok. Think remote debug.
Usually the webhook runs like so:
[K8s fires webhook event] –> [Service] –> [Endpoint] –> [pod running webhook code].
We can see this if we look at the
service object deployed via kubebuilder:
Here kubernetes will fire the event at a service named
psc-webhook-service which will use the endpoint
10.244.0.9:9443 which is the IP and port of the pod running the webhook.
Everything in a huge gif:
If you wanna sit for a minute and see a demo – take a look here:
Big demo gif 🙂
What we’ll do is:
– Deploy the webhook code and config (not covered here)
– Copy the cluster certs to our dev pc
– Start an ngrok session on our dev pc
– Use the ngrok IP / port as the endpoint for the service.
0) Create debug-* versions of the endpoint and service file
We will essentially be overwriting the standard service / endpoint for our running webhook to point at ngrok instead. You’ll need the following 2 files as well as some new
make commands: (you’ll need to update the namespace and names appropriately)
1) Copy cluster certs
To debug our code locally we need the same certs running as are running in the cluster. This way kubernetes can authenticate to our service:
Basically this grabs the certs running in the cluster and does some pokery to base64 decode them and save them locally where our code should be looking for them.
2) Start ngrok and send your webhook events there
- Go to https://www.ngrok.com/ and sign in / sign up for free
- Go to https://dashboard.ngrok.com/auth to get a copy of your auth token and run
ngrok authtoken <token>
make ngrok-debugto open an ngrok proxy, remove the webhook node label and apply the debug endpoint
ngrok-debug command (amazing work Bart!) actually updates the 2 yaml files and applies them to your cluster. Now webhook events should be being fired at your ngrok IP, and your PC should be connected to ngrok, ready to receive them.
You should now be able to set a breakpoint in your webhook and hit F5 (or the equivalent) to run it locally. Apply something to your cluster which should result in an event being raised – in our case we were applying some test yaml which contained some dummy
That’s it… I know this is not a full step-by-step, but doing so is tricky given there are different ways to write and deploy webhooks. I hope this helps get you on the right path, anyway… 🙂