Debugging Kubernetes Webhooks Locally (via ngrok)

We recently had the need to write a Mutating Admissions Webhook in k8s to monitor and mutate jobs being submitted to the cluster. The reason why is for another post – but the first thing I wanted to do was get to the point where I could hit a breakpoint and see what was going on.

This was trickier than I first thought, given the certificates involved and the way that webhooks run in the cluster. We used the very helpful Kubebuilder project to scaffold out our webhooks, but there’s no straightforward way of running one locally to see what k8s passes to it.

My colleague Bart Jansen and I eventually found a way to ‘attach’ a locally running webhook to a running cluster to step through our webhook code, via ngrok. Think remote debug.


Usually the webhook runs like so:
[K8s fires webhook event] –> [Service] –> [Endpoint] –> [pod running webhook code].
We can see this if we look at the service object deployed via kubebuilder:


psc-system psc-webhook-service

Here kubernetes will fire the event at a service named psc-webhook-service which will use the endpoint which is the IP and port of the pod running the webhook.

Everything in a huge gif:

If you wanna sit for a minute and see a demo – take a look here:
Big demo gif 🙂

The Plan:

What we’ll do is:
– Deploy the webhook code and config (not covered here)
– Copy the cluster certs to our dev pc
– Start an ngrok session on our dev pc
– Use the ngrok IP / port as the endpoint for the service.

0) Create debug-* versions of the endpoint and service file

We will essentially be overwriting the standard service / endpoint for our running webhook to point at ngrok instead. You’ll need the following 2 files as well as some new make commands: (you’ll need to update the namespace and names appropriately)


apiVersion: v1
kind: Endpoints
name: psc-webhook-service
namespace: psc-system
view raw debug-endpoint.yaml hosted with ❤ by GitHub


apiVersion: v1
kind: Service
name: psc-webhook-service
namespace: psc-system
selfLink: /api/v1/namespaces/psc-system/services/psc-webhook-service
port: 443
protocol: TCP
targetPort: 9443
sessionAffinity: None
type: ClusterIP
loadBalancer: {}
view raw debug-service.yaml hosted with ❤ by GitHub


kubectl get secret webhook-server-cert -n psc-system -o json | sed 's/ca.crt/cacrt/; s/tls.crt/tlscrt/; s/tls.key/tlskey/' > secrets.json
mkdir -p ./.running-keys
jq -r .data.cacrt < secrets.json | base64 –decode > ./.running-keys/ca.crt
jq -r .data.tlscrt < secrets.json | base64 –decode > ./.running-keys/tls.crt
jq -r .data.tlskey < secrets.json | base64 –decode > ./.running-keys/tls.key
mkdir -p /tmp/k8s-webhook-server/serving-certs
cp ./.running-keys/* /tmp/k8s-webhook-server/serving-certs/
rm secrets.json
mkdir -p .keys && openssl req -nodes -new -x509 -keyout ./.keys/ca.key -out ./.keys/ca.crt -subj "/CN=cronprimer CA"
openssl genrsa -out ./.keys/tls.key 2048
openssl req -new -key ./.keys/tls.key -subj "/CN=webhook-server.webhook.svc" | openssl x509 -req -CA ./.keys/ca.crt -CAkey ./.keys/ca.key -CAcreateserial -out ./.keys/tls.crt
mkdir -p /tmp/k8s-webhook-server/serving-certs
cp ./.keys/* /tmp/k8s-webhook-server/serving-certs/
# Deploy and configure ngrok debugger
ngrok-debug: SHELL:=/bin/bash
ngrok tcp 9443 –region eu > /dev/null & \
sleep 3; \
kubectl set selector service/psc-webhook-service -n psc-system ""; \
sed -i "/^\([[:space:]]*- port: \).*/s//\1$$(curl -s localhost:4040/api/tunnels | jq -r '.tunnels[0].public_url | split(":")[-1]')/" debug/debug-endpoint.yaml; \
kubectl apply -f debug/debug-endpoint.yaml
view raw Makefile-local-debug hosted with ❤ by GitHub

1) Copy cluster certs

To debug our code locally we need the same certs running as are running in the cluster. This way kubernetes can authenticate to our service:
make copy-running-certs

Basically this grabs the certs running in the cluster and does some pokery to base64 decode them and save them locally where our code should be looking for them.

2) Start ngrok and send your webhook events there

The above ngrok-debug command (amazing work Bart!) actually updates the 2 yaml files and applies them to your cluster. Now webhook events should be being fired at your ngrok IP, and your PC should be connected to ngrok, ready to receive them.

3) F5

You should now be able to set a breakpoint in your webhook and hit F5 (or the equivalent) to run it locally. Apply something to your cluster which should result in an event being raised – in our case we were applying some test yaml which contained some dummy jobs.

That’s it… I know this is not a full step-by-step, but doing so is tricky given there are different ways to write and deploy webhooks. I hope this helps get you on the right path, anyway… 🙂

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

About davros85

Software Engineer @ Microsoft, working with key customers to help them be successful on Azure