Building a local copy of pomerium into the ingest-controller
As you'd expect, the pomerium ingest-controller (IC) for k8s has some local code, some pomerium interface code, and depends on pomerium. If you want to try something with the pomerium library code, you need to arrange for a local version of it to end up in your IC container image.
Checkouts
% git clone https://git@github.com/pomerium/ingress-controller.git
ingress-controller% grep "pomerium v" go.mod
github.com/pomerium/pomerium v0.20.1-0.20230526203421-d315e683357a
ingress-controller% git clone https://git@github.com/pomerium/pomerium.git pomerium-src
I want pomerium source inside the IC directory for ease of docker building, but it needs a separate name from the ingress-controller/pomerium/ that's already there.
The grep call above says that this IC version depends on a pomerium-src revision (not just v0.20.1, which took me a while to figure out!)
ingress-controller/pomerium-src% git checkout d315e683357a
go.mod link
go.mod tells the IC build to use a version of pomerium from github. Insert this in go.mod (before the first require block, but maybe that doesn't matter):
replace github.com/pomerium/pomerium => ./pomerium-src
Incorporate pomerium-src's dependencies
Edit ingress-controller's Dockerfile, so the build can incorporate pomerium-src/go.mod.
diff --git a/Dockerfile b/Dockerfile index 9805497..3044a97 100644 --- a/Dockerfile +++ b/Dockerfile @@ -12,10 +12,15 @@ COPY Makefile Makefile RUN mkdir -p pomerium/envoy/bin RUN make envoy + +COPY pomerium-src/go.mod pomerium-src/go.sum pomerium-src/ RUN go mod download COPY Makefile ./Makefile
This docker (actually podman) build takes me 2m45s. I believe the following is a safe speedup. It does some dependency downloads before the source code copy, which caches better. New rebuild time is 2m0s:
diff --git a/Dockerfile b/Dockerfile index 9805497..a9212d2 100644 --- a/Dockerfile +++ b/Dockerfile @@ -16,6 +16,9 @@ RUN go mod download COPY Makefile ./Makefile +# this pulls a bunch of deps that we don't want to repeat in 'make build-go' +RUN make controller-gen + # download ui dependencies from core module RUN mkdir -p internal RUN make internal/ui
Build the docker image with all the local edits
So far, this is building to ingress-controller/bin/*, but we want to affect the build in a docker image.
ingress-controller% export TAG=bang5:5000/pomerium_ic_local ingress-controller% docker build -t ${TAG} . && docker push ${TAG}
Output includes this:
Successfully tagged bang5:5000/pomerium_ic_local:latest 031715bd3fa0da34f7cfdfc26eb020643bf1a9904ad8af228837c2bdcaef39cc
GCR error
You may see this:
[3/3] STEP 1/5: FROM gcr.io/distroless/base:debug-nonroot@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7
Trying to pull gcr.io/distroless/base@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7...
Error: creating build container: initializing source docker://gcr.io/distroless/base@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7: getting username and password: 1 error occurred:
* error getting credentials - err: docker-credential-gcloud resolves to executable in current directory (./docker-credential-gcloud), out: ``
Something from the user setup is breaking this (how?). I used a different account on another computer to work around it.
Use the local docker image in a k8s setup
I run ingress-controller/deployment.yaml with some kustomize patches, so I deploy the new docker image like this:
My existing deployment configs:
upstream% cp .../ingress-controller/deplyment.yaml pomerium-ingress-controller.yaml
My existing upstream/kustomzation.yaml:
bases: - pomerium-ingress-controller.yaml - cert-manager-v1.12.0.yaml patchesStrategicMerge: - "patch.yaml" etc
My upstream/patch.yaml, with the local image:
--- apiVersion: apps/v1 kind: Deployment metadata: name: pomerium namespace: pomerium spec: template: spec: containers: - name: pomerium image: bang5:5000/pomerium_ic_local:latest imagePullPolicy: Always
Apply changes (abridged):
kubectl kustomize upstream | kubectl apply -f -
Test the result
Edit this:
--- a/pomerium/ctrl/run.go +++ b/pomerium/ctrl/run.go @@ -75,6 +75,7 @@ func (r *Runner) Run(ctx context.Context) error { } log.FromContext(ctx).V(1).Info("got bootstrap config, starting pomerium...", "cfg", r.src.GetConfig()) + log.Log.Error(nil, "hello IC world") return pomerium_cmd.Run(ctx, r.src) }
And this:
diff --git a/pomerium-src/authorize/internal/store/store.go b/pomerium-src/authorize/internal/store/store.go index f9f9e08d..3dcac141 100644 --- a/pomerium-src/authorize/internal/store/store.go +++ b/pomerium-src/authorize/internal/store/store.go @@ -31,6 +31,8 @@ type Store struct { // New creates a new Store. func New() *Store { + ctx := context.TODO() + log.Error(ctx).Err(nil).Msg("hello pom world") return &Store{ Store: inmem.New(), }
Rebuild
ingress-controller% docker build -t ${TAG} . && docker push ${TAG}
Also, my docker is podman, so I added "build --network=host" for a speedup.
Redeploy to k8s:
% kubectl rollout restart -n pomerium deploy/pomerium
Use tab-complete (or a app.name selector) to describe the right pod:
% kubectl describe -n pomerium pod/pomerium-79954677c-qzhpj
Image: bang5:5000/pomerium_ic_local:latest
Image ID: bang5:5000/pomerium_ic_local@sha256:0edbab12576bc68054b4f900f96318c04df4fcb9c1f588f2e4c2454cbe3b8a08
Not sure why this 0edb id doesn't match the 'successfully tagged' 0317 one above.
However, we still have great success!
% kubectl logs -n pomerium deploy/pomerium | grep hello {"level":"error","ts":"2023-07-17T01:32:27Z","msg":"hello IC world"} {"level":"error","time":"2023-07-17T01:32:27Z","message":"hello pom world"}