Building a local copy of pomerium into the ingest-controller

As you'd expect, the pomerium ingest-controller (IC) for k8s has some local code, some pomerium interface code, and depends on pomerium. If you want to try something with the pomerium library code, you need to arrange for a local version of it to end up in your IC container image.

Checkouts

% git clone https://git@github.com/pomerium/ingress-controller.git
ingress-controller% grep "pomerium v" go.mod 
    github.com/pomerium/pomerium v0.20.1-0.20230526203421-d315e683357a

ingress-controller% git clone https://git@github.com/pomerium/pomerium.git pomerium-src

I want pomerium source inside the IC directory for ease of docker building, but it needs a separate name from the ingress-controller/pomerium/ that's already there.

The grep call above says that this IC version depends on a pomerium-src revision (not just v0.20.1, which took me a while to figure out!)

ingress-controller/pomerium-src% git checkout d315e683357a

go.mod link

go.mod tells the IC build to use a version of pomerium from github. Insert this in go.mod (before the first require block, but maybe that doesn't matter):

replace github.com/pomerium/pomerium => ./pomerium-src

Incorporate pomerium-src's dependencies

Edit ingress-controller's Dockerfile, so the build can incorporate pomerium-src/go.mod.

diff --git a/Dockerfile b/Dockerfile
index 9805497..3044a97 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,10 +12,15 @@ COPY Makefile Makefile

 RUN mkdir -p pomerium/envoy/bin
 RUN make envoy
+
+COPY pomerium-src/go.mod pomerium-src/go.sum pomerium-src/
 RUN go mod download

 COPY Makefile ./Makefile

This docker (actually podman) build takes me 2m45s. I believe the following is a safe speedup. It does some dependency downloads before the source code copy, which caches better. New rebuild time is 2m0s:

diff --git a/Dockerfile b/Dockerfile
index 9805497..a9212d2 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -16,6 +16,9 @@ RUN go mod download

 COPY Makefile ./Makefile

+# this pulls a bunch of deps that we don't want to repeat in 'make build-go'
+RUN make controller-gen
+
 # download ui dependencies from core module
 RUN mkdir -p internal
 RUN make internal/ui

Build the docker image with all the local edits

So far, this is building to ingress-controller/bin/*, but we want to affect the build in a docker image.

ingress-controller% export TAG=bang5:5000/pomerium_ic_local
ingress-controller% docker build -t ${TAG} . && docker push ${TAG}

Output includes this:

Successfully tagged bang5:5000/pomerium_ic_local:latest
031715bd3fa0da34f7cfdfc26eb020643bf1a9904ad8af228837c2bdcaef39cc

GCR error

You may see this:

       [3/3] STEP 1/5: FROM gcr.io/distroless/base:debug-nonroot@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7
    Trying to pull gcr.io/distroless/base@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7...
        Error: creating build container: initializing source docker://gcr.io/distroless/base@sha256:de8fb012fc630b7cdea6861442a0185213b574c71e246ddc97e9eb1d047048e7: getting username and password: 1 error occurred:
        * error getting credentials - err: docker-credential-gcloud resolves to executable in current directory (./docker-credential-gcloud), out: ``

Something from the user setup is breaking this (how?). I used a different account on another computer to work around it.

Use the local docker image in a k8s setup

I run ingress-controller/deployment.yaml with some kustomize patches, so I deploy the new docker image like this:

My existing deployment configs:

upstream% cp .../ingress-controller/deplyment.yaml pomerium-ingress-controller.yaml

My existing upstream/kustomzation.yaml:

bases:
  - pomerium-ingress-controller.yaml
  - cert-manager-v1.12.0.yaml
patchesStrategicMerge:
  - "patch.yaml"
etc

My upstream/patch.yaml, with the local image:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pomerium
  namespace: pomerium
spec:
  template:
    spec:
      containers:
        - name: pomerium
          image: bang5:5000/pomerium_ic_local:latest
          imagePullPolicy: Always

Apply changes (abridged):

kubectl kustomize upstream | kubectl apply -f -

Test the result

Edit this:

--- a/pomerium/ctrl/run.go
+++ b/pomerium/ctrl/run.go
@@ -75,6 +75,7 @@ func (r *Runner) Run(ctx context.Context) error {
        }

        log.FromContext(ctx).V(1).Info("got bootstrap config, starting pomerium...", "cfg", r.src.GetConfig())
+       log.Log.Error(nil, "hello IC world")

        return pomerium_cmd.Run(ctx, r.src)
 }

And this:

diff --git a/pomerium-src/authorize/internal/store/store.go b/pomerium-src/authorize/internal/store/store.go
index f9f9e08d..3dcac141 100644
--- a/pomerium-src/authorize/internal/store/store.go
+++ b/pomerium-src/authorize/internal/store/store.go
@@ -31,6 +31,8 @@ type Store struct {

 // New creates a new Store.
 func New() *Store {
+       ctx := context.TODO()
+       log.Error(ctx).Err(nil).Msg("hello pom world")
        return &Store{
                Store: inmem.New(),
        }

Rebuild

ingress-controller% docker build -t ${TAG} . && docker push ${TAG}

Also, my docker is podman, so I added "build --network=host" for a speedup.

Redeploy to k8s:

% kubectl rollout restart -n pomerium deploy/pomerium

Use tab-complete (or a app.name selector) to describe the right pod:

% kubectl describe -n pomerium pod/pomerium-79954677c-qzhpj

    Image:         bang5:5000/pomerium_ic_local:latest
    Image ID:      bang5:5000/pomerium_ic_local@sha256:0edbab12576bc68054b4f900f96318c04df4fcb9c1f588f2e4c2454cbe3b8a08

Not sure why this 0edb id doesn't match the 'successfully tagged' 0317 one above.

However, we still have great success!

% kubectl logs -n pomerium deploy/pomerium | grep hello

{"level":"error","ts":"2023-07-17T01:32:27Z","msg":"hello IC world"}
{"level":"error","time":"2023-07-17T01:32:27Z","message":"hello pom world"}