Building a local copy of pomerium into the ingest-controller

As you'd expect, the pomerium ingest-controller (IC) for k8s has some local code, some pomerium interface code, and depends on pomerium. If you want to try something with the pomerium library code, you need to arrange for a local version of it to end up in your IC container image.


% git clone
ingress-controller% grep "pomerium v" go.mod v0.20.1-0.20230526203421-d315e683357a
ingress-controller% git clone pomerium-src

I want pomerium source inside the IC directory for ease of docker building, but it needs a separate name from the ingress-controller/pomerium/ that's already there.

The grep call above says that this IC version depends on a pomerium-src revision (not just v0.20.1, which took me a while to figure out!)

ingress-controller/pomerium-src% git checkout d315e683357a

go.mod link

go.mod tells the IC build to use a version of pomerium from github. Insert this in go.mod (before the first require block, but maybe that doesn't matter):

replace => ./pomerium-src

Incorporate pomerium-src's dependencies

Edit ingress-controller's Dockerfile, so the build can incorporate pomerium-src/go.mod.

diff --git a/Dockerfile b/Dockerfile
index 9805497..3044a97 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,10 +12,15 @@ COPY Makefile Makefile

 RUN mkdir -p pomerium/envoy/bin
 RUN make envoy
+COPY pomerium-src/go.mod pomerium-src/go.sum pomerium-src/
 RUN go mod download

 COPY Makefile ./Makefile

This docker (actually podman) build takes me 2m45s. I believe the following is a safe speedup. It does some dependency downloads before the source code copy, which caches better. New rebuild time is 2m0s:

diff --git a/Dockerfile b/Dockerfile
index 9805497..a9212d2 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -16,6 +16,9 @@ RUN go mod download

 COPY Makefile ./Makefile

+# this pulls a bunch of deps that we don't want to repeat in 'make build-go'
+RUN make controller-gen
 # download ui dependencies from core module
 RUN mkdir -p internal
 RUN make internal/ui

Build the docker image with all the local edits

So far, this is building to ingress-controller/bin/*, but we want to affect the build in a docker image.

ingress-controller% export TAG=bang5:5000/pomerium_ic_local
ingress-controller% docker build -t ${TAG} . && docker push ${TAG}

Output includes this:

Successfully tagged bang5:5000/pomerium_ic_local:latest

GCR error

You may see this:

       [3/3] STEP 1/5: FROM
    Trying to pull
        Error: creating build container: initializing source docker:// getting username and password: 1 error occurred:
        * error getting credentials - err: docker-credential-gcloud resolves to executable in current directory (./docker-credential-gcloud), out: ``

Something from the user setup is breaking this (how?). I used a different account on another computer to work around it.

Use the local docker image in a k8s setup

I run ingress-controller/deployment.yaml with some kustomize patches, so I deploy the new docker image like this:

My existing deployment configs:

upstream% cp .../ingress-controller/deplyment.yaml pomerium-ingress-controller.yaml

My existing upstream/kustomzation.yaml:

  - pomerium-ingress-controller.yaml
  - cert-manager-v1.12.0.yaml
  - "patch.yaml"

My upstream/patch.yaml, with the local image:

apiVersion: apps/v1
kind: Deployment
  name: pomerium
  namespace: pomerium
        - name: pomerium
          image: bang5:5000/pomerium_ic_local:latest
          imagePullPolicy: Always

Apply changes (abridged):

kubectl kustomize upstream | kubectl apply -f -

Test the result

Edit this:

--- a/pomerium/ctrl/run.go
+++ b/pomerium/ctrl/run.go
@@ -75,6 +75,7 @@ func (r *Runner) Run(ctx context.Context) error {

        log.FromContext(ctx).V(1).Info("got bootstrap config, starting pomerium...", "cfg", r.src.GetConfig())
+       log.Log.Error(nil, "hello IC world")

        return pomerium_cmd.Run(ctx, r.src)

And this:

diff --git a/pomerium-src/authorize/internal/store/store.go b/pomerium-src/authorize/internal/store/store.go
index f9f9e08d..3dcac141 100644
--- a/pomerium-src/authorize/internal/store/store.go
+++ b/pomerium-src/authorize/internal/store/store.go
@@ -31,6 +31,8 @@ type Store struct {

 // New creates a new Store.
 func New() *Store {
+       ctx := context.TODO()
+       log.Error(ctx).Err(nil).Msg("hello pom world")
        return &Store{
                Store: inmem.New(),


ingress-controller% docker build -t ${TAG} . && docker push ${TAG}

Also, my docker is podman, so I added "build --network=host" for a speedup.

Redeploy to k8s:

% kubectl rollout restart -n pomerium deploy/pomerium

Use tab-complete (or a selector) to describe the right pod:

% kubectl describe -n pomerium pod/pomerium-79954677c-qzhpj

    Image:         bang5:5000/pomerium_ic_local:latest
    Image ID:      bang5:5000/pomerium_ic_local@sha256:0edbab12576bc68054b4f900f96318c04df4fcb9c1f588f2e4c2454cbe3b8a08

Not sure why this 0edb id doesn't match the 'successfully tagged' 0317 one above.

However, we still have great success!

% kubectl logs -n pomerium deploy/pomerium | grep hello

{"level":"error","ts":"2023-07-17T01:32:27Z","msg":"hello IC world"}
{"level":"error","time":"2023-07-17T01:32:27Z","message":"hello pom world"}