Mercurial > code > home > repos > pomerium
annotate tasks.py @ 36:6f8a6ccb2407 default tip
debugging
| author | drewp@bigasterisk.com |
|---|---|
| date | Wed, 21 Jun 2023 23:03:05 -0700 |
| parents | b1f75b0584f3 |
| children |
| rev | line source |
|---|---|
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
1 import time |
| 6 | 2 from invoke import task |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
3 |
|
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
4 |
| 6 | 5 @task |
| 6 def run(ctx): | |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
7 ctx.run("kubectl delete -n pomerium job/pomerium-gen-secrets --ignore-not-found", echo=True) |
|
34
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
8 ctx.run("kubectl kustomize upstream | kubectl apply -f -", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
9 print("let CM start up") |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
10 time.sleep(15) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
11 ctx.run("kubectl apply -f config/05-idp-secret.yaml", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
12 ctx.run("kubectl apply -f config/dns-secret.yaml", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
13 # ctx.run("kubectl apply -f config/06-postgres.yaml", echo=True) |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
14 ctx.run("kubectl apply -f config/51-pomerium-production-issuer.yaml", echo=True) |
|
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
15 ctx.run("kubectl apply -f config/51-pomerium-staging-issuer.yaml", echo=True) |
|
34
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
16 ctx.run("kubectl apply -f config/dns-issuers.yaml", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
17 ctx.run("./make_global.py no_cert | kubectl apply -f -", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
18 |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
19 ctx.run("./make_global.py output_pom_cert | kubectl apply -f -", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
20 # that will make infinite certs :( Clean up the redundant requests before LE ratelimits! |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
21 # k delete -n pomerium certificaterequests.cert-manager.io <tab> |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
22 |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
23 ctx.run("kubectl apply -f ingress/default.yaml", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
24 ctx.run("kubectl apply -f ingress/static.yaml", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
25 |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
26 # this may wait for |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
27 # 1) nothing; cert+secret exist |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
28 # 2) a letsencrypt session |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
29 # 3) a cert-manager delay before a LE session (e.g. 45 minutes) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
30 ctx.run("./make_global.py wait_for_cert | kubectl apply -f -", echo=True) |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
31 |
| 6 | 32 |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
33 @task |
|
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
34 def delete(ctx): |
|
34
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
35 ctx.run("kubectl delete pomerium/global --ignore-not-found", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
36 ctx.run("kubectl delete -f config/dns-issuers.yaml --ignore-not-found", echo=True) |
| 29 | 37 ctx.run("kubectl delete -f config/51-pomerium-staging-issuer.yaml --ignore-not-found", echo=True) |
| 38 ctx.run("kubectl delete -f config/51-pomerium-production-issuer.yaml --ignore-not-found", echo=True) | |
|
34
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
39 ctx.run("kubectl delete -f config/06-postgres.yaml --ignore-not-found", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
40 ctx.run("kubectl delete -f config/05-idp-secret.yaml --ignore-not-found", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
41 |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
42 # the kustomize workloads and svcs |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
43 for type, ns, name in [ |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
44 ('job', 'pomerium', 'pomerium-gen-secrets'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
45 ('deploy', 'cert-manager', 'cert-manager'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
46 ('deploy', 'cert-manager', 'cert-manager-cainjector'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
47 ('deploy', 'cert-manager', 'cert-manager-webhook'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
48 ('deploy', 'pomerium', 'pomerium'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
49 ('service', 'cert-manager', 'cert-manager'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
50 ('service', 'cert-manager', 'cert-manager-webhook'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
51 ('service', 'pomerium', 'pomerium-metrics'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
52 ('service', 'pomerium', 'pomerium-proxy'), |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
53 ]: |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
54 ctx.run(f"kubectl delete -n {ns} {type} {name} --ignore-not-found", echo=True) |
|
b1f75b0584f3
redo 'run' task and 'delete' (less tested)
drewp@bigasterisk.com
parents:
29
diff
changeset
|
55 |
| 29 | 56 ctx.run("kubectl delete -n pomerium job/pomerium-gen-secrets --ignore-not-found", echo=True) |
|
24
b53ab97e8979
reorganize, and add two retry loops to try to get everything to startup in one 'inv run'
drewp@bigasterisk.com
parents:
23
diff
changeset
|
57 |
| 11 | 58 |
| 59 ''' | |
| 60 troubleshooting, based on | |
| 61 https://cert-manager.io/docs/troubleshooting/ | |
| 62 then | |
| 63 https://cert-manager.io/docs/concepts/acme-orders-challenges/ | |
| 64 | |
| 65 I had these open: | |
| 66 ✨ dash(pts/31):~% watch 'kubectl describe -n pomerium issuers.cert-manager.io letsencrypt-staging' | |
| 67 ✨ dash(pts/31):~% watch 'kubectl describe -n pomerium issuers.cert-manager.io letsencrypt-prod' | |
| 68 ✨ dash(pts/29):~% watch "kubectl get -n pomerium certificates.cert-manager.io -o wide" | |
| 69 ✨ dash(pts/36):~% watch 'kubectl describe -n pomerium certificaterequests.cert-manager.io' | |
| 70 ✨ dash(pts/37):~% watch 'kubectl describe -n pomerium orders.acme.cert-manager.io' | |
| 71 ✨ dash(pts/38):~% watch 'kubectl describe -n pomerium challenges.acme.cert-manager.io ' | |
| 72 | |
| 73 then i checked clusterissuer vs issuer, the ns of the 60-auth-cert.yaml resources, | |
| 74 and i often restarted cert-manager and eventually pomerium too. 10-pom-pom.yaml last line | |
| 75 may need to be toggled. | |
| 76 | |
| 77 The 'cm-acme-http-solver' ingress for LE comes and goes but i didn't have to force it to exist. | |
| 78 | |
| 79 Didn't need 04-gen-secrets-job.yaml | |
| 80 | |
| 81 Also, CM says this a lot which means it may be afraid to renew bigasterisk.com | |
| 82 | |
| 83 I1213 07:00:01.946799 1 sync.go:394] cert-manager/controller/ingress-shim "msg"="certificate resource is not owned by this object. refusing to update non-owned certificate resource for object" "related_resource_kind"="Certificate" "related_resource_name"="bigasterisk.com-tls" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Ingress" "resource_name"="registry" "resource_namespace"="default" "resource_version"="v1" | |
| 84 | |
| 85 ''' |