annotate tasks.py @ 234:3f7ebff619e3

much faster `inv all` since pyinfra bug was fixed
author drewp@bigasterisk.com
date Sun, 29 Oct 2023 21:27:10 -0700
parents 117cd86ffb53
children 3af02e24eaf9
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
1 from invoke import task
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
2
1
b664f1027992 system.py port from ansible
drewp@bigasterisk.com
parents: 0
diff changeset
3 cmd = '''
0
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
4 HOME=/root
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
5 export HOME
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
6 eval `keychain --quiet --eval id_ecdsa`
1550a6db59b3 first ported section from ansible. shorter, faster, clearer.
drewp@bigasterisk.com
parents:
diff changeset
7 cd /my/proj/infra
202
a5399e8b47b6 upgrade some ubuntu
drewp@bigasterisk.com
parents: 155
diff changeset
8 pdm run pyinfra '''
1
b664f1027992 system.py port from ansible
drewp@bigasterisk.com
parents: 0
diff changeset
9
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
10
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
11 def _run(ctx, s):
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
12 ctx.run(cmd + 'inventory.py %s' % s, pty=True, echo=True)
34
d4fb38f13c79 refactor dns and some other non-net setup
drewp@bigasterisk.com
parents: 23
diff changeset
13
103
8b8ef9d8f0fd dead code and templates, reformat, maybe a little refactor
drewp@bigasterisk.com
parents: 96
diff changeset
14
59
fa7a71b8a97f more dns checks, including from inside containers in k8s`
drewp@bigasterisk.com
parents: 54
diff changeset
15 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
16 def users(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
17 _run(ctx, 'users.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
18
59
fa7a71b8a97f more dns checks, including from inside containers in k8s`
drewp@bigasterisk.com
parents: 54
diff changeset
19
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
20 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
21 def ssh(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
22 _run(ctx, 'ssh.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
23
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
24
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
25 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
26 def system(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
27 _run(ctx, 'system.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
28
34
d4fb38f13c79 refactor dns and some other non-net setup
drewp@bigasterisk.com
parents: 23
diff changeset
29
d4fb38f13c79 refactor dns and some other non-net setup
drewp@bigasterisk.com
parents: 23
diff changeset
30 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
31 def apt(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
32 _run(ctx, 'apt.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
33
155
522f26c8f691 split apt.py from system.py
drewp@bigasterisk.com
parents: 149
diff changeset
34
522f26c8f691 split apt.py from system.py
drewp@bigasterisk.com
parents: 149
diff changeset
35 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
36 def packages(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
37 _run(ctx, 'packages.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
38
5
7e8c7de5b490 port wireguard setup
drewp@bigasterisk.com
parents: 3
diff changeset
39
3
61945df2a392 updates to work on recent raspbian installs
drewp@bigasterisk.com
parents: 1
diff changeset
40 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
41 def net(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
42 _run(ctx, 'net.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
43
12
15c5ce7c74b5 refactor, cleanup, split large deploys
drewp@bigasterisk.com
parents: 8
diff changeset
44
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
45 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
46 def dns(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
47 _run(ctx, 'dns.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
48
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
49
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
50 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
51 def dns_check(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
52 _run(ctx, 'dns_check.py -v')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
53
103
8b8ef9d8f0fd dead code and templates, reformat, maybe a little refactor
drewp@bigasterisk.com
parents: 96
diff changeset
54
68
4a28a6b26af4 new task
drewp@bigasterisk.com
parents: 61
diff changeset
55 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
56 def dns_k8s_check(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
57 ctx.run('env/bin/python dns_k8s_check.py', pty=True)
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
58
8
226f3c8419b2 kube setup
drewp@bigasterisk.com
parents: 7
diff changeset
59
14
ac4d24d01b68 add syncthing service
drewp@bigasterisk.com
parents: 12
diff changeset
60 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
61 def wireguard(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
62 _run(ctx, 'wireguard.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
63
14
ac4d24d01b68 add syncthing service
drewp@bigasterisk.com
parents: 12
diff changeset
64
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
65 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
66 def kube(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
67 _run(ctx, 'kube.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
68
8
226f3c8419b2 kube setup
drewp@bigasterisk.com
parents: 7
diff changeset
69
226f3c8419b2 kube setup
drewp@bigasterisk.com
parents: 7
diff changeset
70 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
71 def kube_bang(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
72 _run(ctx, 'kube.py --limit bang')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
73
16
c5784e3d4f8d mail/postfix setup on prime
drewp@bigasterisk.com
parents: 14
diff changeset
74
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
75 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
76 def sync(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
77 _run(ctx, 'sync.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
78
103
8b8ef9d8f0fd dead code and templates, reformat, maybe a little refactor
drewp@bigasterisk.com
parents: 96
diff changeset
79
96
b70070570e10 new task for building ntop from source
drewp@bigasterisk.com
parents: 72
diff changeset
80 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
81 def mail(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
82 _run(ctx, 'mail.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
83
110
de407da9224a homedir links
drewp@bigasterisk.com
parents: 103
diff changeset
84
de407da9224a homedir links
drewp@bigasterisk.com
parents: 103
diff changeset
85 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
86 def home(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
87 _run(ctx, 'home.py')
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
88
139
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
89
105b628e3bfa compact/dry
drewp@bigasterisk.com
parents: 136
diff changeset
90 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
91 def pipe(ctx):
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
92 ctx.run(cmd + 'inventory.py pipe.py --limit pipe', pty=True)
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
93
96
b70070570e10 new task for building ntop from source
drewp@bigasterisk.com
parents: 72
diff changeset
94
111
340d778a1682 start multikube experiment
drewp@bigasterisk.com
parents: 110
diff changeset
95 @task
205
826db3c40fa7 whitespace/etc
drewp@bigasterisk.com
parents: 202
diff changeset
96 def multikube(ctx): # danger- wipes previous k3s
111
340d778a1682 start multikube experiment
drewp@bigasterisk.com
parents: 110
diff changeset
97 from multikube_config import server_node, nodes
340d778a1682 start multikube experiment
drewp@bigasterisk.com
parents: 110
diff changeset
98 ctx.run(cmd + 'inventory.py multikube_wipe.py', pty=True)
340d778a1682 start multikube experiment
drewp@bigasterisk.com
parents: 110
diff changeset
99 ctx.run(cmd + f'inventory.py multikube.py --limit {server_node}', pty=True)
113
c520672d0930 closer to multi-node k3s working, but still doesn't quite network right
drewp@bigasterisk.com
parents: 111
diff changeset
100 if nodes:
136
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
101 ctx.run(cmd + f'inventory.py multikube.py {" ".join(f"--limit {n}" for n in nodes)}', pty=True)
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
102 print("""now try
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
103 curl http://10.43.0.10:9153/metrics | head
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
104 k3s kubectl run hello --image crccheck/hello-world
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
105 curl -m2 http://`kubectl get pod/hello '-o=jsonpath={.status.podIP}'`:8000
375515ab60ba multikube experiment junk. do not run
drewp@bigasterisk.com
parents: 126
diff changeset
106 """)
111
340d778a1682 start multikube experiment
drewp@bigasterisk.com
parents: 110
diff changeset
107
16
c5784e3d4f8d mail/postfix setup on prime
drewp@bigasterisk.com
parents: 14
diff changeset
108
60
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
109 @task
54
00d5cd21cade save ran_all.timestamp
drewp@bigasterisk.com
parents: 50
diff changeset
110 def all(ctx):
60
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
111 configs = [
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
112 'users.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
113 'ssh.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
114 'system.py',
155
522f26c8f691 split apt.py from system.py
drewp@bigasterisk.com
parents: 149
diff changeset
115 'apt.py',
60
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
116 'packages.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
117 'net.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
118 'dns.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
119 'wireguard.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
120 'kube.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
121 'sync.py',
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
122 'mail.py',
110
de407da9224a homedir links
drewp@bigasterisk.com
parents: 103
diff changeset
123 'home.py',
60
a949704defd0 new pyinfra
drewp@bigasterisk.com
parents: 59
diff changeset
124 ]
234
3f7ebff619e3 much faster `inv all` since pyinfra bug was fixed
drewp@bigasterisk.com
parents: 226
diff changeset
125 ctx.run(' '.join([cmd, 'inventory.py'] + configs), pty=True)
54
00d5cd21cade save ran_all.timestamp
drewp@bigasterisk.com
parents: 50
diff changeset
126 ctx.run('touch /my/proj/infra/ran_all.timestamp')
00d5cd21cade save ran_all.timestamp
drewp@bigasterisk.com
parents: 50
diff changeset
127
00d5cd21cade save ran_all.timestamp
drewp@bigasterisk.com
parents: 50
diff changeset
128
16
c5784e3d4f8d mail/postfix setup on prime
drewp@bigasterisk.com
parents: 14
diff changeset
129 @task
3
61945df2a392 updates to work on recent raspbian installs
drewp@bigasterisk.com
parents: 1
diff changeset
130 def get_fact(ctx, host='dash', fact='server.LinuxDistribution'):
8
226f3c8419b2 kube setup
drewp@bigasterisk.com
parents: 7
diff changeset
131 ctx.run(cmd + f'{host} -vv fact {fact}', pty=True)