Mercurial > code > home > repos > homeauto

annotate service/wifi/scrape_unmaintained.py @ 1369:42f4fed9cd25

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
no more immediateUpdate since we push patch events now. and the code was broken for py3 anyway Ignore-this: 54edfd3885f749aa5ef3be2051c2f40f darcs-hash:0f1ccb011e9cf29bf9e2389de3833cbda3550631
author drewp <drewp@bigasterisk.com>
date Sun, 05 May 2019 17:09:12 -0700
parents 7de8f0cd3392
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1226
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
1
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
2 @inlineCallbacks
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
3 def loadUvaData():
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
4 import lxml.html.soupparser
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
5
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
6 config = json.load(open("priv-uva.json"))
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
7 headers = {'Authorization': ['Basic %s' % config['userPass'].encode('base64').strip()]}
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
8 resp = yield fetch('http://10.2.0.2/wlstationlist.cmd', headers=headers)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
9 root = lxml.html.soupparser.fromstring(resp.body)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
10 byMac = {}
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
11 for tr in root.cssselect('tr'):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
12 mac, connected, auth, ssid, iface = [td.text_content().strip() for td in tr.getchildren()]
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
13 if mac == "MAC":
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
14 continue
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
15 connected = connected.lower() == 'yes'
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
16 byMac[mac] = dict(mac=mac, connected=connected, auth=auth == 'Yes', ssid=ssid, iface=iface)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
17
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
18 resp = yield fetch('http://10.2.0.2/DHCPTable.asp', headers=headers)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
19 for row in re.findall(r'new AAA\((.*)\)', resp.body):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
20 clientHostname, ipaddr, mac, expires, iface = [s.strip("'") for s in row.rsplit(',', 4)]
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
21 if clientHostname == 'wlanadv.none':
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
22 continue
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
23 byMac.setdefault(mac, {}).update(dict(
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
24 clientHostname=clientHostname, connection=iface, ipaddr=ipaddr, dhcpExpires=expires))
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
25
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
26 returnValue(sorted(byMac.values()))
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
27
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
28 @inlineCallbacks
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
29 def loadCiscoData():
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
30 config = json.load(open("priv-uva.json"))
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
31 headers = {'Authorization': ['Basic %s' % config['userPass'].encode('base64').strip()]}
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
32 print(headers)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
33 resp = yield fetch('http://10.2.0.2/', headers=headers)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
34 print(resp.body)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
35 returnValue([])
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
36
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
37
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
38 def jsValue(js, variableName):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
39 # using literal_eval instead of json parser to handle the trailing commas
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
40 val = re.search(variableName + r'\s*=\s*(.*?);', js, re.DOTALL).group(1)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
41 return ast.literal_eval(val)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
42
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
43
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
44 def _parseZyxel(self, data, routerName):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
45 import lxml.html.soupparser
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
46
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
47 root = lxml.html.soupparser.fromstring(data)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
48 for tr in root.cssselect('tr'):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
49 mac, assoc, uth, ssid, iface = [td.text_content().strip() for td in tr.getchildren()]
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
50 if mac == "MAC":
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
51 continue
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
52 assoc = assoc.lower() == 'yes'
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
53 yield dict(router=routerName, mac=mac, assoc=assoc, connected=assoc)
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
54
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
55 def _parseTomato(self, data, routerName):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
56 for iface, mac, signal in jsValue(data, 'wldev'):
7de8f0cd3392 very big rewrite. py3; orbi-only for now; n3 config file; delete or move out dead code
drewp <drewp@bigasterisk.com>
parents:
diff changeset
57 yield dict(router=routerName, mac=mac, signal=signal, connected=bool(signal))