annotate service/wifi/wifi.py @ 421:47d7dd31bb2c

port to py3 Ignore-this: e1a2e6bb730111e76f5a5dd2366d498a
author drewp@bigasterisk.com
date Sat, 30 Mar 2019 18:27:17 -0700
parents service/wifi/tomatoWifi.py@a530d9c5b280
children 19460b3f7baf
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
1 #!/usr/bin/python
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
2 """
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
3 scrape the tomato router status pages to see who's connected to the
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
4 wifi access points. Includes leases that aren't currently connected.
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
5
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
6 Returns:
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
7 json listing (for magma page)
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
8 rdf graph (for reasoning)
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
9 activity stream, when we start saving history
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
10
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
11 Todo: this should be the one polling and writing to mongo, not entrancemusic
162
bb70eaa45666 whitespace
drewp@bigasterisk.com
parents: 161
diff changeset
12
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
13 """
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
14 from __future__ import division
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
15 import sys, cyclone.web, json, traceback, time, pystache, datetime, logging
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
16 from cyclone.httpclient import fetch
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
17
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
18 from dateutil import tz
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
19 from twisted.internet import reactor, task
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
20 from twisted.internet.defer import inlineCallbacks
189
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
21 import docopt
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
22 from influxdb import InfluxDBClient
417
10ec2744fe8a mongodb api upgrade
drewp@bigasterisk.com
parents: 383
diff changeset
23 from pymongo import MongoClient as Connection, DESCENDING
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
24 from rdflib import Namespace, Literal, URIRef, ConjunctiveGraph
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
25
421
47d7dd31bb2c port to py3
drewp@bigasterisk.com
parents: 420
diff changeset
26 from scrape import Wifi
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
27
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
28 from patchablegraph import PatchableGraph, CycloneGraphEventsHandler, CycloneGraphHandler
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
29
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
30 from cycloneerr import PrettyErrorHandler
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
31 from logsetup import log
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
32
36
0ab069867c64 rdflib and jsonlib api updates
drewp@bigasterisk.com
parents: 1
diff changeset
33
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
34 DEV = Namespace("http://projects.bigasterisk.com/device/")
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
35 ROOM = Namespace("http://projects.bigasterisk.com/room/")
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
36 reasoning = "http://bang:9071/"
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
37
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
38 class Index(PrettyErrorHandler, cyclone.web.RequestHandler):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
39 def get(self):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
40
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
41 age = time.time() - self.settings.poller.lastPollTime
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
42 if age > 10:
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
43 raise ValueError("poll data is stale. age=%s" % age)
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
44
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
45 self.set_header("Content-Type", "text/html")
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
46 self.write(open("index.html").read())
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
47
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
48 def whenConnected(mongo, macThatIsNowConnected):
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
49 lastArrive = None
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
50 for ev in mongo.find({'address': macThatIsNowConnected.upper()},
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
51 sort=[('created', -1)],
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
52 max_scan=100000):
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
53 if ev['action'] == 'arrive':
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
54 lastArrive = ev
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
55 if ev['action'] == 'leave':
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
56 break
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
57 if lastArrive is None:
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
58 raise ValueError("no past arrivals")
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
59
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
60 return lastArrive['created']
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
61
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
62 def connectedAgoString(conn):
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
63 return web.utils.datestr(
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
64 conn.astimezone(tz.tzutc()).replace(tzinfo=None))
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
65
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
66 class Table(PrettyErrorHandler, cyclone.web.RequestHandler):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
67 def get(self):
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
68 def rowDict(row):
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
69 row['cls'] = "signal" if row.get('connected') else "nosignal"
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
70 if 'name' not in row:
74
bca6d6c63bdc handle wifi users with no clientHostname
drewp@bigasterisk.com
parents: 62
diff changeset
71 row['name'] = row.get('clientHostname', '-')
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
72 if 'signal' not in row:
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
73 row['signal'] = 'yes' if row.get('connected') else 'no'
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
74
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
75 try:
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
76 conn = whenConnected(self.settings.mongo, row.get('mac', '??'))
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
77 row['connectedAgo'] = connectedAgoString(conn)
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
78 except ValueError:
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
79 row['connectedAgo'] = 'yes' if row.get('connected') else ''
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
80 row['router'] = row.get('ssid', '')
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
81 return row
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
82
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
83 self.set_header("Content-Type", "application/xhtml+xml")
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
84 self.write(pystache.render(
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
85 open("table.mustache").read(),
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
86 dict(
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
87 rows=sorted(map(rowDict, self.settings.poller.lastAddrs),
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
88 key=lambda a: (not a.get('connected'),
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
89 a.get('name'))))))
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
90
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
91
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
92 class Json(PrettyErrorHandler, cyclone.web.RequestHandler):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
93 def get(self):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
94 self.set_header("Content-Type", "application/json")
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
95 age = time.time() - self.settings.poller.lastPollTime
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
96 if age > 10:
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
97 raise ValueError("poll data is stale. age=%s" % age)
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
98 self.write(json.dumps({"wifi" : self.settings.poller.lastAddrs,
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
99 "dataAge" : age}))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
100
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
101 class Poller(object):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
102 def __init__(self, wifi, mongo):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
103 self.wifi = wifi
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
104 self.mongo = mongo
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
105 self.lastAddrs = []
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
106 self.lastWithSignal = []
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
107 self.lastPollTime = 0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
108
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
109 def assertCurrent(self):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
110 dt = time.time() - self.lastPollTime
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
111 assert dt < 10, "last poll was %s sec ago" % dt
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
112
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
113 @inlineCallbacks
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
114 def poll(self):
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
115 now = int(time.time())
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
116
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
117 # UVA mode:
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
118 addDhcpData = lambda *args: None
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
119
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
120 try:
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
121 newAddrs = yield self.wifi.getPresentMacAddrs()
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
122 addDhcpData(newAddrs)
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
123
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
124 newWithSignal = [a for a in newAddrs if a.get('connected')]
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
125
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
126 actions = self.computeActions(newWithSignal)
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
127 points = []
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
128 for action in actions:
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
129 log.info("action: %s", action)
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
130 action['created'] = datetime.datetime.now(tz.gettz('UTC'))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
131 mongo.save(action)
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
132 points.append(
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
133 self.influxPoint(now, action['address'].lower(),
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
134 1 if action['action'] == 'arrive' else 0))
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
135 try:
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
136 self.doEntranceMusic(action)
421
47d7dd31bb2c port to py3
drewp@bigasterisk.com
parents: 420
diff changeset
137 except Exception as e:
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
138 log.error("entrancemusic error: %r", e)
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
139
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
140 if now // 3600 > self.lastPollTime // 3600:
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
141 log.info('hourly writes')
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
142 for addr in newWithSignal:
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
143 points.append(self.influxPoint(now, addr['mac'].lower(), 1))
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
144
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
145 influx.write_points(points, time_precision='s')
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
146 self.lastWithSignal = newWithSignal
50
56eeda98cac5 ping reasoning
drewp@bigasterisk.com
parents: 36
diff changeset
147 if actions: # this doesn't currently include signal strength changes
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
148 fetch(reasoning + "immediateUpdate",
279
57c9dbd2fdd0 immediate mode needs a PUT request
drewp@bigasterisk.com
parents: 189
diff changeset
149 method='PUT',
52
875a37be1228 fix fetch calls in tomatowifi
drewp@bigasterisk.com
parents: 51
diff changeset
150 timeout=2,
875a37be1228 fix fetch calls in tomatowifi
drewp@bigasterisk.com
parents: 51
diff changeset
151 headers={'user-agent': ['tomatoWifi']}).addErrback(log.warn)
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
152 self.lastAddrs = newAddrs
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
153 self.lastPollTime = now
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
154
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
155 self.updateGraph(masterGraph)
421
47d7dd31bb2c port to py3
drewp@bigasterisk.com
parents: 420
diff changeset
156 except Exception as e:
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
157 log.error("poll error: %r\n%s", e, traceback.format_exc())
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
158
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
159 def influxPoint(self, now, address, value):
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
160 return {
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
161 'measurement': 'presence',
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
162 'tags': {'sensor': 'wifi', 'address': address,},
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
163 'fields': {'value': value},
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
164 'time': now,
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
165 }
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
166
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
167 def computeActions(self, newWithSignal):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
168 actions = []
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
169
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
170 def makeAction(addr, act):
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
171 d = dict(sensor="wifi",
175
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
172 address=addr.get('mac').upper(), # mongo data is legacy uppercase
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
173 name=addr.get('name'),
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
174 networkName=addr.get('clientHostname'),
c81a451f9b26 rewrites for better graph export, removal of dhcp reader
drewp@bigasterisk.com
parents: 162
diff changeset
175 action=act)
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
176 if act == 'arrive' and 'ip' in addr:
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
177 # this won't cover the possible case that you get on
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
178 # wifi but don't have an ip yet. We'll record an
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
179 # action with no ip and then never record your ip.
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
180 d['ip'] = addr['ip']
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
181 return d
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
182
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
183 for addr in newWithSignal:
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
184 if addr['mac'] not in [r['mac'] for r in self.lastWithSignal]:
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
185 actions.append(makeAction(addr, 'arrive'))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
186
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
187 for addr in self.lastWithSignal:
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
188 if addr['mac'] not in [r['mac'] for r in newWithSignal]:
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
189 actions.append(makeAction(addr, 'leave'))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
190
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
191 return actions
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
192
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
193
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
194 # these need to move out to their own service
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
195 def doEntranceMusic(self, action):
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
196 import restkit, json
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
197 dt = self.deltaSinceLastArrive(action['name'])
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
198 log.debug("dt=%s", dt)
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
199 if dt > datetime.timedelta(hours=1):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
200 hub = restkit.Resource(
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
201 # PSHB not working yet; "http://bang:9030/"
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
202 "http://slash:9049/"
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
203 )
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
204 action = action.copy()
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
205 del action['created']
1
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
206 del action['_id']
26a6cf58743d entrancemusic improve logging
drewp@bigasterisk.com
parents: 0
diff changeset
207 log.info("post to %s", hub)
383
8f5a16a55f64 various docker setups and build fixes
drewp@bigasterisk.com
parents: 340
diff changeset
208 hub.post("visitorNet", payload=json.dumps(action))
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
209
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
210 def deltaSinceLastArrive(self, name):
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
211 results = list(self.mongo.find({'name' : name}).sort('created',
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
212 DESCENDING).limit(1))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
213 if not results:
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
214 return datetime.timedelta.max
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
215 now = datetime.datetime.now(tz.gettz('UTC'))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
216 last = results[0]['created'].replace(tzinfo=tz.gettz('UTC'))
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
217 return now - last
51
d2842eedd56d rewrite tomatowifi from restkit to cyclone httpclient
drewp@bigasterisk.com
parents: 50
diff changeset
218
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
219 def updateGraph(self, masterGraph):
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
220
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
221 g = ConjunctiveGraph()
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
222 ctx = DEV['wifi']
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
223
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
224 # someday i may also record specific AP and their strength,
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
225 # for positioning. But many users just want to know that the
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
226 # device is connected to some bigasterisk AP.
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
227 age = time.time() - self.lastPollTime
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
228 if age > 10:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
229 raise ValueError("poll data is stale. age=%s" % age)
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
230
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
231 for dev in self.lastAddrs:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
232 if not dev.get('connected'):
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
233 continue
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
234 uri = URIRef("http://bigasterisk.com/mac/%s" % dev['mac'].lower())
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
235 g.add((uri, ROOM['macAddress'], Literal(dev['mac'].lower()), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
236
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
237 g.add((uri, ROOM['connected'], {
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
238 'wireless': URIRef("http://bigasterisk.com/wifiAccessPoints"),
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
239 '2.4G': URIRef("http://bigasterisk.com/wifiAccessPoints"),
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
240 '5G': URIRef("http://bigasterisk.com/wifiAccessPoints"),
418
b3935eac90e1 parse another version of 'unknown'
drewp@bigasterisk.com
parents: 417
diff changeset
241 '-': URIRef("http://bigasterisk.com/wifiUnknownConnectionType"),
b3935eac90e1 parse another version of 'unknown'
drewp@bigasterisk.com
parents: 417
diff changeset
242 'Unknown': URIRef("http://bigasterisk.com/wifiUnknownConnectionType"),
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
243 'wired': URIRef("http://bigasterisk.com/houseOpenNet")}[dev['contype']], ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
244 if 'clientHostname' in dev and dev['clientHostname']:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
245 g.add((uri, ROOM['wifiNetworkName'], Literal(dev['clientHostname']), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
246 if 'name' in dev and dev['name']:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
247 g.add((uri, ROOM['deviceName'], Literal(dev['name']), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
248 if 'signal' in dev:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
249 g.add((uri, ROOM['signalStrength'], Literal(dev['signal']), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
250 if 'model' in dev:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
251 g.add((uri, ROOM['networkModel'], Literal(dev['model']), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
252 try:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
253 conn = whenConnected(mongo, dev['mac'])
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
254 except ValueError:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
255 traceback.print_exc()
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
256 pass
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
257 else:
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
258 g.add((uri, ROOM['connectedAgo'],
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
259 Literal(connectedAgoString(conn)), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
260 g.add((uri, ROOM['connected'], Literal(conn), ctx))
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
261 masterGraph.setToGraph(g)
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
262
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
263
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
264 if __name__ == '__main__':
189
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
265 args = docopt.docopt('''
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
266 Usage:
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
267 tomatoWifi [options]
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
268
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
269 Options:
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
270 -v, --verbose more logging
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
271 --port=<n> serve on port [default: 9070]
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
272 --poll=<freq> poll frequency [default: .2]
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
273 ''')
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
274 if args['--verbose']:
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
275 from twisted.python import log as twlog
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
276 twlog.startLogging(sys.stdout)
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
277 log.setLevel(10)
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
278 log.setLevel(logging.DEBUG)
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
279
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
280 mongo = Connection('bang', 27017, tz_aware=True)['visitor']['visitor']
309
68c2a5f1d563 wifi write to influxdb
drewp@bigasterisk.com
parents: 279
diff changeset
281 influx = InfluxDBClient('bang', 9060, 'root', 'root', 'main')
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
282
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
283 masterGraph = PatchableGraph()
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
284 wifi = Wifi()
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
285 poller = Poller(wifi, mongo)
189
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
286 task.LoopingCall(poller.poll).start(1/float(args['--poll']))
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
287
189
fc184ea1416d switch to docopt
drewp@bigasterisk.com
parents: 175
diff changeset
288 reactor.listenTCP(int(args['--port']),
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
289 cyclone.web.Application(
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
290 [
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
291 (r"/", Index),
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
292 (r'/json', Json),
340
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
293 (r'/graph', CycloneGraphHandler, {'masterGraph': masterGraph}),
70954bd426be wifi now uses PatchableGraph
drewp@bigasterisk.com
parents: 309
diff changeset
294 (r'/graph/events', CycloneGraphEventsHandler, {'masterGraph': masterGraph}),
62
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
295 (r'/table', Table),
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
296 #(r'/activity', Activity),
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
297 ],
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
298 wifi=wifi,
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
299 poller=poller,
f8cc3d1baa85 redo wifi scraper to work with zyxel router report page too. add last connected time (from mongo) to web table
drewp@bigasterisk.com
parents: 52
diff changeset
300 mongo=mongo))
421
47d7dd31bb2c port to py3
drewp@bigasterisk.com
parents: 420
diff changeset
301 import twisted; print('twisted', twisted.__version__)
0
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
302 reactor.run()