Mercurial > code > home > repos > light9
annotate bin/collector @ 1939:6f49dc917aa3
start vidref web version. v4l camera frames to web page is working
Ignore-this: 34bcc3b6149a1a3bed31aa5f32a4ddc6
author | Drew Perttula <drewp@bigasterisk.com> |
---|---|
date | Mon, 03 Jun 2019 09:50:29 +0000 |
parents | f29e26811206 |
children | a745bee5c419 |
rev | line source |
---|---|
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
1 #!bin/python |
1590 | 2 """ |
3 Collector receives device attrs from multiple senders, combines | |
4 them, and sends output attrs to hardware. The combining part has | |
5 custom code for some attributes. | |
6 | |
7 Input can be over http or zmq. | |
8 """ | |
9 | |
1697
5c04a54df635
fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents:
1692
diff
changeset
|
10 from run_local import log |
5c04a54df635
fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents:
1692
diff
changeset
|
11 |
1493
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
12 from twisted.internet import reactor, utils |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
13 import json |
1289
5a4e74f1e36a
Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents:
1288
diff
changeset
|
14 import logging |
1372
f427801da9f6
collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents:
1307
diff
changeset
|
15 import optparse |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
16 import traceback |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
17 import cyclone.web, cyclone.websocket |
1289
5a4e74f1e36a
Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents:
1288
diff
changeset
|
18 from greplin import scales |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
19 |
1861 | 20 from cycloneerr import PrettyErrorHandler |
1866
3c523c71da29
pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents:
1861
diff
changeset
|
21 from light9 import networking |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
22 from light9.collector.collector import Collector |
1866
3c523c71da29
pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents:
1861
diff
changeset
|
23 from light9.collector.weblisteners import WebListeners |
1926
1a7e5b07bf17
use my greplin fork's stats/ code instead of an old local one
Drew Perttula <drewp@bigasterisk.com>
parents:
1922
diff
changeset
|
24 from greplin.scales.cyclonehandler import StatsHandler |
1866
3c523c71da29
pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents:
1861
diff
changeset
|
25 from light9.namespaces import L9 |
3c523c71da29
pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents:
1861
diff
changeset
|
26 from light9.zmqtransport import parseJsonMessage, startZmq |
3c523c71da29
pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents:
1861
diff
changeset
|
27 from rdfdb.syncedgraph import SyncedGraph |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
28 |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
29 from light9.collector.output import Udmx, DummyOutput # noqa |
1858 | 30 |
31 | |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
32 class Updates(cyclone.websocket.WebSocketHandler): |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
33 |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
34 def connectionMade(self, *args, **kwargs): |
1543
c8cffe82b537
collector gui updates
Drew Perttula <drewp@bigasterisk.com>
parents:
1541
diff
changeset
|
35 log.info('socket connect %s', self) |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
36 self.settings.listeners.addClient(self) |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
37 |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
38 def connectionLost(self, reason): |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
39 self.settings.listeners.delClient(self) |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
40 |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
41 def messageReceived(self, message): |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
42 json.loads(message) |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
43 |
1858 | 44 |
1927 | 45 stats = scales.collection( |
46 '/webServer', | |
1937
f29e26811206
pmfstats now update at 1hz. some ui improvements to stats lines.
Drew Perttula <drewp@bigasterisk.com>
parents:
1927
diff
changeset
|
47 scales.PmfStat('setAttr', recalcPeriod=1), |
1927 | 48 scales.RecentFpsStat('setAttrFps'), |
1922
11e2f63bb2f2
more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents:
1916
diff
changeset
|
49 ) |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
50 |
1858 | 51 |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
52 class Attrs(PrettyErrorHandler, cyclone.web.RequestHandler): |
1858 | 53 |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
54 def put(self): |
1922
11e2f63bb2f2
more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents:
1916
diff
changeset
|
55 stats.setAttrFps.mark() |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
56 with stats.setAttr.time(): |
1858 | 57 client, clientSession, settings, sendTime = parseJsonMessage( |
58 self.request.body) | |
59 self.settings.collector.setAttrs(client, clientSession, settings, | |
60 sendTime) | |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
61 self.set_status(202) |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
62 |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
63 |
1490
649d482737e0
start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents:
1489
diff
changeset
|
64 def launch(graph, doLoadTest=False): |
1530 | 65 try: |
66 # todo: drive outputs with config files | |
67 outputs = [ | |
1927 | 68 Udmx(L9['output/dmxA/'], bus=None, address=None, |
69 lastDmxChannel=221), | |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
70 DummyOutput(L9['output/dmxB/']), |
1530 | 71 ] |
1799
0bb7b9df12e5
collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents:
1798
diff
changeset
|
72 except Exception: |
0bb7b9df12e5
collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents:
1798
diff
changeset
|
73 log.error("setting up outputs:") |
1530 | 74 traceback.print_exc() |
75 raise | |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
76 listeners = WebListeners() |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
77 c: Collector = Collector(graph, outputs, listeners) |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
78 |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
79 startZmq(networking.collectorZmq.port, c) |
1858 | 80 |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
81 reactor.listenTCP(networking.collector.port, |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
82 cyclone.web.Application(handlers=[ |
1858 | 83 (r'/()', cyclone.web.StaticFileHandler, { |
84 "path": "light9/collector/web", | |
85 "default_filename": "index.html" | |
86 }), | |
1541
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
87 (r'/updates', Updates), |
c1bf296b0a74
collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents:
1530
diff
changeset
|
88 (r'/attrs', Attrs), |
1926
1a7e5b07bf17
use my greplin fork's stats/ code instead of an old local one
Drew Perttula <drewp@bigasterisk.com>
parents:
1922
diff
changeset
|
89 (r'/stats/(.*)', StatsHandler, { |
1a7e5b07bf17
use my greplin fork's stats/ code instead of an old local one
Drew Perttula <drewp@bigasterisk.com>
parents:
1922
diff
changeset
|
90 'serverName': 'collector' |
1a7e5b07bf17
use my greplin fork's stats/ code instead of an old local one
Drew Perttula <drewp@bigasterisk.com>
parents:
1922
diff
changeset
|
91 }), |
1858 | 92 ], |
93 collector=c, | |
94 listeners=listeners), | |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
95 interface='::') |
1289
5a4e74f1e36a
Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents:
1288
diff
changeset
|
96 log.info('serving http on %s, zmq on %s', networking.collector.port, |
5a4e74f1e36a
Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents:
1288
diff
changeset
|
97 networking.collectorZmq.port) |
1490
649d482737e0
start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents:
1489
diff
changeset
|
98 if doLoadTest: |
1493
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
99 # in a subprocess since we don't want this client to be |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
100 # cooperating with the main event loop and only sending |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
101 # requests when there's free time |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
102 def afterWarmup(): |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
103 log.info('running collector_loadtest') |
1858 | 104 d = utils.getProcessValue('bin/python', |
105 ['bin/collector_loadtest.py']) | |
106 | |
1493
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
107 def done(*a): |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
108 log.info('loadtest done') |
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
109 reactor.stop() |
1858 | 110 |
1493
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
111 d.addCallback(done) |
1858 | 112 |
1493
4294ed82ee16
move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents:
1492
diff
changeset
|
113 reactor.callLater(2, afterWarmup) |
1858 | 114 |
115 | |
1307
8863b4485fd4
collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1289
diff
changeset
|
116 def main(): |
1372
f427801da9f6
collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents:
1307
diff
changeset
|
117 parser = optparse.OptionParser() |
1858 | 118 parser.add_option("-v", |
119 "--verbose", | |
120 action="store_true", | |
1372
f427801da9f6
collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents:
1307
diff
changeset
|
121 help="logging.DEBUG") |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
122 parser.add_option("--logdmx", action="store_true", help="log all dmx sends") |
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
123 |
1858 | 124 parser.add_option("--loadtest", |
125 action="store_true", | |
1490
649d482737e0
start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents:
1489
diff
changeset
|
126 help="call myself with some synthetic load then exit") |
1372
f427801da9f6
collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents:
1307
diff
changeset
|
127 (options, args) = parser.parse_args() |
f427801da9f6
collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents:
1307
diff
changeset
|
128 log.setLevel(logging.DEBUG if options.verbose else logging.INFO) |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
129 logging.getLogger('output').setLevel(logging.DEBUG) |
1307
8863b4485fd4
collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1289
diff
changeset
|
130 |
1884
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
131 logging.getLogger('output.allDmx').setLevel( |
5cde72dfdc22
change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents:
1881
diff
changeset
|
132 logging.DEBUG if options.logdmx else logging.INFO) |
1596
7d5d6e7bc526
collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents:
1595
diff
changeset
|
133 logging.getLogger('colormath').setLevel(logging.INFO) |
1858 | 134 |
1307
8863b4485fd4
collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1289
diff
changeset
|
135 graph = SyncedGraph(networking.rdfdb.url, "collector") |
8863b4485fd4
collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1289
diff
changeset
|
136 |
1858 | 137 graph.initiallySynced.addCallback(lambda _: launch(graph, options.loadtest) |
138 ).addErrback(lambda e: reactor.crash()) | |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
139 reactor.run() |
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
140 |
1858 | 141 |
1288
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
142 if __name__ == '__main__': |
5e76c8fd8a03
rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff
changeset
|
143 main() |