Mercurial > code > home > repos > light9

annotate bin/collector @ 1922:11e2f63bb2f2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
more stats to measure sequencer framerate better Ignore-this: 5df74b41a9847296432a31d248b31857
author Drew Perttula <drewp@bigasterisk.com>
date Sat, 01 Jun 2019 23:43:44 +0000
parents 070c6d185276
children 1a7e5b07bf17
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
1 #!bin/python
1590
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
2 """
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
3 Collector receives device attrs from multiple senders, combines
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
4 them, and sends output attrs to hardware. The combining part has
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
5 custom code for some attributes.
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
6
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
7 Input can be over http or zmq.
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
8 """
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
9
1697
5c04a54df635 fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents: 1692
diff changeset
10 from run_local import log
5c04a54df635 fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents: 1692
diff changeset
11
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
12 from twisted.internet import reactor, utils
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
13 import json
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
14 import logging
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
15 import optparse
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
16 import traceback
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
17 import cyclone.web, cyclone.websocket
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
18 from greplin import scales
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
19
1861
40cc863d2b63 start py3 and other dep fixes
drewp@bigasterisk.com
parents: 1860
diff changeset
20 from cycloneerr import PrettyErrorHandler
1866
3c523c71da29 pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents: 1861
diff changeset
21 from light9 import networking
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
22 from light9.collector.collector import Collector
1866
3c523c71da29 pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents: 1861
diff changeset
23 from light9.collector.weblisteners import WebListeners
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
24 from light9.greplin_cyclone import StatsForCyclone
1866
3c523c71da29 pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents: 1861
diff changeset
25 from light9.namespaces import L9
3c523c71da29 pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents: 1861
diff changeset
26 from light9.zmqtransport import parseJsonMessage, startZmq
3c523c71da29 pyflakes cleanups and some refactors
Drew Perttula <drewp@bigasterisk.com>
parents: 1861
diff changeset
27 from rdfdb.syncedgraph import SyncedGraph
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
28
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
29 from light9.collector.output import Udmx, DummyOutput # noqa
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
30
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
31
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
32 class Updates(cyclone.websocket.WebSocketHandler):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
33
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
34 def connectionMade(self, *args, **kwargs):
1543
c8cffe82b537 collector gui updates
Drew Perttula <drewp@bigasterisk.com>
parents: 1541
diff changeset
35 log.info('socket connect %s', self)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
36 self.settings.listeners.addClient(self)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
37
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
38 def connectionLost(self, reason):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
39 self.settings.listeners.delClient(self)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
40
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
41 def messageReceived(self, message):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
42 json.loads(message)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
43
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
44
1922
11e2f63bb2f2 more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents: 1916
diff changeset
45 stats = scales.collection('/webServer',
11e2f63bb2f2 more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents: 1916
diff changeset
46 scales.PmfStat('setAttr'),
11e2f63bb2f2 more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents: 1916
diff changeset
47 scales.RecentFpsStat('setAttrFps'),
11e2f63bb2f2 more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents: 1916
diff changeset
48 )
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
49
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
50
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
51 class Attrs(PrettyErrorHandler, cyclone.web.RequestHandler):
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
52
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
53 def put(self):
1922
11e2f63bb2f2 more stats to measure sequencer framerate better
Drew Perttula <drewp@bigasterisk.com>
parents: 1916
diff changeset
54 stats.setAttrFps.mark()
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
55 with stats.setAttr.time():
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
56 client, clientSession, settings, sendTime = parseJsonMessage(
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
57 self.request.body)
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
58 self.settings.collector.setAttrs(client, clientSession, settings,
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
59 sendTime)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
60 self.set_status(202)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
61
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
62
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
63 def launch(graph, doLoadTest=False):
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
64 try:
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
65 # todo: drive outputs with config files
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
66 outputs = [
1916
070c6d185276 collector takes an explicit dmx message length (again). more collector timing
Drew Perttula <drewp@bigasterisk.com>
parents: 1884
diff changeset
67 Udmx(L9['output/dmxA/'], bus=None, address=None, lastDmxChannel=221),
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
68 DummyOutput(L9['output/dmxB/']),
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
69 ]
1799
0bb7b9df12e5 collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents: 1798
diff changeset
70 except Exception:
0bb7b9df12e5 collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents: 1798
diff changeset
71 log.error("setting up outputs:")
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
72 traceback.print_exc()
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
73 raise
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
74 listeners = WebListeners()
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
75 c: Collector = Collector(graph, outputs, listeners)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
76
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
77 startZmq(networking.collectorZmq.port, c)
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
78
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
79 reactor.listenTCP(networking.collector.port,
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
80 cyclone.web.Application(handlers=[
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
81 (r'/()', cyclone.web.StaticFileHandler, {
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
82 "path": "light9/collector/web",
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
83 "default_filename": "index.html"
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
84 }),
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
85 (r'/updates', Updates),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
86 (r'/attrs', Attrs),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
87 (r'/stats', StatsForCyclone),
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
88 ],
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
89 collector=c,
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
90 listeners=listeners),
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
91 interface='::')
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
92 log.info('serving http on %s, zmq on %s', networking.collector.port,
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
93 networking.collectorZmq.port)
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
94 if doLoadTest:
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
95 # in a subprocess since we don't want this client to be
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
96 # cooperating with the main event loop and only sending
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
97 # requests when there's free time
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
98 def afterWarmup():
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
99 log.info('running collector_loadtest')
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
100 d = utils.getProcessValue('bin/python',
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
101 ['bin/collector_loadtest.py'])
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
102
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
103 def done(*a):
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
104 log.info('loadtest done')
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
105 reactor.stop()
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
106
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
107 d.addCallback(done)
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
108
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
109 reactor.callLater(2, afterWarmup)
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
110
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
111
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
112 def main():
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
113 parser = optparse.OptionParser()
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
114 parser.add_option("-v",
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
115 "--verbose",
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
116 action="store_true",
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
117 help="logging.DEBUG")
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
118 parser.add_option("--logdmx", action="store_true", help="log all dmx sends")
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
119
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
120 parser.add_option("--loadtest",
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
121 action="store_true",
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
122 help="call myself with some synthetic load then exit")
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
123 (options, args) = parser.parse_args()
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
124 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
125 logging.getLogger('output').setLevel(logging.DEBUG)
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
126
1884
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
127 logging.getLogger('output.allDmx').setLevel(
5cde72dfdc22 change collector output code to use very specific types. Might fix bugs too.
Drew Perttula <drewp@bigasterisk.com>
parents: 1881
diff changeset
128 logging.DEBUG if options.logdmx else logging.INFO)
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
129 logging.getLogger('colormath').setLevel(logging.INFO)
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
130
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
131 graph = SyncedGraph(networking.rdfdb.url, "collector")
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
132
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
133 graph.initiallySynced.addCallback(lambda _: launch(graph, options.loadtest)
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
134 ).addErrback(lambda e: reactor.crash())
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
135 reactor.run()
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
136
1858
7772cc48e016 reformat all python
drewp@bigasterisk.com
parents: 1826
diff changeset
137
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
138 if __name__ == '__main__':
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
139 main()