Mercurial > code > home > repos > light9

annotate bin/collector @ 1826:d6ec468112cb

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
not sure highwatermark setting did what i wanted Ignore-this: 3a3b2835ea3a30d7fb0e2007a3b06bb2
author drewp@bigasterisk.com
date Sat, 09 Jun 2018 02:01:36 +0000
parents 0da5fcfe4ea5
children 7772cc48e016
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
1 #!bin/python
1590
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
2 """
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
3 Collector receives device attrs from multiple senders, combines
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
4 them, and sends output attrs to hardware. The combining part has
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
5 custom code for some attributes.
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
6
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
7 Input can be over http or zmq.
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
8 """
8268224c1b23 WIP support for choice/enum attrs
drewp@bigasterisk.com
parents: 1564
diff changeset
9
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
10 from __future__ import division
1697
5c04a54df635 fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents: 1692
diff changeset
11 from run_local import log
5c04a54df635 fix startup (this might be breaking mypy)
Drew Perttula <drewp@bigasterisk.com>
parents: 1692
diff changeset
12
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
13 from rdflib import URIRef, Literal
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
14 from twisted.internet import reactor, utils
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
15 from txzmq import ZmqEndpoint, ZmqFactory, ZmqPullConnection
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
16 import json
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
17 import logging
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
18 import optparse
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
19 import time
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
20 import traceback
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
21 import cyclone.web, cyclone.websocket
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
22 from greplin import scales
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
23
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
24 from lib.cycloneerr import PrettyErrorHandler
1605
64c1bcff604e dummyoutput for collector
drewp@bigasterisk.com
parents: 1596
diff changeset
25 from light9.collector.output import EnttecDmx, Udmx, DummyOutput
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
26 from light9.collector.collector import Collector
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
27 from light9.namespaces import L9
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
28 from light9 import networking
1692
6fa4288da8a6 rdfdb is its own package now
drewp@bigasterisk.com
parents: 1630
diff changeset
29 from rdfdb.syncedgraph import SyncedGraph
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
30 from light9.greplin_cyclone import StatsForCyclone
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
31
1491
c0742e710eeb refactor collector json parsing
drewp@bigasterisk.com
parents: 1490
diff changeset
32 def parseJsonMessage(msg):
1492
ce97f298bfb8 restore zmq transport to collector
drewp@bigasterisk.com
parents: 1491
diff changeset
33 body = json.loads(msg)
1491
c0742e710eeb refactor collector json parsing
drewp@bigasterisk.com
parents: 1490
diff changeset
34 settings = []
c0742e710eeb refactor collector json parsing
drewp@bigasterisk.com
parents: 1490
diff changeset
35 for device, attr, value in body['settings']:
1595
013cbd7a0f08 choice-type attrs in live
Drew Perttula <drewp@bigasterisk.com>
parents: 1590
diff changeset
36 if isinstance(value, basestring) and value.startswith('http'):
013cbd7a0f08 choice-type attrs in live
Drew Perttula <drewp@bigasterisk.com>
parents: 1590
diff changeset
37 value = URIRef(value)
013cbd7a0f08 choice-type attrs in live
Drew Perttula <drewp@bigasterisk.com>
parents: 1590
diff changeset
38 else:
013cbd7a0f08 choice-type attrs in live
Drew Perttula <drewp@bigasterisk.com>
parents: 1590
diff changeset
39 value = Literal(value)
013cbd7a0f08 choice-type attrs in live
Drew Perttula <drewp@bigasterisk.com>
parents: 1590
diff changeset
40 settings.append((URIRef(device), URIRef(attr), value))
1491
c0742e710eeb refactor collector json parsing
drewp@bigasterisk.com
parents: 1490
diff changeset
41 return body['client'], body['clientSession'], settings, body['sendTime']
c0742e710eeb refactor collector json parsing
drewp@bigasterisk.com
parents: 1490
diff changeset
42
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
43 def startZmq(port, collector):
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
44 stats = scales.collection('/zmqServer',
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
45 scales.PmfStat('setAttr'))
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
46
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
47 zf = ZmqFactory()
1492
ce97f298bfb8 restore zmq transport to collector
drewp@bigasterisk.com
parents: 1491
diff changeset
48 addr = 'tcp://*:%s' % port
ce97f298bfb8 restore zmq transport to collector
drewp@bigasterisk.com
parents: 1491
diff changeset
49 log.info('creating zmq endpoint at %r', addr)
ce97f298bfb8 restore zmq transport to collector
drewp@bigasterisk.com
parents: 1491
diff changeset
50 e = ZmqEndpoint('bind', addr)
1809
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
51 class Pull(ZmqPullConnection):
1826
d6ec468112cb not sure highwatermark setting did what i wanted
drewp@bigasterisk.com
parents: 1822
diff changeset
52 #highWaterMark = 3
1809
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
53 def onPull(self, message):
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
54 with stats.setAttr.time():
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
55 # todo: new compressed protocol where you send all URIs up
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
56 # front and then use small ints to refer to devices and
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
57 # attributes in subsequent requests.
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
58 client, clientSession, settings, sendTime = parseJsonMessage(message[0])
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
59 collector.setAttrs(client, clientSession, settings, sendTime)
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
60
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
61 s = Pull(zf, e)
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
62
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
63
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
64 class WebListeners(object):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
65 def __init__(self):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
66 self.clients = []
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
67 self.pendingMessageForDev = {} # dev: (attrs, outputmap)
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
68 self.lastFlush = 0
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
69
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
70 def addClient(self, client):
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
71 self.clients.append([client, {}]) # seen = {dev: attrs}
1806
5668ad92a98e bug in collector log
Drew Perttula <drewp@bigasterisk.com>
parents: 1801
diff changeset
72 log.info('added client %s %s', len(self.clients), client)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
73
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
74 def delClient(self, client):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
75 self.clients = [[c, t] for c, t in self.clients if c != client]
1543
c8cffe82b537 collector gui updates
Drew Perttula <drewp@bigasterisk.com>
parents: 1541
diff changeset
76 log.info('delClient %s, %s left', client, len(self.clients))
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
77
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
78 def outputAttrsSet(self, dev, attrs, outputMap):
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
79 """called often- don't be slow"""
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
80
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
81 self.pendingMessageForDev[dev] = (attrs, outputMap)
1809
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
82 try:
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
83 self._flush()
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
84 except Exception:
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
85 traceback.print_exc()
778c67ab70c9 set zmq highWaterMark to dump stale messages, especially those sent when collector isn't running
drewp@bigasterisk.com
parents: 1806
diff changeset
86 raise
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
87
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
88 def _flush(self):
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
89 now = time.time()
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
90 if now < self.lastFlush + .05 or not self.clients:
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
91 return
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
92 self.lastFlush = now
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
93
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
94 while self.pendingMessageForDev:
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
95 dev, (attrs, outputMap) = self.pendingMessageForDev.popitem()
1564
f2e6d96d02de minor comments and refactors, I think, except the removal of 'row update' logging made a big perf difference
Drew Perttula <drewp@bigasterisk.com>
parents: 1543
diff changeset
96
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
97 msg = None # lazy, since makeMsg is slow
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
98
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
99 # this omits repeats, but can still send many
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
100 # messages/sec. Not sure if piling up messages for the browser
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
101 # could lead to slowdowns in the real dmx output.
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
102 for client, seen in self.clients:
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
103 if seen.get(dev) == attrs:
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
104 continue
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
105 if msg is None:
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
106 msg = self.makeMsg(dev, attrs, outputMap)
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
107
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
108 seen[dev] = attrs
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
109 client.sendMessage(msg)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
110
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
111 def makeMsg(self, dev, attrs, outputMap):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
112 attrRows = []
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
113 for attr, val in attrs.items():
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
114 output, index = outputMap[(dev, attr)]
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
115 attrRows.append({'attr': attr.rsplit('/')[-1],
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
116 'val': val,
1543
c8cffe82b537 collector gui updates
Drew Perttula <drewp@bigasterisk.com>
parents: 1541
diff changeset
117 'chan': (output.shortId(), index + 1)})
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
118 attrRows.sort(key=lambda r: r['chan'])
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
119 for row in attrRows:
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
120 row['chan'] = '%s %s' % (row['chan'][0], row['chan'][1])
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
121
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
122 msg = json.dumps({'outputAttrsSet': {'dev': dev, 'attrs': attrRows}}, sort_keys=True)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
123 return msg
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
124
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
125 class Updates(cyclone.websocket.WebSocketHandler):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
126
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
127 def connectionMade(self, *args, **kwargs):
1543
c8cffe82b537 collector gui updates
Drew Perttula <drewp@bigasterisk.com>
parents: 1541
diff changeset
128 log.info('socket connect %s', self)
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
129 self.settings.listeners.addClient(self)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
130
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
131 def connectionLost(self, reason):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
132 self.settings.listeners.delClient(self)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
133
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
134 def messageReceived(self, message):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
135 json.loads(message)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
136
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
137 stats = scales.collection('/webServer', scales.PmfStat('setAttr'))
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
138
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
139 class Attrs(PrettyErrorHandler, cyclone.web.RequestHandler):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
140 def put(self):
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
141 with stats.setAttr.time():
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
142 client, clientSession, settings, sendTime = parseJsonMessage(self.request.body)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
143 self.settings.collector.setAttrs(client, clientSession, settings, sendTime)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
144 self.set_status(202)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
145
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
146
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
147
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
148 def launch(graph, doLoadTest=False):
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
149 try:
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
150 # todo: drive outputs with config files
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
151 outputs = [
1822
0da5fcfe4ea5 collector debug logging
drewp@bigasterisk.com
parents: 1809
diff changeset
152 # EnttecDmx(L9['output/dmxA/'], '/dev/dmx3', 80),
0da5fcfe4ea5 collector debug logging
drewp@bigasterisk.com
parents: 1809
diff changeset
153 Udmx(L9['output/dmxA/'], bus=5, numChannels=80),
0da5fcfe4ea5 collector debug logging
drewp@bigasterisk.com
parents: 1809
diff changeset
154 #DummyOutput(L9['output/dmxA/'], 80),
0da5fcfe4ea5 collector debug logging
drewp@bigasterisk.com
parents: 1809
diff changeset
155 Udmx(L9['output/dmxB/'], bus=7, numChannels=500),
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
156 ]
1799
0bb7b9df12e5 collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents: 1798
diff changeset
157 except Exception:
0bb7b9df12e5 collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents: 1798
diff changeset
158 log.error("setting up outputs:")
1530
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
159 traceback.print_exc()
a5a44077c54c docs and error msgs
drewp@bigasterisk.com
parents: 1493
diff changeset
160 raise
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
161 listeners = WebListeners()
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
162 c = Collector(graph, outputs, listeners)
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
163
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
164 startZmq(networking.collectorZmq.port, c)
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
165
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
166 reactor.listenTCP(networking.collector.port,
1541
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
167 cyclone.web.Application(handlers=[
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
168 (r'/()', cyclone.web.StaticFileHandler,
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
169 {"path" : "light9/collector/web", "default_filename" : "index.html"}),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
170 (r'/updates', Updates),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
171 (r'/attrs', Attrs),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
172 (r'/stats', StatsForCyclone),
c1bf296b0a74 collector uses cyclone and gets a web ui showing output attrs
Drew Perttula <drewp@bigasterisk.com>
parents: 1530
diff changeset
173 ], collector=c, listeners=listeners),
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
174 interface='::')
1289
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
175 log.info('serving http on %s, zmq on %s', networking.collector.port,
5a4e74f1e36a Fixed client session clearing bugs.
Drew Perttula <drewp@bigasterisk.com>
parents: 1288
diff changeset
176 networking.collectorZmq.port)
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
177 if doLoadTest:
1493
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
178 # in a subprocess since we don't want this client to be
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
179 # cooperating with the main event loop and only sending
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
180 # requests when there's free time
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
181 def afterWarmup():
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
182 log.info('running collector_loadtest')
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
183 d = utils.getProcessValue('bin/python', ['bin/collector_loadtest.py'])
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
184 def done(*a):
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
185 log.info('loadtest done')
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
186 reactor.stop()
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
187 d.addCallback(done)
4294ed82ee16 move collector_loadtest and arrange for collector to be able to run the test itself, but that last part isn't working. you can run collector and collector_loadtest in two shells, though
drewp@bigasterisk.com
parents: 1492
diff changeset
188 reactor.callLater(2, afterWarmup)
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
189
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
190 def main():
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
191 parser = optparse.OptionParser()
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
192 parser.add_option("-v", "--verbose", action="store_true",
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
193 help="logging.DEBUG")
1490
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
194 parser.add_option("--loadtest", action="store_true",
649d482737e0 start of a collector --loadtest mode
Drew Perttula <drewp@bigasterisk.com>
parents: 1489
diff changeset
195 help="call myself with some synthetic load then exit")
1372
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
196 (options, args) = parser.parse_args()
f427801da9f6 collector properly merges repeated attr settings in the same message
Drew Perttula <drewp@bigasterisk.com>
parents: 1307
diff changeset
197 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
198
1596
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
199 logging.getLogger('colormath').setLevel(logging.INFO)
7d5d6e7bc526 collector web view speedups- don't json encode all devs all the time
Drew Perttula <drewp@bigasterisk.com>
parents: 1595
diff changeset
200
1307
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
201 graph = SyncedGraph(networking.rdfdb.url, "collector")
8863b4485fd4 collector uses rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents: 1289
diff changeset
202
1799
0bb7b9df12e5 collector warnings and errors. the reactor.crash isn't working.
drewp@bigasterisk.com
parents: 1798
diff changeset
203 graph.initiallySynced.addCallback(lambda _: launch(graph, options.loadtest)).addErrback(lambda e: reactor.crash())
1288
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
204 reactor.run()
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
205
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
206 if __name__ == '__main__':
5e76c8fd8a03 rewrite dmx outputter to a new service
Drew Perttula <drewp@bigasterisk.com>
parents:
diff changeset
207 main()