Mercurial > code > home > repos > homeauto

annotate service/reasoning/reasoning.py @ 252:57f16890ab3a

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
evtiming in reasoning service Ignore-this: aa4aba730ca916bda2b8e99c743cac2a
author drewp@bigasterisk.com
date Fri, 12 Feb 2016 02:42:29 -0800
parents 254df9f881a6
children 32cc1eda8389
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
2 """
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
3 Graph consists of:
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
4 input/* (read at startup)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
5 webinput/* (new files are noticed in here)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
6 any number of remote graphs, specified in the other graph as objects of (:reasoning, :source, *), reread constantly
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
7
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
8 gather subgraphs from various services, run them through a rules
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
9 engine, and make http requests with the conclusions.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
10
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
11 E.g. 'when drew's phone is near the house, and someone is awake,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
12 unlock the door when the door's motion sensor is activated'
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
13
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
14 When do we gather? The services should be able to trigger us, perhaps
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
15 with PSHB, that their graph has changed.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
16 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
17
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
18
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
19 from twisted.internet import reactor, task
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
20 from twisted.internet.defer import inlineCallbacks, gatherResults
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
21 from twisted.python.filepath import FilePath
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
22 import time, traceback, sys, json, logging
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
23 from rdflib import Graph, ConjunctiveGraph
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
24 from rdflib import Namespace, URIRef, Literal, RDF
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
25 from rdflib.parser import StringInputSource
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
26
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
27 import cyclone.web, cyclone.websocket
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
28 from inference import infer
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
29 from rdflibtrig import addTrig
48
571e773c77c3 refactor graph ops
drewp@bigasterisk.com
parents: 47
diff changeset
30 from graphop import graphEqual
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
31 from docopt import docopt
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
32 from actions import Actions
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
33 from FuXi.Rete.RuleStore import N3RuleStore
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
34
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
35 sys.path.append("../../lib")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
36 from logsetup import log
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
37 log.setLevel(logging.WARN)
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
38 outlog = logging.getLogger('output')
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
39 outlog.setLevel(logging.WARN)
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
40
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
41 sys.path.append('../../../ffg/ffg')
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
42 import evtiming
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
43
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
44 ROOM = Namespace("http://projects.bigasterisk.com/room/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
45 DEV = Namespace("http://projects.bigasterisk.com/device/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
46
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
47
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
48 class InputGraph(object):
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
49 def __init__(self, inputDirs, onChange):
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
50 """
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
51 this has one Graph that's made of:
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
52 - all .n3 files from inputDirs (read at startup)
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
53 - all the remote graphs, specified in the file graphs
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
54
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
55 call updateFileData or updateRemoteData to reread those
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
56 graphs. getGraph to access the combined graph.
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
57
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
58 onChange(self) is called if the contents of the full graph
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
59 change (in an interesting way) during updateFileData or
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
60 updateRemoteData. Interesting means statements other than the
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
61 ones with the predicates on the boring list. onChange(self,
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
62 oneShot=True) means: don't store the result of this change
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
63 anywhere; it needs to be processed only once
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
64 """
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
65 self.inputDirs = inputDirs
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
66 self.onChange = onChange
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
67 self._fileGraph = Graph()
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
68 self._remoteGraph = None
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
69 self._combinedGraph = None
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
70 self._oneShotAdditionGraph = None
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
71
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
72 def updateFileData(self):
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
73 """
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
74 make sure we contain the correct data from the files in inputDirs
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
75 """
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
76 # this sample one is actually only needed for the output, but I don't
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
77 # think I want to have a separate graph for the output
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
78 # handling
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
79 log.debug("read file graphs")
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
80 for fp in FilePath("input").walk():
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
81 if fp.isdir():
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
82 continue
111
e4a4ee70a514 switch reasoning from restkit to async cyclone http client
drewp@bigasterisk.com
parents: 77
diff changeset
83 if fp.splitext()[1] != '.n3':
e4a4ee70a514 switch reasoning from restkit to async cyclone http client
drewp@bigasterisk.com
parents: 77
diff changeset
84 continue
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
85 log.debug("read %s", fp)
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
86 # todo: if this fails, leave the report in the graph
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
87 self._fileGraph.parse(fp.open(), format="n3")
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
88 self._combinedGraph = None
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
89
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
90 self.onChange(self)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
91
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
92 @inlineCallbacks
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
93 def updateRemoteData(self):
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
94 """
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
95 read all remote graphs (which are themselves enumerated within
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
96 the file data)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
97 """
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
98 t1 = time.time()
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
99 log.debug("read remote graphs")
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
100 g = ConjunctiveGraph()
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
101
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
102 @inlineCallbacks
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
103 def fetchOne(source):
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
104 try:
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
105 # this part could be parallelized
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
106 fetchTime = yield addTrig(g, source)
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
107 except Exception, e:
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
108 log.error(" can't add source %s: %s", source, e)
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
109 g.add((URIRef(source), ROOM['graphLoadError'], Literal(str(e))))
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
110 g.add((URIRef(source), RDF.type, ROOM['FailedGraphLoad']))
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
111 else:
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
112 g.add((URIRef(source), ROOM['graphLoadMs'],
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
113 Literal(round(fetchTime * 1000, 1))))
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
114
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
115 fetchDone = []
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
116 for source in self._fileGraph.objects(ROOM['reasoning'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
117 ROOM['source']):
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
118 fetchDone.append(fetchOne(source))
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
119 yield gatherResults(fetchDone, consumeErrors=True)
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
120 log.debug("loaded all in %.1f ms", 1000 * (time.time() - t1))
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
121
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
122 prevGraph = self._remoteGraph
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
123 self._remoteGraph = g
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
124 self._combinedGraph = None
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
125 if (prevGraph is None or
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
126 not graphEqual(g, prevGraph, ignorePredicates=[
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
127 ROOM['signalStrength'],
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
128 # perhaps anything with a number-datatype for its
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
129 # object should be filtered out, and you have to make
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
130 # an upstream quantization (e.g. 'temp high'/'temp
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
131 # low') if you want to do reasoning on the difference
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
132 URIRef("http://bigasterisk.com/map#lastSeenAgoSec"),
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
133 URIRef("http://bigasterisk.com/map#lastSeenAgo"),
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
134 ROOM['usingPower'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
135 ROOM['idleTimeMinutes'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
136 ROOM['idleTimeMs'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
137 ROOM['graphLoadMs'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
138 ROOM['localTimeToSecond'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
139 ROOM['history'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
140 ROOM['temperatureF'],
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
141 ROOM['connectedAgo'],
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
142 ])):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
143 log.debug(" remote graph changed")
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
144 self.onChange(self)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
145 else:
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
146 log.debug(" remote graph has no changes to trigger rules")
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
147
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
148 def addOneShot(self, g):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
149 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
150 add this graph to the total, call onChange, and then revert
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
151 the addition of this graph
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
152 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
153 self._oneShotAdditionGraph = g
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
154 self._combinedGraph = None
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
155 try:
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
156 self.onChange(self, oneShot=True, oneShotGraph=g)
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
157 finally:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
158 self._oneShotAdditionGraph = None
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
159 self._combinedGraph = None
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
160
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
161 def getGraph(self):
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
162 """rdflib Graph with the file+remote contents of the input graph"""
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
163 # this could be much faster with the combined readonly graph
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
164 # view from rdflib
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
165 if self._combinedGraph is None:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
166 self._combinedGraph = Graph()
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
167 if self._fileGraph:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
168 for s in self._fileGraph:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
169 self._combinedGraph.add(s)
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
170 if self._remoteGraph:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
171 for s in self._remoteGraph:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
172 self._combinedGraph.add(s)
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
173 if self._oneShotAdditionGraph:
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
174 for s in self._oneShotAdditionGraph:
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
175 self._combinedGraph.add(s)
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
176
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
177 return self._combinedGraph
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
178
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
179
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
180 class Reasoning(object):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
181 def __init__(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
182 self.prevGraph = None
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
183 self.lastPollTime = 0
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
184 self.lastError = ""
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
185
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
186 self.actions = Actions(sendToLiveClients)
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
187
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
188 self.rulesN3 = "(not read yet)"
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
189 self.inferred = Graph() # gets replaced in each graphChanged call
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
190
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
191 self.inputGraph = InputGraph([], self.graphChanged)
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
192 self.inputGraph.updateFileData()
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
193
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
194 @evtiming.serviceLevel.timed('readRules')
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
195 def readRules(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
196 self.rulesN3 = open('rules.n3').read() # for web display
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
197 self.ruleStore = N3RuleStore()
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
198 self.ruleGraph = Graph(self.ruleStore)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
199 self.ruleGraph.parse('rules.n3', format='n3') # for inference
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
200
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
201 @inlineCallbacks
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
202 def poll(self):
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
203 t1 = time.time()
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
204 try:
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
205 yield self.inputGraph.updateRemoteData()
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
206 self.lastPollTime = time.time()
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
207 except Exception, e:
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
208 log.error(traceback.format_exc())
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
209 self.lastError = str(e)
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
210 evtiming.serviceLevel.addData('poll', time.time() - t1)
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
211
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
212 def updateRules(self):
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
213 try:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
214 t1 = time.time()
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
215 self.readRules()
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
216 ruleParseTime = time.time() - t1
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
217 except ValueError:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
218 # this is so if you're just watching the inferred output,
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
219 # you'll see the error too
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
220 self.inferred = Graph()
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
221 self.inferred.add((ROOM['reasoner'], ROOM['ruleParseError'],
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
222 Literal(traceback.format_exc())))
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
223 raise
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
224 return [(ROOM['reasoner'], ROOM['ruleParseTime'],
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
225 Literal(ruleParseTime))]
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
226
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
227 evtiming.serviceLevel.timed('graphChanged')
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
228 def graphChanged(self, inputGraph, oneShot=False, oneShotGraph=None):
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
229 t1 = time.time()
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
230 oldInferred = self.inferred
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
231 try:
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
232 ruleStmts = self.updateRules()
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
233
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
234 g = inputGraph.getGraph()
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
235 self.inferred = self._makeInferred(g)
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
236 [self.inferred.add(s) for s in ruleStmts]
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
237
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
238 if oneShot:
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
239 # unclear where this should go, but the oneshot'd
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
240 # statements should be just as usable as inferred
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
241 # ones.
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
242 for s in oneShotGraph:
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
243 self.inferred.add(s)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
244
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
245 t2 = time.time()
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
246 self.actions.putResults(self.inputGraph.getGraph(), self.inferred)
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
247 putResultsTime = time.time() - t2
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
248 finally:
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
249 if oneShot:
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
250 self.inferred = oldInferred
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
251 log.info("graphChanged %.1f ms (putResults %.1f ms)" %
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
252 ((time.time() - t1) * 1000,
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
253 putResultsTime * 1000))
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
254
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
255 def _makeInferred(self, inputGraph):
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
256 t1 = time.time()
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
257 out = infer(inputGraph, self.ruleStore)
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
258 inferenceTime = time.time() - t1
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
259
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
260 out.add((ROOM['reasoner'], ROOM['inferenceTime'],
111
e4a4ee70a514 switch reasoning from restkit to async cyclone http client
drewp@bigasterisk.com
parents: 77
diff changeset
261 Literal(inferenceTime)))
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
262 return out
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
263
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
264
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
265
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
266 class Index(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
267 def get(self):
252
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
268 print evtiming.serviceLevel.serviceJsonReport()
57f16890ab3a evtiming in reasoning service
drewp@bigasterisk.com
parents: 251
diff changeset
269
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
270 # make sure GET / fails if our poll loop died
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
271 ago = time.time() - self.settings.reasoning.lastPollTime
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
272 if ago > 2:
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
273 self.set_status(500)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
274 self.finish("last poll was %s sec ago. last error: %s" %
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
275 (ago, self.settings.reasoning.lastError))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
276 return
180
47682350e6f2 reasoning index page to html
drewp@bigasterisk.com
parents: 179
diff changeset
277 self.set_header("Content-Type", "text/html")
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
278 self.write(open('index.html').read())
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
279
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
280 class ImmediateUpdate(cyclone.web.RequestHandler):
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
281 @inlineCallbacks
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
282 def put(self):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
283 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
284 request an immediate load of the remote graphs; the thing we
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
285 do in the background anyway. No payload.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
286
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
287 Using PUT because this is idempotent and retryable and
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
288 everything.
54
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
289
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
290 todo: this should do the right thing when many requests come
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
291 in very quickly
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
292 """
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
293 print self.request.headers
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
294 log.info("immediateUpdate from %s",
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
295 self.request.headers.get('User-Agent', '?'))
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
296 yield r.poll()
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
297 self.set_status(202)
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
298
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
299 def parseRdf(text, contentType):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
300 g = Graph()
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
301 g.parse(StringInputSource(text), format={
250
c1287ab87add fix oneshot. more time reportin
drewp@bigasterisk.com
parents: 249
diff changeset
302 'text/n3': 'n3',
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
303 }[contentType])
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
304 return g
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
305
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
306 class OneShot(cyclone.web.RequestHandler):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
307 def post(self):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
308 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
309 payload is an rdf graph. The statements are momentarily added
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
310 to the input graph for exactly one update.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
311
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
312 todo: how do we go from a transition like doorclosed-to-open
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
313 to a oneshot event? the upstream shouldn't have to do it. Do
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
314 we make those oneshot events here? for every object change?
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
315 there are probably special cases regarding startup time when
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
316 everything appears to be a 'change'.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
317 """
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
318 try:
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
319 g = parseRdf(self.request.body, self.request.headers['content-type'])
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
320 for s in g:
251
254df9f881a6 start sending oneshot events from some devices
drewp@bigasterisk.com
parents: 250
diff changeset
321 log.debug("oneshot stmt %r", s)
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
322 if not len(g):
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
323 log.warn("incoming oneshot graph had no statements: %r", self.request.body)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
324 return
250
c1287ab87add fix oneshot. more time reportin
drewp@bigasterisk.com
parents: 249
diff changeset
325 t1 = time.time()
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
326 self.settings.reasoning.inputGraph.addOneShot(g)
250
c1287ab87add fix oneshot. more time reportin
drewp@bigasterisk.com
parents: 249
diff changeset
327 self.set_header('x-graph-ms', str(1000 * (time.time() - t1)))
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
328 except Exception as e:
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
329 log.error(e)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
330 raise
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
331
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
332 # for reuse
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
333 class GraphResource(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
334 def get(self, which):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
335 self.set_header("Content-Type", "application/json")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
336 r = self.settings.reasoning
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
337 g = {'lastInput': r.inputGraph.getGraph(),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
338 'lastOutput': r.inferred,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
339 }[which]
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
340 self.write(self.jsonRdf(g))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
341
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
342 def jsonRdf(self, g):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
343 return json.dumps(sorted(list(g)))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
344
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
345 class NtGraphs(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
346 """same as what gets posted above"""
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
347 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
348 r = self.settings.reasoning
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
349 inputGraphNt = r.inputGraph.getGraph().serialize(format="nt")
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
350 inferredNt = r.inferred.serialize(format="nt")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
351 self.set_header("Content-Type", "application/json")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
352 self.write(json.dumps({"input": inputGraphNt,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
353 "inferred": inferredNt}))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
354
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
355 class Rules(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
356 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
357 self.set_header("Content-Type", "text/plain")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
358 self.write(self.settings.reasoning.rulesN3)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
359
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
360 class Status(cyclone.web.RequestHandler):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
361 def get(self):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
362 self.set_header("Content-Type", "text/plain")
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
363 g = self.settings.reasoning.inputGraph.getGraph()
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
364 msg = ""
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
365 for badSource in g.subjects(RDF.type, ROOM['FailedGraphLoad']):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
366 msg += "GET %s failed (%s). " % (
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
367 badSource, g.value(badSource, ROOM['graphLoadError']))
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
368 if not msg:
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
369 self.finish("all inputs ok")
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
370 return
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
371 self.set_status(500)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
372 self.finish(msg)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
373
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
374 class Static(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
375 def get(self, p):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
376 self.write(open(p).read())
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
377
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
378 liveClients = set()
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
379 def sendToLiveClients(d=None, asJson=None):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
380 j = asJson or json.dumps(d)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
381 for c in liveClients:
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
382 c.sendMessage(j)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
383
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
384 class Events(cyclone.websocket.WebSocketHandler):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
385
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
386 def connectionMade(self, *args, **kwargs):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
387 log.info("websocket opened")
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
388 liveClients.add(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
389
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
390 def connectionLost(self, reason):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
391 log.info("websocket closed")
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
392 liveClients.remove(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
393
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
394 def messageReceived(self, message):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
395 log.info("got message %s" % message)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
396
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
397 class Application(cyclone.web.Application):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
398 def __init__(self, reasoning):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
399 handlers = [
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
400 (r"/", Index),
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
401 (r"/immediateUpdate", ImmediateUpdate),
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
402 (r"/oneShot", OneShot),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
403 (r'/(jquery.min.js)', Static),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
404 (r'/(lastInput|lastOutput)Graph', GraphResource),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
405 (r'/ntGraphs', NtGraphs),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
406 (r'/rules', Rules),
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
407 (r'/status', Status),
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
408 (r'/events', Events),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
409 ]
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
410 cyclone.web.Application.__init__(self, handlers, reasoning=reasoning)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
411
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
412 if __name__ == '__main__':
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
413
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
414 arg = docopt("""
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
415 Usage: reasoning.py [options]
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
416
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
417 -v Verbose (and slow updates)
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
418 """)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
419
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
420 r = Reasoning()
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
421 if arg['-v']:
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
422 from colorlog import ColoredFormatter
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
423 log.handlers[0].setFormatter(ColoredFormatter("%(log_color)s%(levelname)-8s%(reset)s %(white)s%(message)s",
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
424 datefmt=None,
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
425 reset=True,
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
426 log_colors={
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
427 'DEBUG': 'cyan',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
428 'INFO': 'green',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
429 'WARNING': 'yellow',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
430 'ERROR': 'red',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
431 'CRITICAL': 'red,bg_white',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
432 },
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
433 secondary_log_colors={},
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
434 style='%'
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
435 ))
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
436
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
437 import twisted.python.log
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
438 twisted.python.log.startLogging(sys.stdout)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
439 log.setLevel(logging.DEBUG)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
440 outlog.setLevel(logging.DEBUG)
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
441
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
442 task.LoopingCall(r.poll).start(1.0 if not arg['-v'] else 10)
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
443 reactor.listenTCP(9071, Application(r), interface='::')
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
444 reactor.run()