Mercurial > code > home > repos > homeauto

annotate service/reasoning/reasoning.py @ 760:3c18b4b3b72c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
more py3 fixes Ignore-this: f212b4a5edf8e599e9efd70bc65e7651
author drewp@bigasterisk.com
date Fri, 14 Feb 2020 00:33:31 -0800
parents f3f667769aef
children c8562ace4917
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
2 """
46
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
3 Graph consists of:
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
4 input/* (read at startup)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
5 webinput/* (new files are noticed in here)
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
6 any number of remote graphs, specified in the other graph as objects of (:reasoning, :source, *), reread constantly
f5623d9b07fd rewriting reasoning to use graphs for config
drewp@bigasterisk.com
parents: 45
diff changeset
7
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
8 gather subgraphs from various services, run them through a rules
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
9 engine, and make http requests with the conclusions.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
10
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
11 E.g. 'when drew's phone is near the house, and someone is awake,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
12 unlock the door when the door's motion sensor is activated'
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
13
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
14 When do we gather? The services should be able to trigger us, perhaps
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
15 with PSHB, that their graph has changed.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
16 """
303
66fe7a93753d reasoning uses sse_collector
drewp@bigasterisk.com
parents: 288
diff changeset
17 from crochet import no_setup
66fe7a93753d reasoning uses sse_collector
drewp@bigasterisk.com
parents: 288
diff changeset
18 no_setup()
66fe7a93753d reasoning uses sse_collector
drewp@bigasterisk.com
parents: 288
diff changeset
19
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
20 import json, time, traceback, sys
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
21 from logging import getLogger, DEBUG, WARN
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
22 from typing import Dict, Optional, Set, Tuple
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
23
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
24 from colorlog import ColoredFormatter
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
25 from docopt import docopt
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
26 from rdflib import Namespace, Literal, RDF, Graph, URIRef
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
27 from rdflib.term import Node
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
28 from twisted.internet import reactor, defer
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
29 import cyclone.web, cyclone.websocket
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
30 from FuXi.Rete.RuleStore import N3RuleStore
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
31
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
32 from greplin import scales
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
33 from greplin.scales.cyclonehandler import StatsHandler
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
34
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 283
diff changeset
35 from inference import infer, readRules
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
36 from actions import Actions, PutOutputsTable
275
d3733587e749 refactor inputgraph
drewp@bigasterisk.com
parents: 265
diff changeset
37 from inputgraph import InputGraph
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 283
diff changeset
38 from escapeoutputstatements import unquoteOutputStatements
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
39
600
a34dee00a989 comments and logging
drewp@bigasterisk.com
parents: 571
diff changeset
40 from standardservice.logsetup import log, verboseLogging
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
41 from patchablegraph import PatchableGraph, CycloneGraphHandler, CycloneGraphEventsHandler
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
42
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
43
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
44 ROOM = Namespace("http://projects.bigasterisk.com/room/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
45 DEV = Namespace("http://projects.bigasterisk.com/device/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
46
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
47 NS = {'': ROOM, 'dev': DEV}
283
0b0fb67b0b3a fix input/output statement ambiguity problem by allowing quoted output statements
drewp@bigasterisk.com
parents: 282
diff changeset
48
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
49 STATS = scales.collection('/web',
392
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
50 scales.PmfStat('graphChanged'),
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
51 scales.PmfStat('updateRules'),
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
52 )
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
53
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
54 def ntStatement(stmt: Tuple[Node, Node, Node]):
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
55 def compact(u):
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
56 if isinstance(u, URIRef) and u.startswith(ROOM):
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
57 return 'room:' + u[len(ROOM):]
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
58 return u.n3()
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
59 return '%s %s %s .' % (compact(stmt[0]), compact(stmt[1]), compact(stmt[2]))
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
60
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
61 class Reasoning(object):
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
62 ruleStore: N3RuleStore
608
62171aa2bc4e mock output mode
drewp@bigasterisk.com
parents: 600
diff changeset
63 def __init__(self, mockOutput=False):
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
64 self.prevGraph = None
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
65
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
66 self.rulesN3 = "(not read yet)"
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
67 self.inferred = Graph() # gets replaced in each graphChanged call
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
68 self.outputGraph = PatchableGraph() # copy of inferred, for now
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
69
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
70 self.inputGraph = InputGraph([], self.graphChanged)
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
71 self.actions = Actions(self.inputGraph, sendToLiveClients, mockOutput=mockOutput)
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
72 self.inputGraph.updateFileData()
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
73
392
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
74 @STATS.updateRules.time()
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
75 def updateRules(self):
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 283
diff changeset
76 rulesPath = 'rules.n3'
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
77 try:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
78 t1 = time.time()
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
79 self.rulesN3, self.ruleStore = readRules(
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 283
diff changeset
80 rulesPath, outputPatterns=[
286
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
81 # Incomplete. See escapeoutputstatements.py for
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
82 # explanation.
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
83 (None, ROOM['brightness'], None),
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
84 (None, ROOM['playState'], None),
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
85 (None, ROOM['powerState'], None),
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
86 (None, ROOM['state'], None),
da0b3a1394a3 more output statement patterns
drewp@bigasterisk.com
parents: 285
diff changeset
87 ])
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
88 ruleParseTime = time.time() - t1
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
89 except ValueError:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
90 # this is so if you're just watching the inferred output,
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
91 # you'll see the error too
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
92 self.inferred = Graph()
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
93 self.inferred.add((ROOM['reasoner'], ROOM['ruleParseError'],
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
94 Literal(traceback.format_exc())))
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
95 self.copyOutput()
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
96 raise
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
97 return [(ROOM['reasoner'], ROOM['ruleParseTime'],
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
98 Literal(ruleParseTime))], ruleParseTime
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
99
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
100 @STATS.graphChanged.time()
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
101 def graphChanged(self, inputGraph: InputGraph, oneShot=False, oneShotGraph: Graph = None):
285
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
102 """
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
103 If we're getting called for a oneShot event, the oneShotGraph
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
104 statements are already in inputGraph.getGraph().
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
105 """
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
106 log.info("----------------------")
392
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
107 log.info("graphChanged (oneShot=%s %s stmts):",
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
108 oneShot, len(oneShotGraph) if oneShotGraph is not None else 0)
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
109 if oneShotGraph:
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
110 for stmt in oneShotGraph:
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
111 log.info(" oneshot -> %s", ntStatement(stmt))
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
112 t1 = time.time()
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
113 oldInferred = self.inferred
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
114 try:
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
115 ruleStatStmts, ruleParseSec = self.updateRules()
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
116
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
117 self.inferred, inferSec = self._makeInferred(inputGraph.getGraph())
283
0b0fb67b0b3a fix input/output statement ambiguity problem by allowing quoted output statements
drewp@bigasterisk.com
parents: 282
diff changeset
118
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 283
diff changeset
119 self.inferred += unquoteOutputStatements(self.inferred)
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
120
285
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
121 self.inferred += ruleStatStmts
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
122
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
123 if oneShot:
285
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
124 # It's possible a oneShotGraph statement didn't
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
125 # trigger a rule to do something, but was itself the
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
126 # output statement. Probably we could just mix in the
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
127 # whole inputGraph here and not special-case the
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
128 # oneShotGraph.
c7476e2387dc minor cleanups
drewp@bigasterisk.com
parents: 284
diff changeset
129 self.inferred += oneShotGraph
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
130
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
131 t3 = time.time()
755
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot
drewp@bigasterisk.com
parents: 722
diff changeset
132 self.actions.putResults(self.inferred)
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
133 putResultsTime = time.time() - t3
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
134 finally:
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
135 if oneShot:
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
136 self.inferred = oldInferred
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
137 log.info("graphChanged took %.1f ms (rule parse %.1f ms, infer %.1f ms, putResults %.1f ms)" %
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
138 ((time.time() - t1) * 1000,
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
139 ruleParseSec * 1000,
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
140 inferSec * 1000,
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
141 putResultsTime * 1000))
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
142 if not oneShot:
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
143 self.copyOutput()
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
144
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
145 def copyOutput(self):
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
146 self.outputGraph.setToGraph((s,p,o,ROOM['inferred']) for s,p,o in self.inferred)
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
147
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
148 def _makeInferred(self, inputGraph: InputGraph):
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
149 t1 = time.time()
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
150
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
151 out = infer(inputGraph, self.ruleStore)
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
152 for p, n in NS.items():
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
153 out.bind(p, n, override=True)
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
154
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
155 inferenceTime = time.time() - t1
49
0aeb8d6ea124 cleanup
drewp@bigasterisk.com
parents: 48
diff changeset
156 out.add((ROOM['reasoner'], ROOM['inferenceTime'],
111
e4a4ee70a514 switch reasoning from restkit to async cyclone http client
drewp@bigasterisk.com
parents: 77
diff changeset
157 Literal(inferenceTime)))
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
158 return out, inferenceTime
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
159
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
160
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
161
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
162 class Index(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
163 def get(self):
180
47682350e6f2 reasoning index page to html
drewp@bigasterisk.com
parents: 179
diff changeset
164 self.set_header("Content-Type", "text/html")
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
165 self.write(open('index.html').read())
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
166
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
167 class ImmediateUpdate(cyclone.web.RequestHandler):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
168 def put(self):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
169 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
170 request an immediate load of the remote graphs; the thing we
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
171 do in the background anyway. No payload.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
172
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
173 Using PUT because this is idempotent and retryable and
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
174 everything.
54
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
175
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
176 todo: this should do the right thing when many requests come
42411726a7ca screen out more values that change on every update
drewp@bigasterisk.com
parents: 49
diff changeset
177 in very quickly
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
178 """
303
66fe7a93753d reasoning uses sse_collector
drewp@bigasterisk.com
parents: 288
diff changeset
179 log.warn("immediateUpdate from %s %s - ignored",
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
180 self.request.headers.get('User-Agent', '?'),
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
181 self.request.headers['Host'])
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
182 self.set_status(202)
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
183
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
184 class OneShot(cyclone.web.RequestHandler):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
185 def post(self):
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
186 """
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
187 payload is an rdf graph. The statements are momentarily added
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
188 to the input graph for exactly one update.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
189
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
190 todo: how do we go from a transition like doorclosed-to-open
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
191 to a oneshot event? the upstream shouldn't have to do it. Do
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
192 we make those oneshot events here? for every object change?
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
193 there are probably special cases regarding startup time when
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
194 everything appears to be a 'change'.
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
195 """
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
196 try:
392
79d041273e26 mqtt has two devices now. various older cleanups.
drewp@bigasterisk.com
parents: 351
diff changeset
197 log.info('POST to oneShot, headers=%s', self.request.headers)
760
3c18b4b3b72c more py3 fixes
drewp@bigasterisk.com
parents: 756
diff changeset
198 ct = self.request.headers.get(
3c18b4b3b72c more py3 fixes
drewp@bigasterisk.com
parents: 756
diff changeset
199 'Content-Type',
3c18b4b3b72c more py3 fixes
drewp@bigasterisk.com
parents: 756
diff changeset
200 self.request.headers.get('content-type', ''))
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
201 dt = self.settings.reasoning.inputGraph.addOneShotFromString(
760
3c18b4b3b72c more py3 fixes
drewp@bigasterisk.com
parents: 756
diff changeset
202 self.request.body, ct)
281
9728288c7f2f refactor
drewp@bigasterisk.com
parents: 280
diff changeset
203 self.set_header('x-graph-ms', str(1000 * dt))
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
204 except Exception as e:
315
8db492695a51 log oneshot tracebacks
drewp@bigasterisk.com
parents: 314
diff changeset
205 traceback.print_exc()
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
206 log.error(e)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
207 raise
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
208
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
209 # for reuse
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
210 class GraphResource(cyclone.web.RequestHandler):
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
211 def get(self, which: str):
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
212 self.set_header("Content-Type", "application/json")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
213 r = self.settings.reasoning
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
214 g = {'lastInput': r.inputGraph.getGraph(),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
215 'lastOutput': r.inferred,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
216 }[which]
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
217 self.write(self.jsonRdf(g))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
218
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
219 def jsonRdf(self, g):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
220 return json.dumps(sorted(list(g)))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
221
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
222 class NtGraphs(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
223 """same as what gets posted above"""
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
224 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
225 r = self.settings.reasoning
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
226 inputGraphNt = r.inputGraph.getGraph().serialize(format="nt")
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
227 inferredNt = r.inferred.serialize(format="nt")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
228 self.set_header("Content-Type", "application/json")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
229 self.write(json.dumps({"input": inputGraphNt,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
230 "inferred": inferredNt}))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
231
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
232 class Rules(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
233 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
234 self.set_header("Content-Type", "text/plain")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
235 self.write(self.settings.reasoning.rulesN3)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
236
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
237 class Status(cyclone.web.RequestHandler):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
238 def get(self):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
239 self.set_header("Content-Type", "text/plain")
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
240 g = self.settings.reasoning.inputGraph.getGraph()
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
241 msg = ""
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
242 for badSource in g.subjects(RDF.type, ROOM['FailedGraphLoad']):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
243 msg += "GET %s failed (%s). " % (
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
244 badSource, g.value(badSource, ROOM['graphLoadError']))
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
245 if not msg:
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
246 self.finish("all inputs ok")
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
247 return
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
248 self.set_status(500)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
249 self.finish(msg)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
250
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
251 class Static(cyclone.web.RequestHandler):
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
252 def get(self, p: str):
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
253 self.write(open(p).read())
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
254
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
255 liveClients: Set[cyclone.websocket.WebSocketHandler] = set()
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
256 def sendToLiveClients(d: Dict[str, object]=None, asJson: Optional[str]=None):
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
257 j = asJson or json.dumps(d)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
258 for c in liveClients:
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
259 c.sendMessage(j)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
260
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
261 class Events(cyclone.websocket.WebSocketHandler):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
262
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
263 def connectionMade(self, *args, **kwargs):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
264 log.info("websocket opened")
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
265 liveClients.add(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
266
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
267 def connectionLost(self, reason):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
268 log.info("websocket closed")
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
269 liveClients.remove(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
270
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
271 def messageReceived(self, message):
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
272 log.info("got message %s" % message)
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
273
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
274 class Application(cyclone.web.Application):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
275 def __init__(self, reasoning):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
276 handlers = [
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
277 (r"/", Index),
47
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
278 (r"/immediateUpdate", ImmediateUpdate),
0448fbd96a31 scan more input files. oneshot and immediate update features.
drewp@bigasterisk.com
parents: 46
diff changeset
279 (r"/oneShot", OneShot),
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
280 (r'/putOutputs', PutOutputsTable),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
281 (r'/(jquery.min.js)', Static),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
282 (r'/(lastInput|lastOutput)Graph', GraphResource),
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
283
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
284 (r"/graph/reasoning", CycloneGraphHandler,
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
285 {'masterGraph': reasoning.outputGraph}),
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
286 (r"/graph/reasoning/events", CycloneGraphEventsHandler,
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
287 {'masterGraph': reasoning.outputGraph}),
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
288
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
289 (r'/ntGraphs', NtGraphs),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
290 (r'/rules', Rules),
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
291 (r'/status', Status),
64
e573af8c2428 reasoning has a websocket server that broadcasts some events
drewp@bigasterisk.com
parents: 54
diff changeset
292 (r'/events', Events),
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 286
diff changeset
293 (r'/stats/(.*)', StatsHandler, {'serverName': 'reasoning'}),
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
294 ]
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
295 cyclone.web.Application.__init__(self, handlers, reasoning=reasoning)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
296
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
297 def configLogging(arg):
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
298 log.setLevel(WARN)
609
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now
drewp@bigasterisk.com
parents: 608
diff changeset
299
600
a34dee00a989 comments and logging
drewp@bigasterisk.com
parents: 571
diff changeset
300 if arg['-i'] or arg['-r'] or arg['-o'] or arg['-v']:
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
301 log.handlers[0].setFormatter(ColoredFormatter(
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
302 "%(log_color)s%(levelname)-8s %(name)-6s %(filename)-12s:%(lineno)-3s %(funcName)-20s%(reset)s %(white)s%(message)s",
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
303 datefmt=None,
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
304 reset=True,
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
305 log_colors={
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
306 'DEBUG': 'cyan',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
307 'INFO': 'green',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
308 'WARNING': 'yellow',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
309 'ERROR': 'red',
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
310 'CRITICAL': 'red,bg_white',
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
311 },
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
312 secondary_log_colors={},
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
313 style='%'
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 755
diff changeset
314 ))
463
1ceb26846eca add separate :matchPredicate support. some build and log edits.
drewp@bigasterisk.com
parents: 392
diff changeset
315 defer.setDebugging(True)
1ceb26846eca add separate :matchPredicate support. some build and log edits.
drewp@bigasterisk.com
parents: 392
diff changeset
316
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
317 if arg['-i']:
179
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
318 import twisted.python.log
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands
drewp@bigasterisk.com
parents: 129
diff changeset
319 twisted.python.log.startLogging(sys.stdout)
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
320
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
321 getLogger('fetch').setLevel(DEBUG if arg['-i'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
322 log.setLevel(DEBUG if arg['-r'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
323 getLogger('output').setLevel(DEBUG if arg['-o'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
324
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
325
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
326 if __name__ == '__main__':
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
327 arg = docopt("""
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
328 Usage: reasoning.py [options]
45
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches
drewp@bigasterisk.com
parents: 33
diff changeset
329
303
66fe7a93753d reasoning uses sse_collector
drewp@bigasterisk.com
parents: 288
diff changeset
330 -i Verbose log on the input phase
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
331 -r Verbose log on the reasoning phase and web stuff
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
332 -o Verbose log on the actions/output phase
600
a34dee00a989 comments and logging
drewp@bigasterisk.com
parents: 571
diff changeset
333 -v Everywhere verbose
608
62171aa2bc4e mock output mode
drewp@bigasterisk.com
parents: 600
diff changeset
334 --mockoutput Don't make outgoing requests
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
335 """)
608
62171aa2bc4e mock output mode
drewp@bigasterisk.com
parents: 600
diff changeset
336
62171aa2bc4e mock output mode
drewp@bigasterisk.com
parents: 600
diff changeset
337 r = Reasoning(arg['--mockoutput'])
280
c192d37b2bc8 lots of logging updates (patch may be imprecise)
drewp@bigasterisk.com
parents: 275
diff changeset
338 configLogging(arg)
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 180
diff changeset
339 reactor.listenTCP(9071, Application(r), interface='::')
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
340 reactor.run()