homeauto: service/reasoning/reasoning.py annotate

annotate service/reasoning/reasoning.py @ 1749:4b29ce991e59

cloudfree plug sends mqtt metrics, which we export to victoriametrics

author	drewp@bigasterisk.com
date	Sun, 28 Apr 2024 16:01:38 -0700
parents	c8562ace4917
children

rev	line source
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	1 """
46 f5623d9b07fd rewriting reasoning to use graphs for config drewp@bigasterisk.com parents: 45 diff changeset	2 Graph consists of:
f5623d9b07fd rewriting reasoning to use graphs for config drewp@bigasterisk.com parents: 45 diff changeset	3 input/* (read at startup)
f5623d9b07fd rewriting reasoning to use graphs for config drewp@bigasterisk.com parents: 45 diff changeset	4 webinput/* (new files are noticed in here)
f5623d9b07fd rewriting reasoning to use graphs for config drewp@bigasterisk.com parents: 45 diff changeset	5 any number of remote graphs, specified in the other graph as objects of (:reasoning, :source, *), reread constantly
f5623d9b07fd rewriting reasoning to use graphs for config drewp@bigasterisk.com parents: 45 diff changeset	6
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	7 gather subgraphs from various services, run them through a rules
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	8 engine, and make http requests with the conclusions.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	9
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	10 E.g. 'when drew's phone is near the house, and someone is awake,
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	11 unlock the door when the door's motion sensor is activated'
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	12
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	13 When do we gather? The services should be able to trigger us, perhaps
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	14 with PSHB, that their graph has changed.
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	15 """
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	16 import json
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	17 import sys
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	18 import time
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	19 import traceback
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	20 from logging import DEBUG, WARN, getLogger
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	21 from typing import Dict, Optional, Set, Tuple
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	22
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	23 import cyclone.web
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	24 import cyclone.websocket
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	25 from colorlog import ColoredFormatter
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	26 from docopt import docopt
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	27 from patchablegraph import (CycloneGraphEventsHandler, CycloneGraphHandler, PatchableGraph)
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	28 from prometheus_client import Counter, Gauge, Histogram, Summary
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	29 from prometheus_client.exposition import generate_latest
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	30 from prometheus_client.registry import REGISTRY
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	31 from rdflib import RDF, Graph, Literal, Namespace, URIRef
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	32 from rdflib.graph import ConjunctiveGraph
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	33 from rdflib.term import Node
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	34 from standardservice.logsetup import log, verboseLogging
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	35 from twisted.internet import defer, reactor
287 3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms) drewp@bigasterisk.com parents: 286 diff changeset	36
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	37 from actions import Actions, PutOutputsTable
284 95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them. drewp@bigasterisk.com parents: 283 diff changeset	38 from escapeoutputstatements import unquoteOutputStatements
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	39 from inference import infer, readRules
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	40 from inputgraph import InputGraph
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	41
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	42 ROOM = Namespace("http://projects.bigasterisk.com/room/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	43 DEV = Namespace("http://projects.bigasterisk.com/device/")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	44
281 9728288c7f2f refactor drewp@bigasterisk.com parents: 280 diff changeset	45 NS = {'': ROOM, 'dev': DEV}
283 0b0fb67b0b3a fix input/output statement ambiguity problem by allowing quoted output statements drewp@bigasterisk.com parents: 282 diff changeset	46
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	47 GRAPH_CHANGED_CALLS = Summary('graph_changed_calls', 'calls')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	48 UPDATE_RULES_CALLS = Summary('update_rules_calls', 'calls')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	49
287 3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms) drewp@bigasterisk.com parents: 286 diff changeset	50
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	51 def ntStatement(stmt: Tuple[Node, Node, Node]):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	52
755 ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	53 def compact(u):
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	54 if isinstance(u, URIRef) and u.startswith(ROOM):
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	55 return 'room:' + u[len(ROOM):]
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	56 return u.n3()
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	57
755 ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	58 return '%s %s %s .' % (compact(stmt[0]), compact(stmt[1]), compact(stmt[2]))
ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	59
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	60
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	61 class Reasoning(object):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	62 ruleStore: ConjunctiveGraph
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	63
608 62171aa2bc4e mock output mode drewp@bigasterisk.com parents: 600 diff changeset	64 def __init__(self, mockOutput=False):
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	65 self.prevGraph = None
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	66
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	67 self.rulesN3 = "(not read yet)"
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	68 self.inferred = Graph() # gets replaced in each graphChanged call
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	69 self.outputGraph = PatchableGraph() # copy of inferred, for now
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	70
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	71 self.inputGraph = InputGraph([], self.graphChanged)
755 ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	72 self.actions = Actions(self.inputGraph, sendToLiveClients, mockOutput=mockOutput)
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	73 self.inputGraph.updateFileData()
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	74
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	75 @UPDATE_RULES_CALLS.time()
240 0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	76 def updateRules(self):
284 95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them. drewp@bigasterisk.com parents: 283 diff changeset	77 rulesPath = 'rules.n3'
240 0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	78 try:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	79 t1 = time.time()
287 3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms) drewp@bigasterisk.com parents: 286 diff changeset	80 self.rulesN3, self.ruleStore = readRules(
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	81 rulesPath,
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	82 outputPatterns=[
286 da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	83 # Incomplete. See escapeoutputstatements.py for
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	84 # explanation.
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	85 (None, ROOM['brightness'], None),
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	86 (None, ROOM['playState'], None),
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	87 (None, ROOM['powerState'], None),
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	88 (None, ROOM['state'], None),
da0b3a1394a3 more output statement patterns drewp@bigasterisk.com parents: 285 diff changeset	89 ])
240 0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	90 ruleParseTime = time.time() - t1
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	91 except ValueError:
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	92 # this is so if you're just watching the inferred output,
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	93 # you'll see the error too
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	94 self.inferred = Graph()
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	95 self.inferred.add((ROOM['reasoner'], ROOM['ruleParseError'], Literal(traceback.format_exc())))
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	96 self.copyOutput()
240 0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	97 raise
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	98 return [(ROOM['reasoner'], ROOM['ruleParseTime'], Literal(ruleParseTime))], ruleParseTime
240 0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file drewp@bigasterisk.com parents: 235 diff changeset	99
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	100 @GRAPH_CHANGED_CALLS.time()
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	101 def graphChanged(self, inputGraph: InputGraph, oneShot=False, oneShotGraph: Graph = None):
285 c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	102 """
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	103 If we're getting called for a oneShot event, the oneShotGraph
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	104 statements are already in inputGraph.getGraph().
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	105 """
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	106 log.info("----------------------")
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	107 log.info("graphChanged (oneShot=%s %s stmts):", oneShot, len(oneShotGraph) if oneShotGraph is not None else 0)
392 79d041273e26 mqtt has two devices now. various older cleanups. drewp@bigasterisk.com parents: 351 diff changeset	108 if oneShotGraph:
79d041273e26 mqtt has two devices now. various older cleanups. drewp@bigasterisk.com parents: 351 diff changeset	109 for stmt in oneShotGraph:
755 ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	110 log.info(" oneshot -> %s", ntStatement(stmt))
64 e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	111 t1 = time.time()
49 0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	112 oldInferred = self.inferred
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	113 try:
281 9728288c7f2f refactor drewp@bigasterisk.com parents: 280 diff changeset	114 ruleStatStmts, ruleParseSec = self.updateRules()
287 3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms) drewp@bigasterisk.com parents: 286 diff changeset	115
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	116 self.inferred, inferSec = self._makeInferred(inputGraph.getGraph())
283 0b0fb67b0b3a fix input/output statement ambiguity problem by allowing quoted output statements drewp@bigasterisk.com parents: 282 diff changeset	117
284 95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them. drewp@bigasterisk.com parents: 283 diff changeset	118 self.inferred += unquoteOutputStatements(self.inferred)
287 3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms) drewp@bigasterisk.com parents: 286 diff changeset	119
285 c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	120 self.inferred += ruleStatStmts
49 0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	121
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	122 if oneShot:
285 c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	123 # It's possible a oneShotGraph statement didn't
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	124 # trigger a rule to do something, but was itself the
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	125 # output statement. Probably we could just mix in the
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	126 # whole inputGraph here and not special-case the
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	127 # oneShotGraph.
c7476e2387dc minor cleanups drewp@bigasterisk.com parents: 284 diff changeset	128 self.inferred += oneShotGraph
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	129
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	130 t3 = time.time()
755 ffcad6bf9c57 something with rx on inputgraph, i forget. also cleanup imports and logging of oneshot drewp@bigasterisk.com parents: 722 diff changeset	131 self.actions.putResults(self.inferred)
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	132 putResultsTime = time.time() - t3
49 0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	133 finally:
0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	134 if oneShot:
0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	135 self.inferred = oldInferred
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	136 log.info("graphChanged took %.1f ms (rule parse %.1f ms, infer %.1f ms, putResults %.1f ms)" %
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	137 ((time.time() - t1) * 1000, ruleParseSec * 1000, inferSec * 1000, putResultsTime * 1000))
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	138 if not oneShot:
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	139 self.copyOutput()
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	140
5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	141 def copyOutput(self):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	142 self.outputGraph.setToGraph((s, p, o, ROOM['inferred']) for s, p, o in self.inferred)
49 0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	143
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	144 def _makeInferred(self, inputGraph: ConjunctiveGraph):
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	145 t1 = time.time()
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	146
49 0aeb8d6ea124 cleanup drewp@bigasterisk.com parents: 48 diff changeset	147 out = infer(inputGraph, self.ruleStore)
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	148 for p, n in NS.items():
281 9728288c7f2f refactor drewp@bigasterisk.com parents: 280 diff changeset	149 out.bind(p, n, override=True)
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	150
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	151 inferenceTime = time.time() - t1
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	152 out.add((ROOM['reasoner'], ROOM['inferenceTime'], Literal(inferenceTime)))
288 e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization. drewp@bigasterisk.com parents: 287 diff changeset	153 return out, inferenceTime
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	154
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	155
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	156 class Index(cyclone.web.RequestHandler):
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	157
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	158 def get(self):
180 47682350e6f2 reasoning index page to html drewp@bigasterisk.com parents: 179 diff changeset	159 self.set_header("Content-Type", "text/html")
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	160 self.write(open('index.html').read())
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	161
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	162
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	163 class ImmediateUpdate(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	164
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	165 def put(self):
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	166 """
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	167 request an immediate load of the remote graphs; the thing we
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	168 do in the background anyway. No payload.
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	169
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	170 Using PUT because this is idempotent and retryable and
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	171 everything.
54 42411726a7ca screen out more values that change on every update drewp@bigasterisk.com parents: 49 diff changeset	172
42411726a7ca screen out more values that change on every update drewp@bigasterisk.com parents: 49 diff changeset	173 todo: this should do the right thing when many requests come
42411726a7ca screen out more values that change on every update drewp@bigasterisk.com parents: 49 diff changeset	174 in very quickly
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	175 """
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	176 log.warn("immediateUpdate from %s %s - ignored", self.request.headers.get('User-Agent', '?'),
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	177 self.request.headers['Host'])
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	178 self.set_status(202)
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	179
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	180
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	181 class OneShot(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	182
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	183 def post(self):
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	184 """
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	185 payload is an rdf graph. The statements are momentarily added
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	186 to the input graph for exactly one update.
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	187
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	188 todo: how do we go from a transition like doorclosed-to-open
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	189 to a oneshot event? the upstream shouldn't have to do it. Do
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	190 we make those oneshot events here? for every object change?
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	191 there are probably special cases regarding startup time when
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	192 everything appears to be a 'change'.
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	193 """
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	194 try:
392 79d041273e26 mqtt has two devices now. various older cleanups. drewp@bigasterisk.com parents: 351 diff changeset	195 log.info('POST to oneShot, headers=%s', self.request.headers)
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	196 ct = self.request.headers.get('Content-Type', self.request.headers.get('content-type', ''))
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	197 dt = self.settings.reasoning.inputGraph.addOneShotFromString(self.request.body, ct)
281 9728288c7f2f refactor drewp@bigasterisk.com parents: 280 diff changeset	198 self.set_header('x-graph-ms', str(1000 * dt))
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	199 except Exception as e:
315 8db492695a51 log oneshot tracebacks drewp@bigasterisk.com parents: 314 diff changeset	200 traceback.print_exc()
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	201 log.error(e)
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	202 raise
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	203
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	204
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	205 # for reuse
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	206 class GraphResource(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	207
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	208 def get(self, which: str):
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	209 self.set_header("Content-Type", "application/json")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	210 r = self.settings.reasoning
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	211 g = {
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	212 'lastInput': r.inputGraph.getGraph(),
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	213 'lastOutput': r.inferred,
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	214 }[which]
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	215 self.write(self.jsonRdf(g))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	216
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	217 def jsonRdf(self, g):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	218 return json.dumps(sorted(list(g)))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	219
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	220
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	221 class NtGraphs(cyclone.web.RequestHandler):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	222 """same as what gets posted above"""
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	223
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	224 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	225 r = self.settings.reasoning
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	226 inputGraphNt = r.inputGraph.getGraph().serialize(format="nt")
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	227 inferredNt = r.inferred.serialize(format="nt")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	228 self.set_header("Content-Type", "application/json")
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	229 self.write(json.dumps({"input": inputGraphNt, "inferred": inferredNt}))
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	230
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	231
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	232 class Rules(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	233
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	234 def get(self):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	235 self.set_header("Content-Type", "text/plain")
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	236 self.write(self.settings.reasoning.rulesN3)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	237
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	238
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	239 class Status(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	240
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	241 def get(self):
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	242 self.set_header("Content-Type", "text/plain")
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	243 g = self.settings.reasoning.inputGraph.getGraph()
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	244 msg = ""
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	245 for badSource in g.subjects(RDF.type, ROOM['FailedGraphLoad']):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	246 msg += "GET %s failed (%s). " % (badSource, g.value(badSource, ROOM['graphLoadError']))
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	247 if not msg:
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	248 self.finish("all inputs ok")
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	249 return
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	250 self.set_status(500)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	251 self.finish(msg)
5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	252
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	253
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	254 class Static(cyclone.web.RequestHandler):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	255
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	256 def get(self, p: str):
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	257 self.write(open(p).read())
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	258
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	259
756 f3f667769aef python 3! and some types and cleanups drewp@bigasterisk.com parents: 755 diff changeset	260 liveClients: Set[cyclone.websocket.WebSocketHandler] = set()
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	261
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	262
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	263 def sendToLiveClients(d: Dict[str, object] = None, asJson: Optional[str] = None):
64 e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	264 j = asJson or json.dumps(d)
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	265 for c in liveClients:
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	266 c.sendMessage(j)
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	267
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	268
64 e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	269 class Events(cyclone.websocket.WebSocketHandler):
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	270
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	271 def connectionMade(self, args, *kwargs):
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	272 log.info("websocket opened")
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	273 liveClients.add(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	274
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	275 def connectionLost(self, reason):
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	276 log.info("websocket closed")
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	277 liveClients.remove(self)
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	278
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	279 def messageReceived(self, message):
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	280 log.info("got message %s" % message)
e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	281
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	282
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	283 class Metrics(cyclone.web.RequestHandler):
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	284
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	285 def get(self):
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	286 self.add_header('content-type', 'text/plain')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	287 self.write(generate_latest(REGISTRY))
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	288
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	289
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	290 class Application(cyclone.web.Application):
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	291
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	292 def __init__(self, reasoning):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	293 handlers = [
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	294 (r"/", Index),
47 0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	295 (r"/immediateUpdate", ImmediateUpdate),
0448fbd96a31 scan more input files. oneshot and immediate update features. drewp@bigasterisk.com parents: 46 diff changeset	296 (r"/oneShot", OneShot),
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	297 (r'/putOutputs', PutOutputsTable),
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	298 (r'/(jquery.min.js)', Static),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	299 (r'/(lastInput\|lastOutput)Graph', GraphResource),
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	300 (r"/graph/reasoning", CycloneGraphHandler, {
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	301 'masterGraph': reasoning.outputGraph
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	302 }),
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	303 (r"/graph/reasoning/events", CycloneGraphEventsHandler, {
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	304 'masterGraph': reasoning.outputGraph
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	305 }),
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	306 (r'/ntGraphs', NtGraphs),
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	307 (r'/rules', Rules),
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	308 (r'/status', Status),
64 e573af8c2428 reasoning has a websocket server that broadcasts some events drewp@bigasterisk.com parents: 54 diff changeset	309 (r'/events', Events),
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	310 (r'/metrics', Metrics),
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	311 ]
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	312 cyclone.web.Application.__init__(self, handlers, reasoning=reasoning)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	313
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	314
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	315 def configLogging(arg):
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	316 log.setLevel(WARN)
609 5290df01d911 reasoning web page uses rdf/browse/graphView for inputs and outputs now drewp@bigasterisk.com parents: 608 diff changeset	317
600 a34dee00a989 comments and logging drewp@bigasterisk.com parents: 571 diff changeset	318 if arg['-i'] or arg['-r'] or arg['-o'] or arg['-v']:
795 c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	319 log.handlers[0].setFormatter(
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	320 ColoredFormatter(
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	321 "%(log_color)s%(levelname)-8s %(name)-6s %(filename)-12s:%(lineno)-3s %(funcName)-20s%(reset)s %(white)s%(message)s",
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	322 datefmt=None,
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	323 reset=True,
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	324 log_colors={
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	325 'DEBUG': 'cyan',
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	326 'INFO': 'green',
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	327 'WARNING': 'yellow',
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	328 'ERROR': 'red',
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	329 'CRITICAL': 'red,bg_white',
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	330 },
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	331 secondary_log_colors={},
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics. drewp@bigasterisk.com parents: 760 diff changeset	332 style='%'))
463 1ceb26846eca add separate :matchPredicate support. some build and log edits. drewp@bigasterisk.com parents: 392 diff changeset	333 defer.setDebugging(True)
1ceb26846eca add separate :matchPredicate support. some build and log edits. drewp@bigasterisk.com parents: 392 diff changeset	334
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	335 if arg['-i']:
179 0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	336 import twisted.python.log
0f2ac014d9ea reasoning: -v verbose setting, more oneshot support, some custom stuff for mpd commands drewp@bigasterisk.com parents: 129 diff changeset	337 twisted.python.log.startLogging(sys.stdout)
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	338
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	339 getLogger('fetch').setLevel(DEBUG if arg['-i'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	340 log.setLevel(DEBUG if arg['-r'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	341 getLogger('output').setLevel(DEBUG if arg['-o'] else WARN)
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	342
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	343
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	344 if __name__ == '__main__':
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	345 arg = docopt("""
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	346 Usage: reasoning.py [options]
45 5b0f970e3d52 /status page, errors on failed graphs, time reports of successful fetches drewp@bigasterisk.com parents: 33 diff changeset	347
303 66fe7a93753d reasoning uses sse_collector drewp@bigasterisk.com parents: 288 diff changeset	348 -i Verbose log on the input phase
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	349 -r Verbose log on the reasoning phase and web stuff
c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	350 -o Verbose log on the actions/output phase
600 a34dee00a989 comments and logging drewp@bigasterisk.com parents: 571 diff changeset	351 -v Everywhere verbose
608 62171aa2bc4e mock output mode drewp@bigasterisk.com parents: 600 diff changeset	352 --mockoutput Don't make outgoing requests
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	353 """)
608 62171aa2bc4e mock output mode drewp@bigasterisk.com parents: 600 diff changeset	354
62171aa2bc4e mock output mode drewp@bigasterisk.com parents: 600 diff changeset	355 r = Reasoning(arg['--mockoutput'])
280 c192d37b2bc8 lots of logging updates (patch may be imprecise) drewp@bigasterisk.com parents: 275 diff changeset	356 configLogging(arg)
235 5ad229334a88 rdflib can parse trig now drewp@bigasterisk.com parents: 180 diff changeset	357 reactor.listenTCP(9071, Application(r), interface='::')
20 3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma drewp@bigasterisk.com parents: diff changeset	358 reactor.run()

Mercurial > code > home > repos > homeauto

annotate service/reasoning/reasoning.py @ 1749:4b29ce991e59