Mercurial > code > home > repos > homeauto

annotate service/reasoning/inference.py @ 1434:3ec2ca0b63ff

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix download url separator Ignore-this: 2426fe3cc2f5fd30e5c9c87903fed4f4 darcs-hash:be42c3ea21d197f0ed222855aaad88253d477313
author drewp <drewp@bigasterisk.com>
date Sun, 11 Aug 2019 03:25:50 -0700
parents e3ae71fbd3bd
children f3f667769aef
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
1 """
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
2 see ./reasoning for usage
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
3 """
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
4
1093
e3ae71fbd3bd logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp <drewp@bigasterisk.com>
parents: 1092
diff changeset
5 import os, contextlib
919
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents: 850
diff changeset
6 try:
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents: 850
diff changeset
7 from rdflib.Graph import Graph
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents: 850
diff changeset
8 except ImportError:
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents: 850
diff changeset
9 from rdflib import Graph
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
10
1093
e3ae71fbd3bd logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp <drewp@bigasterisk.com>
parents: 1092
diff changeset
11 from rdflib import Namespace
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
12 from rdflib.parser import StringInputSource
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
13
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
14 from FuXi.Rete.Util import generateTokenSet
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
15 from FuXi.Rete import ReteNetwork
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
16 from FuXi.Rete.RuleStore import N3RuleStore
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
17
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
18 from greplin import scales
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
19 STATS = scales.collection('/web',
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
20 scales.PmfStat('readRules'))
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
21
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
22 from escapeoutputstatements import escapeOutputStatements
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
23 ROOM = Namespace("http://projects.bigasterisk.com/room/")
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
24
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
25 def _loadAndEscape(ruleStore, n3, outputPatterns):
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
26 ruleGraph = Graph(ruleStore)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
27
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
28 # Can't escapeOutputStatements in the ruleStore since it
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
29 # doesn't support removals. Can't copy plainGraph into
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
30 # ruleGraph since something went wrong with traversing the
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
31 # triples inside quoted graphs, and I lose all the bodies
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
32 # of my rules. This serialize/parse version is very slow (400ms),
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
33 # but it only runs when the file changes.
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
34 plainGraph = Graph()
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
35 plainGraph.parse(StringInputSource(n3), format='n3') # for inference
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
36 escapeOutputStatements(plainGraph, outputPatterns=outputPatterns)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
37 expandedN3 = plainGraph.serialize(format='n3')
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
38
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
39 ruleGraph.parse(StringInputSource(expandedN3), format='n3')
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
40
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
41 _rulesCache = (None, None, None, None)
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
42 def readRules(rulesPath, outputPatterns):
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
43 """
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
44 returns (rulesN3, ruleStore)
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
45
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
46 This includes escaping certain statements in the output
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
47 (implied) subgraaphs so they're not confused with input
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
48 statements.
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
49 """
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
50 global _rulesCache
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
51
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
52 with STATS.readRules.time():
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
53 mtime = os.path.getmtime(rulesPath)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
54 key = (rulesPath, mtime)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
55 if _rulesCache[:2] == key:
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
56 _, _, rulesN3, ruleStore = _rulesCache
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
57 else:
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
58 rulesN3 = open(rulesPath).read() # for web display
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
59
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
60 ruleStore = N3RuleStore()
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
61 _loadAndEscape(ruleStore, rulesN3, outputPatterns)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
62 log.debug('%s rules' % len(ruleStore.rules))
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
63
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
64 _rulesCache = key + (rulesN3, ruleStore)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
65 return rulesN3, ruleStore
1089
cb7fa2f30df9 rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp <drewp@bigasterisk.com>
parents: 1086
diff changeset
66
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
67 def infer(graph, rules):
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
68 """
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
69 returns new graph of inferred statements. Plain rete api seems to
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
70 alter rules.formulae and rules.rules, but this function does not
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
71 alter the incoming rules object, so you can cache it.
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
72 """
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
73 # based on fuxi/tools/rdfpipe.py
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
74 target = Graph()
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
75 tokenSet = generateTokenSet(graph)
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
76 with _dontChangeRulesStore(rules):
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
77 network = ReteNetwork(rules, inferredTarget=target)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
78 network.feedFactsToAdd(tokenSet)
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
79
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
80 return target
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
81
1092
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
82 @contextlib.contextmanager
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
83 def _dontChangeRulesStore(rules):
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
84 if not hasattr(rules, '_stashOriginalRules'):
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
85 rules._stashOriginalRules = rules.rules[:]
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
86 yield
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
87 for k in rules.formulae.keys():
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
88 if not k.startswith('_:Formula'):
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
89 del rules.formulae[k]
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
90 rules.rules = rules._stashOriginalRules[:]
54de5144900d switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp <drewp@bigasterisk.com>
parents: 1089
diff changeset
91
825
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
92 import time, logging
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
93 log = logging.getLogger()
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
94 def logTime(func):
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
95 def inner(*args, **kw):
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
96 t1 = time.time()
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
97 try:
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
98 ret = func(*args, **kw)
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
99 finally:
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
100 log.info("Call to %s took %.1f ms" % (
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
101 func.__name__, 1000 * (time.time() - t1)))
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
102 return ret
fc753b24f69a move reasoning from /my/proj/room, new integration with magma
drewp <drewp@bigasterisk.com>
parents:
diff changeset
103 return inner