Mercurial > code > home > repos > homeauto

annotate service/reasoning/inference.py @ 756:f3f667769aef

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
python 3! and some types and cleanups Ignore-this: 3453a547ee745fa83668f36956c835cd
author drewp@bigasterisk.com
date Fri, 14 Feb 2020 00:07:23 -0800
parents e03696277b32
children c8562ace4917
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
1 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
2 see ./reasoning for usage
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
3 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
4
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
5 import os, contextlib
114
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents: 45
diff changeset
6 try:
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents: 45
diff changeset
7 from rdflib.Graph import Graph
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents: 45
diff changeset
8 except ImportError:
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents: 45
diff changeset
9 from rdflib import Graph
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
10
288
e03696277b32 logging and cleanups. Upgrade fuxi to have the cPickle optimization.
drewp@bigasterisk.com
parents: 287
diff changeset
11 from rdflib import Namespace
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
12 from rdflib.parser import StringInputSource
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
13
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
14 from FuXi.Rete.Util import generateTokenSet
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
15 from FuXi.Rete import ReteNetwork
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
16 from FuXi.Rete.RuleStore import N3RuleStore
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
17
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
18 from greplin import scales
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
19 STATS = scales.collection('/web',
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
20 scales.PmfStat('readRules'))
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
21
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
22 from escapeoutputstatements import escapeOutputStatements
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
23 ROOM = Namespace("http://projects.bigasterisk.com/room/")
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
24
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
25 def _loadAndEscape(ruleStore, n3, outputPatterns):
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
26 ruleGraph = Graph(ruleStore)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
27
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
28 # Can't escapeOutputStatements in the ruleStore since it
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
29 # doesn't support removals. Can't copy plainGraph into
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
30 # ruleGraph since something went wrong with traversing the
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
31 # triples inside quoted graphs, and I lose all the bodies
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
32 # of my rules. This serialize/parse version is very slow (400ms),
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
33 # but it only runs when the file changes.
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
34 plainGraph = Graph()
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
35 plainGraph.parse(StringInputSource(n3.encode('utf8')), format='n3') # for inference
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
36 escapeOutputStatements(plainGraph, outputPatterns=outputPatterns)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
37 expandedN3 = plainGraph.serialize(format='n3')
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
38
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
39 ruleGraph.parse(StringInputSource(expandedN3), format='n3')
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
40
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
41 _rulesCache = (None, None, None, None)
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
42 def readRules(rulesPath, outputPatterns):
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
43 """
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
44 returns (rulesN3, ruleStore)
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
45
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
46 This includes escaping certain statements in the output
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
47 (implied) subgraaphs so they're not confused with input
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
48 statements.
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
49 """
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
50 global _rulesCache
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
51
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
52 with STATS.readRules.time():
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
53 mtime = os.path.getmtime(rulesPath)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
54 key = (rulesPath, mtime)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
55 if _rulesCache[:2] == key:
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
56 _, _, rulesN3, ruleStore = _rulesCache
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
57 else:
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
58 rulesN3 = open(rulesPath).read() # for web display
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
59
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
60 ruleStore = N3RuleStore()
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
61 _loadAndEscape(ruleStore, rulesN3, outputPatterns)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
62 log.debug('%s rules' % len(ruleStore.rules))
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
63
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
64 _rulesCache = key + (rulesN3, ruleStore)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
65 return rulesN3, ruleStore
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
66
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
67 def infer(graph, rules):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
68 """
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
69 returns new graph of inferred statements. Plain rete api seems to
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
70 alter rules.formulae and rules.rules, but this function does not
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
71 alter the incoming rules object, so you can cache it.
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
72 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
73 # based on fuxi/tools/rdfpipe.py
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
74 target = Graph()
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
75 tokenSet = generateTokenSet(graph)
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
76 with _dontChangeRulesStore(rules):
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
77 network = ReteNetwork(rules, inferredTarget=target)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
78 network.feedFactsToAdd(tokenSet)
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
79
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
80 return target
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
81
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
82 @contextlib.contextmanager
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
83 def _dontChangeRulesStore(rules):
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
84 if not hasattr(rules, '_stashOriginalRules'):
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
85 rules._stashOriginalRules = rules.rules[:]
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
86 yield
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
87 for k in list(rules.formulae.keys()):
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
88 if not k.startswith('_:Formula'):
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
89 del rules.formulae[k]
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
90 rules.rules = rules._stashOriginalRules[:]
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
91
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
92 import time, logging
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
93 log = logging.getLogger()
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
94 def logTime(func):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
95 def inner(*args, **kw):
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
96 t1 = time.time()
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
97 try:
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
98 ret = func(*args, **kw)
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
99 finally:
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
100 log.info("Call to %s took %.1f ms" % (
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
101 func.__name__, 1000 * (time.time() - t1)))
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
102 return ret
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
103 return inner