Mercurial > code > home > repos > homeauto

annotate service/reasoning/inference.py @ 1685:6b80a6c58907

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
minor changes to several services
author drewp@bigasterisk.com
date Mon, 27 Sep 2021 23:12:43 -0700
parents c8562ace4917
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
1 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
2 see ./reasoning for usage
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
3 """
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
4
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
5 import os
1685
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
6 import logging
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
7
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
8 from prometheus_client import Summary
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
9 from rdflib import Graph, Namespace
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
10 from rdflib.graph import ConjunctiveGraph
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
11 from rdflib.parser import StringInputSource
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
12
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
13 from escapeoutputstatements import escapeOutputStatements
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
14
1685
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
15 log = logging.getLogger()
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
16
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
17 READ_RULES_CALLS = Summary('read_rules_calls', 'calls')
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
18
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
19 ROOM = Namespace("http://projects.bigasterisk.com/room/")
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
20 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
21
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
22
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
23 def _loadAndEscape(ruleStore: ConjunctiveGraph, n3: bytes, outputPatterns):
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
24 ruleStore.parse(StringInputSource(n3), format='n3')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
25 return
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
26 ruleGraph = Graph(ruleStore)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
27
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
28 # Can't escapeOutputStatements in the ruleStore since it
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
29 # doesn't support removals. Can't copy plainGraph into
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
30 # ruleGraph since something went wrong with traversing the
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
31 # triples inside quoted graphs, and I lose all the bodies
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
32 # of my rules. This serialize/parse version is very slow (400ms),
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
33 # but it only runs when the file changes.
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
34 plainGraph = Graph()
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
35 plainGraph.parse(StringInputSource(n3.encode('utf8')), format='n3') # for inference
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
36 escapeOutputStatements(plainGraph, outputPatterns=outputPatterns)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
37 expandedN3 = plainGraph.serialize(format='n3')
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
38
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
39 ruleGraph.parse(StringInputSource(expandedN3), format='n3')
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
40
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
41
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
42 _rulesCache = (None, None, None, None)
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
43
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
44
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
45 def readRules(rulesPath, outputPatterns):
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
46 """
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
47 returns (rulesN3, ruleStore)
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
48
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
49 This includes escaping certain statements in the output
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
50 (implied) subgraaphs so they're not confused with input
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
51 statements.
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
52 """
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
53 global _rulesCache
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
54
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
55 with READ_RULES_CALLS.time():
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
56 mtime = os.path.getmtime(rulesPath)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
57 key = (rulesPath, mtime)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
58 if _rulesCache[:2] == key:
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
59 _, _, rulesN3, ruleStore = _rulesCache
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
60 else:
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
61 rulesN3 = open(rulesPath, 'rb').read() # for web display
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
62
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
63 ruleStore = ConjunctiveGraph()
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
64 _loadAndEscape(ruleStore, rulesN3, outputPatterns)
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
65 log.debug('%s rules' % len(ruleStore))
756
f3f667769aef python 3! and some types and cleanups
drewp@bigasterisk.com
parents: 288
diff changeset
66
287
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
67 _rulesCache = key + (rulesN3, ruleStore)
3b61c0dfaaef switch from evtiming to greplin.scales. Optimize rules reader to reuse previous data (400ms -> 0.6ms)
drewp@bigasterisk.com
parents: 284
diff changeset
68 return rulesN3, ruleStore
284
95f72a22965d rules become simple-looking again; fix the ambiguity in memory after loading them.
drewp@bigasterisk.com
parents: 281
diff changeset
69
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
70
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
71 def infer(graph: ConjunctiveGraph, rules: ConjunctiveGraph):
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
72 """
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
73 returns new graph of inferred statements.
20
3f0dd03112b5 move reasoning from /my/proj/room, new integration with magma
drewp@bigasterisk.com
parents:
diff changeset
74 """
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
75 log.info(f'Begin inference of graph len={len(graph)} with rules len={len(rules)}:')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
76
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
77 workingSet = ConjunctiveGraph()
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
78 workingSet.addN(graph.quads())
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
79
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
80 implied = ConjunctiveGraph()
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
81
1685
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
82 bailout_iterations = 100
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
83 delta = 1
1685
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
84 while delta > 0 and bailout_iterations > 0:
6b80a6c58907 minor changes to several services
drewp@bigasterisk.com
parents: 795
diff changeset
85 bailout_iterations -= 1
795
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
86 delta = -len(implied)
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
87
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
88 for r in rules:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
89 if r[1] == LOG['implies']:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
90 containsSetup = all(st in workingSet for st in r[0])
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
91 if containsSetup:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
92 log.info(f' Rule {r[0]} -> present={containsSetup}')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
93 for st in r[0]:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
94 log.info(f' {st[0].n3()} {st[1].n3()} {st[2].n3()}')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
95
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
96 log.info(f' ...implies {len(r[2])} statements')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
97 if containsSetup:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
98 for st in r[2]:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
99 workingSet.add(st)
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
100 implied.add(st)
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
101 else:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
102 log.info(f' {r}')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
103 delta += len(implied)
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
104 log.info(f' this inference round added {delta} more implied stmts')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
105 log.info(f'{len(implied)} stmts implied:')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
106 for st in implied:
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
107 log.info(f' {st}')
c8562ace4917 big updates for k8s, py3, drop FuXi, use prometheus for metrics.
drewp@bigasterisk.com
parents: 756
diff changeset
108 return implied