Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1696:cdf706cf5f82 master

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
fix a test (odometer rollover direction)
author drewp@bigasterisk.com
date Mon, 27 Sep 2021 22:56:25 -0700
parents 5c2565e63297
children 88f6e9bf69d1
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
1 """
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
4 """
1588
0757fafbfdab WIP inferencer - partial var and function support
drewp@bigasterisk.com
parents: 1587
diff changeset
5 import itertools
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
6 import logging
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
7 import time
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
8 from collections import defaultdict
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
9 from dataclasses import dataclass
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
10 from typing import Dict, Iterator, List, Optional, Tuple, Union, cast
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
11 from pathlib import Path
1588
0757fafbfdab WIP inferencer - partial var and function support
drewp@bigasterisk.com
parents: 1587
diff changeset
12
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
13 from prometheus_client import Histogram, Summary
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
14 from rdflib import Graph, Namespace
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
15 from rdflib.graph import ConjunctiveGraph
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
16 from rdflib.term import Node, URIRef, Variable
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
17
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
18 from candidate_binding import CandidateBinding
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
19 from inference_types import (BindingUnknown, Inconsistent, RhsBnode, RuleUnboundBnode, Triple, WorkingSetBnode)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
20 from lhs_evaluation import functionsFor
1650
2061df259224 move graphDump (on its way out, since reprs are getting better)
drewp@bigasterisk.com
parents: 1649
diff changeset
21 from rdf_debug import graphDump
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
22 from stmt_chunk import AlignedRuleChunk, Chunk, ChunkedGraph, applyChunky
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
23 from structured_log import StructuredLog
1605
449746d1598f WIP move evaluation to new file
drewp@bigasterisk.com
parents: 1603
diff changeset
24
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
25 log = logging.getLogger('infer')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
26 odolog = logging.getLogger('infer.odo') # the "odometer" logic
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
27 ringlog = logging.getLogger('infer.ring') # for ChunkLooper
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
28
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
29 INDENT = ' '
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
30
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
31 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
32 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
33
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
34 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
35 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
36 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
37
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
38
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
39 class NoOptions(ValueError):
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
40 """ChunkLooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
41
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
42
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
43 def debug(logger, slog: Optional[StructuredLog], msg):
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
44 logger.debug(msg)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
45 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
46 slog.say(msg)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
47
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
48
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
49 _chunkLooperShortId = itertools.count()
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
50
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
51
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
52 @dataclass
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
53 class ChunkLooper:
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
54 """given one LHS Chunk, iterate through the possible matches for it,
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
55 returning what bindings they would imply. Only distinct bindings are
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
56 returned. The bindings build on any `prev` ChunkLooper's results.
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
57
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
58 In the odometer metaphor used below, this is one of the rings.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
59
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
60 This iterator is restartable."""
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
61 lhsChunk: Chunk
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
62 prev: Optional['ChunkLooper']
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
63 workingSet: 'ChunkedGraph'
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
64 slog: Optional[StructuredLog]
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
65
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
66 def __repr__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
67 return f'{self.__class__.__name__}{self._shortId}{"<pastEnd>" if self.pastEnd() else ""}'
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
68
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
69 def __post_init__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
70 self._shortId = next(_chunkLooperShortId)
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
71 self._alignedMatches = list(self.lhsChunk.ruleMatchesFrom(self.workingSet))
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
72 del self.workingSet
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
73
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
74 # only ours- do not store prev, since it could change without us
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
75 self._current = CandidateBinding({})
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
76 self.currentSourceChunk: Optional[Chunk] = None # for debugging only
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
77 self._pastEnd = False
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
78 self._seenBindings: List[CandidateBinding] = [] # combined bindings (up to our ring) that we've returned
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
79
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
80 if ringlog.isEnabledFor(logging.DEBUG):
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
81 ringlog.debug('')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
82 msg = f'{INDENT*6} introducing {self!r}({self.lhsChunk}, {self._alignedMatches=})'
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
83 msg = msg.replace('AlignedRuleChunk', f'\n{INDENT*12}AlignedRuleChunk')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
84 ringlog.debug(msg)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
85
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
86 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
87
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
88 def _prevBindings(self) -> CandidateBinding:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
89 if not self.prev or self.prev.pastEnd():
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
90 return CandidateBinding({})
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
91
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
92 return self.prev.currentBinding()
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
93
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
94 def advance(self):
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
95 """update _current to a new set of valid bindings we haven't seen (since
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
96 last restart), or go into pastEnd mode. Note that _current is just our
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
97 contribution, but returned valid bindings include all prev rings."""
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
98 if self._pastEnd:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
99 raise NotImplementedError('need restart')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
100 ringlog.debug('')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
101 debug(ringlog, self.slog, f'{INDENT*6} --> {self}.advance start:')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
102
1693
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone
drewp@bigasterisk.com
parents: 1692
diff changeset
103 self._currentIsFromFunc = None
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
104 augmentedWorkingSet: List[AlignedRuleChunk] = []
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
105 if self.prev is None:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
106 augmentedWorkingSet = self._alignedMatches
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
107 else:
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
108 augmentedWorkingSet = list(applyChunky(self.prev.currentBinding(), self._alignedMatches))
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
109
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
110 if self._advanceWithPlainMatches(augmentedWorkingSet):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
111 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance finished with plain matches')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
112 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
113
1639
ae5ca4ba8954 rm unused computation
drewp@bigasterisk.com
parents: 1638
diff changeset
114 if self._advanceWithFunctions():
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
115 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance finished with function matches')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
116 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
117
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
118 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance had nothing and is now past end')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
119 self._pastEnd = True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
120
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
121 def _advanceWithPlainMatches(self, augmentedWorkingSet: List[AlignedRuleChunk]) -> bool:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
122 # if augmentedWorkingSet:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
123 # debug(ringlog, self.slog, f'{INDENT*7} {self} mines {len(augmentedWorkingSet)} matching augmented statements')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
124 # for s in augmentedWorkingSet:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
125 # debug(ringlog, self.slog, f'{INDENT*8} {s}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
126
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
127 for aligned in augmentedWorkingSet:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
128 try:
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
129 newBinding = aligned.newBindingIfMatched(self._prevBindings())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
130 except Inconsistent as exc:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
131 debug(ringlog, self.slog,
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
132 f'{INDENT*7} ChunkLooper{self._shortId} - {aligned} would be inconsistent with prev bindings ({exc})')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
133 continue
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
134
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
135 if self._testAndKeepNewBinding(newBinding, aligned.workingSetChunk):
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
136 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
137 return False
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
138
1639
ae5ca4ba8954 rm unused computation
drewp@bigasterisk.com
parents: 1638
diff changeset
139 def _advanceWithFunctions(self) -> bool:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
140 pred: Node = self.lhsChunk.predicate
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
141 if not isinstance(pred, URIRef):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
142 raise NotImplementedError
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
143
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
144 for functionType in functionsFor(pred):
1661
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
145 fn = functionType(self.lhsChunk)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
146 # debug(ringlog, self.slog, f'{INDENT*7} ChunkLooper{self._shortId} advanceWithFunctions, {functionType=}')
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
147
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
148 try:
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
149 log.debug(f'fn.bind {self._prevBindings()} ...')
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
150 #fullBinding = self._prevBindings().copy()
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
151 newBinding = fn.bind(self._prevBindings())
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
152 log.debug(f'...makes {newBinding=}')
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
153 except BindingUnknown:
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
154 pass
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
155 else:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
156 if newBinding is not None:
1693
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone
drewp@bigasterisk.com
parents: 1692
diff changeset
157 self._currentIsFromFunc = fn
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
158 if self._testAndKeepNewBinding(newBinding, self.lhsChunk):
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
159 return True
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
160
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
161 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
162
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
163 def _testAndKeepNewBinding(self, newBinding: CandidateBinding, sourceChunk: Chunk):
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
164 fullBinding: CandidateBinding = self._prevBindings().copy()
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
165 fullBinding.addNewBindings(newBinding)
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
166 isNew = fullBinding not in self._seenBindings
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
167 ringlog.debug(f'{INDENT*7} {self} considering {newBinding=} to make {fullBinding}. {isNew=}')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
168 # if self.slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
169 # self.slog.looperConsider(self, newBinding, fullBinding, isNew)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
170
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
171 if isNew:
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
172 self._seenBindings.append(fullBinding.copy())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
173 self._current = newBinding
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
174 self.currentSourceChunk = sourceChunk
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
175 return True
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
176 return False
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
177
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
178 def localBinding(self) -> CandidateBinding:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
179 if self.pastEnd():
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
180 raise NotImplementedError()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
181 return self._current
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
182
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
183 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
184 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
185 raise NotImplementedError()
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
186 together = self._prevBindings().copy()
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
187 together.addNewBindings(self._current)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
188 return together
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
189
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
190 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
191 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
192
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
193 def restart(self):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
194 try:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
195 self._pastEnd = False
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
196 self._seenBindings = []
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
197 self.advance()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
198 if self.pastEnd():
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
199 raise NoOptions()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
200 finally:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
201 debug(ringlog, self.slog, f'{INDENT*7} ChunkLooper{self._shortId} restarts: pastEnd={self.pastEnd()}')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
202
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
203
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
204 @dataclass
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
205 class Lhs:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
206 graph: ChunkedGraph # our full LHS graph, as input. See below for the statements partitioned into groups.
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
207
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
208 def __post_init__(self):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
209
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
210 self.myPreds = self.graph.allPredicatesExceptFunctions()
1602
e3c44ac6d3c5 do findEvals once at setRules time
drewp@bigasterisk.com
parents: 1601
diff changeset
211
1609
34f2817320cc new tests for a smaller part of the inner loop
drewp@bigasterisk.com
parents: 1608
diff changeset
212 def __repr__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
213 return f"Lhs({self.graph!r})"
1695
5c2565e63297 take out some misduided fixes
drewp@bigasterisk.com
parents: 1694
diff changeset
214
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
215 def findCandidateBindings(self, knownTrue: ChunkedGraph, stats, slog: Optional[StructuredLog],
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
216 ruleStatementsIterationLimit) -> Iterator['BoundLhs']:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
217 """distinct bindings that fit the LHS of a rule, using statements from
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
218 workingSet and functions from LHS"""
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
219 if not self.graph:
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
220 # special case- no LHS!
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
221 yield BoundLhs(self, CandidateBinding({}))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
222 return
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
223
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
224 if self._checkPredicateCounts(knownTrue):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
225 stats['_checkPredicateCountsCulls'] += 1
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
226 return
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
227
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
228 if not all(ch in knownTrue for ch in self.graph.staticChunks):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
229 stats['staticStmtCulls'] += 1
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
230 return
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
231 # After this point we don't need to consider self.graph.staticChunks.
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
232
1656
d8e5b2232474 bugfix for simple (no arg lists) functions not evaluating
drewp@bigasterisk.com
parents: 1653
diff changeset
233 if not self.graph.patternChunks and not self.graph.chunksUsedByFuncs:
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
234 # static only
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
235 yield BoundLhs(self, CandidateBinding({}))
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
236 return
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
237
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
238 log.debug('')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
239 try:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
240 chunkStack = self._assembleRings(knownTrue, stats, slog)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
241 except NoOptions:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
242 ringlog.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
243 return
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
244 log.debug('')
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
245 log.debug('')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
246 self._debugChunkStack('time to spin: initial odometer is', chunkStack)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
247
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
248 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
249 slog.say('time to spin')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
250 slog.odometer(chunkStack)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
251 self._assertAllRingsAreValid(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
252
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
253 lastRing = chunkStack[-1]
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
254 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
255 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
256 iterCount += 1
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
257 if iterCount > ruleStatementsIterationLimit:
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
258 raise ValueError('rule too complex')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
259
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
260 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
261
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
262 yield BoundLhs(self, lastRing.currentBinding())
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
263
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
264 # self._debugChunkStack('odometer', chunkStack)
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
265
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
266 done = self._advanceTheStack(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
267
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
268 self._debugChunkStack(f'odometer after ({done=})', chunkStack)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
269 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
270 slog.odometer(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
271
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
272 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
273 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
274 break
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
275
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
276 def _debugChunkStack(self, label: str, chunkStack: List[ChunkLooper]):
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
277 odolog.debug(f'{INDENT*4} {label}:')
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
278 for i, l in enumerate(chunkStack):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
279 odolog.debug(f'{INDENT*5} [{i}] {l} curbind={l.localBinding() if not l.pastEnd() else "<end>"}')
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
280
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
281 def _checkPredicateCounts(self, knownTrue):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
282 """raise NoOptions quickly in some cases"""
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
283
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
284 if self.graph.noPredicatesAppear(self.myPreds):
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
285 log.debug(f'{INDENT*3} checkPredicateCounts does cull because not all {self.myPreds=} are in knownTrue')
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
286 return True
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
287 log.debug(f'{INDENT*3} checkPredicateCounts does not cull because all {self.myPreds=} are in knownTrue')
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
288 return False
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
289
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
290 def _assembleRings(self, knownTrue: ChunkedGraph, stats, slog) -> List[ChunkLooper]:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
291 """make ChunkLooper for each stmt in our LHS graph, but do it in a way that they all
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
292 start out valid (or else raise NoOptions). static chunks have already been confirmed."""
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
293
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
294 log.debug(f'{INDENT*4} stats={dict(stats)}')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
295 odolog.debug(f'{INDENT*3} build new ChunkLooper stack')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
296 chunks = list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs))
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
297 chunks.sort(key=None)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
298 odolog.info(f' {INDENT*3} taking permutations of {len(chunks)=}')
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
299
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
300 permsTried = 0
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
301
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
302 for perm in self._partitionedGraphPermutations():
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
303 looperRings: List[ChunkLooper] = []
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
304 prev: Optional[ChunkLooper] = None
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
305 if odolog.isEnabledFor(logging.DEBUG):
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
306 odolog.debug(
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
307 f'{INDENT*4} [perm {permsTried}] try rule chunks in this order: {" THEN ".join(repr(p) for p in perm)}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
308
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
309 for ruleChunk in perm:
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
310 try:
1661
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
311 # These are getting rebuilt a lot which takes time. It would
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
312 # be nice if they could accept a changing `prev` order
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
313 # (which might already be ok).
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
314 looper = ChunkLooper(ruleChunk, prev, knownTrue, slog)
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
315 except NoOptions:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
316 odolog.debug(f'{INDENT*5} permutation didnt work, try another')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
317 break
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
318 looperRings.append(looper)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
319 prev = looperRings[-1]
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
320 else:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
321 # bug: At this point we've only shown that these are valid
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
322 # starting rings. The rules might be tricky enough that this
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
323 # permutation won't get us to the solution.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
324 return looperRings
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
325 if permsTried > 50000:
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
326 raise NotImplementedError(f'trying too many permutations {len(chunks)=}')
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
327 permsTried += 1
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
328
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
329 odolog.debug(f'{INDENT*5} no perms worked- rule cannot match anything')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
330 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
331
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
332 def _unpartitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
333 for perm in itertools.permutations(sorted(list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs)))):
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
334 yield perm
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
335
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
336 def _partitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
337 """always puts function chunks after pattern chunks
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
338
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
339 (and, if we cared, static chunks could go before that. Currently they're
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
340 culled out elsewhere, but that's done as a special case)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
341 """
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
342 tupleOfNoChunks: Tuple[Chunk, ...] = ()
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
343 pats = sorted(self.graph.patternChunks)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
344 funcs = sorted(self.graph.chunksUsedByFuncs)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
345 for patternPart in itertools.permutations(pats) if pats else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
346 for funcPart in itertools.permutations(funcs) if funcs else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
347 perm = patternPart + funcPart
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
348 yield perm
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
349
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
350 def _advanceTheStack(self, looperRings: List[ChunkLooper]) -> bool:
1696
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
351 toRestart: List[ChunkLooper] = []
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
352 pos = len(looperRings) - 1
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
353 while True:
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
354 looperRings[pos].advance()
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
355 if looperRings[pos].pastEnd():
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
356 if pos == 0:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
357 return True
1696
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
358 toRestart.append(looperRings[pos])
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
359 pos -= 1
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
360 else:
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
361 break
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
362 for ring in reversed(toRestart):
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
363 ring.restart()
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
364 return False
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
365
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
366 def _assertAllRingsAreValid(self, looperRings):
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
367 if any(ring.pastEnd() for ring in looperRings): # this is an unexpected debug assertion
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
368 odolog.warning(f'{INDENT*4} some rings started at pastEnd {looperRings}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
369 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
370
1592
d7b66234064b pure reordering of funcs to make the next diffs smaller
drewp@bigasterisk.com
parents: 1591
diff changeset
371
1622
38bd8ef9ef67 add CandidateTermMatches, unused so far
drewp@bigasterisk.com
parents: 1621
diff changeset
372 @dataclass
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
373 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
374 lhs: Lhs
1610
6fc48ef4c696 mysteriously lost an important line
drewp@bigasterisk.com
parents: 1609
diff changeset
375 binding: CandidateBinding
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
376
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
377
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
378 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
379 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
380 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
381 rhsGraph: Graph
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
382
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
383 def __post_init__(self):
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
384 self.lhs = Lhs(ChunkedGraph(self.lhsGraph, RuleUnboundBnode, functionsFor))
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
385
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
386 self.maps = {}
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
387
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
388 self.rhsGraphConvert: List[Triple] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
389 for s, p, o in self.rhsGraph:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
390 from rdflib import BNode
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
391 if isinstance(s, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
392 s = RhsBnode(s)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
393 if isinstance(p, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
394 p = RhsBnode(p)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
395 if isinstance(o, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
396 o = RhsBnode(o)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
397 self.rhsGraphConvert.append((s, p, o))
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
398
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
399 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict, slog: Optional[StructuredLog],
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
400 ruleStatementsIterationLimit):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
401 # this does not change for the current applyRule call. The rule will be
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
402 # tried again in an outer loop, in case it can produce more.
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
403 workingSetChunked = ChunkedGraph(workingSet, WorkingSetBnode, functionsFor)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
404
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
405 for bound in self.lhs.findCandidateBindings(workingSetChunked, stats, slog, ruleStatementsIterationLimit):
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
406 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
407 slog.foundBinding(bound)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
408 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
409
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
410 newStmts = self.generateImpliedFromRhs(bound.binding)
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
411
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
412 for newStmt in newStmts:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
413 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
414 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
415 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
416
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
417 def generateImpliedFromRhs(self, binding: CandidateBinding) -> List[Triple]:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
418
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
419 out: List[Triple] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
420
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
421 # Each time the RHS is used (in a rule firing), its own BNodes (which
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
422 # are subtype RhsBnode) need to be turned into distinct ones. Note that
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
423 # bnodes that come from the working set should not be remapped.
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
424 rhsBnodeMap: Dict[RhsBnode, WorkingSetBnode] = {}
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
425
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
426 # but, the iteration loop could come back with the same bindings again
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
427 key = binding.key()
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
428 rhsBnodeMap = self.maps.setdefault(key, {})
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
429
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
430 for stmt in binding.apply(self.rhsGraphConvert):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
431
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
432 outStmt: List[Node] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
433
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
434 for t in stmt:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
435 if isinstance(t, RhsBnode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
436 if t not in rhsBnodeMap:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
437 rhsBnodeMap[t] = WorkingSetBnode()
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
438 t = rhsBnodeMap[t]
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
439
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
440 outStmt.append(t)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
441
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
442 log.debug(f'{INDENT*6} rhs stmt {stmt} became {outStmt}')
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
443 out.append((outStmt[0], outStmt[1], outStmt[2]))
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
444
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
445 return out
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
446
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
447
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
448 @dataclass
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
449 class Inference:
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
450 rulesIterationLimit = 4
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
451 ruleStatementsIterationLimit = 5000
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
452
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
453 def __init__(self) -> None:
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
454 self.rules: List[Rule] = []
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
455 self._nonRuleStmts: List[Triple] = []
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
456
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
457 def setRules(self, g: ConjunctiveGraph):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
458 self.rules = []
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
459 self._nonRuleStmts = []
1599
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules
drewp@bigasterisk.com
parents: 1598
diff changeset
460 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules
drewp@bigasterisk.com
parents: 1598
diff changeset
461 if stmt[1] == LOG['implies']:
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
462 self.rules.append(Rule(stmt[0], stmt[2]))
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
463 else:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
464 self._nonRuleStmts.append(stmt)
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
465
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
466 def nonRuleStatements(self) -> List[Triple]:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
467 return self._nonRuleStmts
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
468
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
469 @INFER_CALLS.time()
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
470 def infer(self, graph: Graph, htmlLog: Optional[Path] = None):
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
471 """
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
472 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
473 """
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
474 n = graph.__len__()
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
475 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
476 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
477 startTime = time.time()
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
478 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
479
1589
5c1055be3c36 WIP more debugging, working towards bnode-matching support
drewp@bigasterisk.com
parents: 1588
diff changeset
480 # everything that is true: the input graph, plus every rule conclusion we can make
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
481 workingSet = Graph()
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
482 workingSet += self._nonRuleStmts
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
483 workingSet += graph
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
484
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
485 # just the statements that came from RHS's of rules that fired.
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
486 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
487
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
488 slog = StructuredLog(htmlLog) if htmlLog else None
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
489
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
490 rulesIterations = 0
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
491 delta = 1
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
492 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
493 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
494 slog.workingSet = workingSet
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
495
1674
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
496 while delta > 0:
1620
92f8deb59735 log layout
drewp@bigasterisk.com
parents: 1616
diff changeset
497 log.debug('')
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
498 log.info(f'{INDENT*1}*iteration {rulesIterations}')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
499 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
500 slog.startIteration(rulesIterations)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
501
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
502 delta = -len(implied)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
503 self._iterateAllRules(workingSet, implied, stats, slog)
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
504 delta += len(implied)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
505 rulesIterations += 1
1597
387a9cb66517 logging adjustments
drewp@bigasterisk.com
parents: 1596
diff changeset
506 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1674
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
507 if rulesIterations >= self.rulesIterationLimit:
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
508 raise ValueError(f"rule too complex after {rulesIterations=}")
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
509 stats['iterations'] = rulesIterations
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
510 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
511 stats['impliedStmts'] = len(implied)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
512 log.info(f'{INDENT*0} Inference done {dict(stats)}.')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
513 log.debug('Implied:')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
514 log.debug(graphDump(implied))
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
515
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
516 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
517 slog.render()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
518 log.info(f'wrote {htmlLog}')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
519
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
520 return implied
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
521
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
522 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats, slog: Optional[StructuredLog]):
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
523 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
524 self._logRuleApplicationHeader(workingSet, i, rule)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
525 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
526 slog.rule(workingSet, i, rule)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
527 rule.applyRule(workingSet, implied, stats, slog, self.ruleStatementsIterationLimit)
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
528
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
529 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
530 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
531 return
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
532
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
533 log.debug('')
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
534 log.debug(f'{INDENT*2} workingSet:')
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
535 # for j, stmt in enumerate(sorted(workingSet)):
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
536 # log.debug(f'{INDENT*3} ({j}) {stmt}')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
537 log.debug(f'{INDENT*3} {graphDump(workingSet, oneLine=False)}')
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
538
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
539 log.debug('')
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
540 log.debug(f'{INDENT*2}-applying rule {i}')
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
541 log.debug(f'{INDENT*3} rule def lhs:')
1677
aa35ae7a1acc add new bug test (no fix yet)
drewp@bigasterisk.com
parents: 1674
diff changeset
542 for stmt in sorted(r.lhs.graph.allChunks()):
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
543 log.debug(f'{INDENT*4} {stmt}')
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
544 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')