Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1697:88f6e9bf69d1

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
stats and non-debug mode speedups
author drewp@bigasterisk.com
date Tue, 28 Sep 2021 00:32:16 -0700
parents cdf706cf5f82
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
1 """
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
4 """
1588
0757fafbfdab WIP inferencer - partial var and function support
drewp@bigasterisk.com
parents: 1587
diff changeset
5 import itertools
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
6 import logging
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
7 import time
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
8 from collections import defaultdict
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
9 from dataclasses import dataclass
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
10 from typing import Dict, Iterator, List, Optional, Tuple, Union, cast
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
11 from pathlib import Path
1588
0757fafbfdab WIP inferencer - partial var and function support
drewp@bigasterisk.com
parents: 1587
diff changeset
12
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
13 from prometheus_client import Histogram, Summary
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
14 from rdflib import Graph, Namespace
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
15 from rdflib.graph import ConjunctiveGraph
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
16 from rdflib.term import Node, URIRef, Variable
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
17
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
18 from candidate_binding import CandidateBinding
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
19 from inference_types import (BindingUnknown, Inconsistent, RhsBnode, RuleUnboundBnode, Triple, WorkingSetBnode)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
20 from lhs_evaluation import functionsFor
1650
2061df259224 move graphDump (on its way out, since reprs are getting better)
drewp@bigasterisk.com
parents: 1649
diff changeset
21 from rdf_debug import graphDump
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
22 from stmt_chunk import AlignedRuleChunk, Chunk, ChunkedGraph, applyChunky
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
23 from structured_log import StructuredLog
1605
449746d1598f WIP move evaluation to new file
drewp@bigasterisk.com
parents: 1603
diff changeset
24
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
25 log = logging.getLogger('infer')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
26 odolog = logging.getLogger('infer.odo') # the "odometer" logic
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
27 ringlog = logging.getLogger('infer.ring') # for ChunkLooper
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
28
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
29 INDENT = ' '
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
30
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
31 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
32 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
33
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
34 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
35 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
36 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
37
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
38
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
39 class NoOptions(ValueError):
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
40 """ChunkLooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
41
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
42
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
43 def debug(logger, slog: Optional[StructuredLog], msg):
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
44 logger.debug(msg)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
45 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
46 slog.say(msg)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
47
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
48
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
49 _chunkLooperShortId = itertools.count()
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
50
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
51
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
52 @dataclass
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
53 class ChunkLooper:
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
54 """given one LHS Chunk, iterate through the possible matches for it,
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
55 returning what bindings they would imply. Only distinct bindings are
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
56 returned. The bindings build on any `prev` ChunkLooper's results.
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
57
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
58 In the odometer metaphor used below, this is one of the rings.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
59
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
60 This iterator is restartable."""
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
61 lhsChunk: Chunk
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
62 prev: Optional['ChunkLooper']
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
63 workingSet: 'ChunkedGraph'
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
64 slog: Optional[StructuredLog]
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
65
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
66 def __repr__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
67 return f'{self.__class__.__name__}{self._shortId}{"<pastEnd>" if self.pastEnd() else ""}'
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
68
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
69 def __post_init__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
70 self._shortId = next(_chunkLooperShortId)
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
71 self._alignedMatches = list(self.lhsChunk.ruleMatchesFrom(self.workingSet))
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
72 del self.workingSet
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
73
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
74 # only ours- do not store prev, since it could change without us
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
75 self._current = CandidateBinding({})
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
76 self.currentSourceChunk: Optional[Chunk] = None # for debugging only
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
77 self._pastEnd = False
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
78 self._seenBindings: List[CandidateBinding] = [] # combined bindings (up to our ring) that we've returned
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
79
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
80 if ringlog.isEnabledFor(logging.DEBUG):
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
81 ringlog.debug('')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
82 msg = f'{INDENT*6} introducing {self!r}({self.lhsChunk}, {self._alignedMatches=})'
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
83 msg = msg.replace('AlignedRuleChunk', f'\n{INDENT*12}AlignedRuleChunk')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
84 ringlog.debug(msg)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
85
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
86 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
87
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
88 def _prevBindings(self) -> CandidateBinding:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
89 if not self.prev or self.prev.pastEnd():
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
90 return CandidateBinding({})
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
91
1635
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts
drewp@bigasterisk.com
parents: 1634
diff changeset
92 return self.prev.currentBinding()
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
93
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
94 def advance(self):
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
95 """update _current to a new set of valid bindings we haven't seen (since
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
96 last restart), or go into pastEnd mode. Note that _current is just our
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
97 contribution, but returned valid bindings include all prev rings."""
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
98 if self._pastEnd:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
99 raise NotImplementedError('need restart')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
100 ringlog.debug('')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
101 debug(ringlog, self.slog, f'{INDENT*6} --> {self}.advance start:')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
102
1693
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone
drewp@bigasterisk.com
parents: 1692
diff changeset
103 self._currentIsFromFunc = None
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
104 augmentedWorkingSet: List[AlignedRuleChunk] = []
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
105 if self.prev is None:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
106 augmentedWorkingSet = self._alignedMatches
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
107 else:
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
108 augmentedWorkingSet = list(applyChunky(self.prev.currentBinding(), self._alignedMatches))
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
109
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
110 if self._advanceWithPlainMatches(augmentedWorkingSet):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
111 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance finished with plain matches')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
112 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
113
1639
ae5ca4ba8954 rm unused computation
drewp@bigasterisk.com
parents: 1638
diff changeset
114 if self._advanceWithFunctions():
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
115 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance finished with function matches')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
116 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
117
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
118 debug(ringlog, self.slog, f'{INDENT*6} <-- {self}.advance had nothing and is now past end')
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
119 self._pastEnd = True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
120
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
121 def _advanceWithPlainMatches(self, augmentedWorkingSet: List[AlignedRuleChunk]) -> bool:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
122 # if augmentedWorkingSet:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
123 # debug(ringlog, self.slog, f'{INDENT*7} {self} mines {len(augmentedWorkingSet)} matching augmented statements')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
124 # for s in augmentedWorkingSet:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
125 # debug(ringlog, self.slog, f'{INDENT*8} {s}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
126
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
127 for aligned in augmentedWorkingSet:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
128 try:
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
129 newBinding = aligned.newBindingIfMatched(self._prevBindings())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
130 except Inconsistent as exc:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
131 debug(ringlog, self.slog,
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
132 f'{INDENT*7} ChunkLooper{self._shortId} - {aligned} would be inconsistent with prev bindings ({exc})')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
133 continue
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
134
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
135 if self._testAndKeepNewBinding(newBinding, aligned.workingSetChunk):
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
136 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
137 return False
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
138
1639
ae5ca4ba8954 rm unused computation
drewp@bigasterisk.com
parents: 1638
diff changeset
139 def _advanceWithFunctions(self) -> bool:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
140 pred: Node = self.lhsChunk.predicate
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
141 if not isinstance(pred, URIRef):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
142 raise NotImplementedError
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
143
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
144 for functionType in functionsFor(pred):
1661
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
145 fn = functionType(self.lhsChunk)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
146 # debug(ringlog, self.slog, f'{INDENT*7} ChunkLooper{self._shortId} advanceWithFunctions, {functionType=}')
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
147
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
148 try:
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
149 log.debug(f'fn.bind {self._prevBindings()} ...')
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
150 #fullBinding = self._prevBindings().copy()
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
151 newBinding = fn.bind(self._prevBindings())
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
152 log.debug(f'...makes {newBinding=}')
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
153 except BindingUnknown:
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
154 pass
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
155 else:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
156 if newBinding is not None:
1693
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone
drewp@bigasterisk.com
parents: 1692
diff changeset
157 self._currentIsFromFunc = fn
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
158 if self._testAndKeepNewBinding(newBinding, self.lhsChunk):
1637
ec3f98d0c1d8 refactor rules eval
drewp@bigasterisk.com
parents: 1636
diff changeset
159 return True
1634
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
160
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
161 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers
drewp@bigasterisk.com
parents: 1633
diff changeset
162
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
163 def _testAndKeepNewBinding(self, newBinding: CandidateBinding, sourceChunk: Chunk):
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
164 fullBinding: CandidateBinding = self._prevBindings().copy()
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
165 fullBinding.addNewBindings(newBinding)
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
166 isNew = fullBinding not in self._seenBindings
1697
88f6e9bf69d1 stats and non-debug mode speedups
drewp@bigasterisk.com
parents: 1696
diff changeset
167
88f6e9bf69d1 stats and non-debug mode speedups
drewp@bigasterisk.com
parents: 1696
diff changeset
168 if ringlog.isEnabledFor(logging.DEBUG):
88f6e9bf69d1 stats and non-debug mode speedups
drewp@bigasterisk.com
parents: 1696
diff changeset
169 ringlog.debug(f'{INDENT*7} {self} considering {newBinding=} to make {fullBinding}. {isNew=}')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
170 # if self.slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
171 # self.slog.looperConsider(self, newBinding, fullBinding, isNew)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
172
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
173 if isNew:
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
174 self._seenBindings.append(fullBinding.copy())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
175 self._current = newBinding
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
176 self.currentSourceChunk = sourceChunk
1668
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
177 return True
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
178 return False
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos
drewp@bigasterisk.com
parents: 1667
diff changeset
179
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
180 def localBinding(self) -> CandidateBinding:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
181 if self.pastEnd():
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
182 raise NotImplementedError()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
183 return self._current
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
184
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
185 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
186 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
187 raise NotImplementedError()
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
188 together = self._prevBindings().copy()
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
189 together.addNewBindings(self._current)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
190 return together
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
191
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
192 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
193 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
194
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
195 def restart(self):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
196 try:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
197 self._pastEnd = False
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
198 self._seenBindings = []
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
199 self.advance()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
200 if self.pastEnd():
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
201 raise NoOptions()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
202 finally:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
203 debug(ringlog, self.slog, f'{INDENT*7} ChunkLooper{self._shortId} restarts: pastEnd={self.pastEnd()}')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
204
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
205
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
206 @dataclass
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
207 class Lhs:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
208 graph: ChunkedGraph # our full LHS graph, as input. See below for the statements partitioned into groups.
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
209
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
210 def __post_init__(self):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
211
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
212 self.myPreds = self.graph.allPredicatesExceptFunctions()
1602
e3c44ac6d3c5 do findEvals once at setRules time
drewp@bigasterisk.com
parents: 1601
diff changeset
213
1609
34f2817320cc new tests for a smaller part of the inner loop
drewp@bigasterisk.com
parents: 1608
diff changeset
214 def __repr__(self):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
215 return f"Lhs({self.graph!r})"
1695
5c2565e63297 take out some misduided fixes
drewp@bigasterisk.com
parents: 1694
diff changeset
216
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
217 def findCandidateBindings(self, knownTrue: ChunkedGraph, stats, slog: Optional[StructuredLog],
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
218 ruleStatementsIterationLimit) -> Iterator['BoundLhs']:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
219 """distinct bindings that fit the LHS of a rule, using statements from
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
220 workingSet and functions from LHS"""
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
221 if not self.graph:
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
222 # special case- no LHS!
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
223 yield BoundLhs(self, CandidateBinding({}))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
224 return
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
225
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
226 if self._checkPredicateCounts(knownTrue):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
227 stats['_checkPredicateCountsCulls'] += 1
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
228 return
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
229
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
230 if not all(ch in knownTrue for ch in self.graph.staticChunks):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
231 stats['staticStmtCulls'] += 1
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
232 return
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
233 # After this point we don't need to consider self.graph.staticChunks.
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
234
1656
d8e5b2232474 bugfix for simple (no arg lists) functions not evaluating
drewp@bigasterisk.com
parents: 1653
diff changeset
235 if not self.graph.patternChunks and not self.graph.chunksUsedByFuncs:
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
236 # static only
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
237 yield BoundLhs(self, CandidateBinding({}))
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
238 return
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
239
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
240 log.debug('')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
241 try:
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
242 chunkStack = self._assembleRings(knownTrue, stats, slog)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
243 except NoOptions:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
244 ringlog.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
245 return
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
246 log.debug('')
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
247 log.debug('')
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
248 self._debugChunkStack('time to spin: initial odometer is', chunkStack)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
249
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
250 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
251 slog.say('time to spin')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
252 slog.odometer(chunkStack)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
253 self._assertAllRingsAreValid(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
254
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
255 lastRing = chunkStack[-1]
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
256 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
257 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
258 iterCount += 1
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
259 if iterCount > ruleStatementsIterationLimit:
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
260 raise ValueError('rule too complex')
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
261
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
262 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
263
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
264 yield BoundLhs(self, lastRing.currentBinding())
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
265
1667
a2347393b43e comments, debug, dead code
drewp@bigasterisk.com
parents: 1664
diff changeset
266 # self._debugChunkStack('odometer', chunkStack)
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
267
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
268 done = self._advanceTheStack(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
269
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
270 self._debugChunkStack(f'odometer after ({done=})', chunkStack)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
271 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
272 slog.odometer(chunkStack)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
273
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
274 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
275 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
276 break
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
277
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
278 def _debugChunkStack(self, label: str, chunkStack: List[ChunkLooper]):
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
279 odolog.debug(f'{INDENT*4} {label}:')
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
280 for i, l in enumerate(chunkStack):
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
281 odolog.debug(f'{INDENT*5} [{i}] {l} curbind={l.localBinding() if not l.pastEnd() else "<end>"}')
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
282
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
283 def _checkPredicateCounts(self, knownTrue):
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
284 """raise NoOptions quickly in some cases"""
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
285
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
286 if self.graph.noPredicatesAppear(self.myPreds):
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
287 log.debug(f'{INDENT*3} checkPredicateCounts does cull because not all {self.myPreds=} are in knownTrue')
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
288 return True
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
289 log.debug(f'{INDENT*3} checkPredicateCounts does not cull because all {self.myPreds=} are in knownTrue')
1640
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
290 return False
4bb6f593ebf3 speedups: abort some rules faster
drewp@bigasterisk.com
parents: 1639
diff changeset
291
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
292 def _assembleRings(self, knownTrue: ChunkedGraph, stats, slog) -> List[ChunkLooper]:
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
293 """make ChunkLooper for each stmt in our LHS graph, but do it in a way that they all
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
294 start out valid (or else raise NoOptions). static chunks have already been confirmed."""
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
295
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
296 log.debug(f'{INDENT*4} stats={dict(stats)}')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
297 odolog.debug(f'{INDENT*3} build new ChunkLooper stack')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
298 chunks = list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs))
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
299 chunks.sort(key=None)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
300 odolog.info(f' {INDENT*3} taking permutations of {len(chunks)=}')
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
301
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
302 permsTried = 0
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
303
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
304 for perm in self._partitionedGraphPermutations():
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
305 looperRings: List[ChunkLooper] = []
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
306 prev: Optional[ChunkLooper] = None
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
307 if odolog.isEnabledFor(logging.DEBUG):
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
308 odolog.debug(
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
309 f'{INDENT*4} [perm {permsTried}] try rule chunks in this order: {" THEN ".join(repr(p) for p in perm)}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
310
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
311 for ruleChunk in perm:
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
312 try:
1661
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
313 # These are getting rebuilt a lot which takes time. It would
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
314 # be nice if they could accept a changing `prev` order
00a5624d1d14 cleanups and optimizations
drewp@bigasterisk.com
parents: 1656
diff changeset
315 # (which might already be ok).
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
316 looper = ChunkLooper(ruleChunk, prev, knownTrue, slog)
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
317 except NoOptions:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
318 odolog.debug(f'{INDENT*5} permutation didnt work, try another')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
319 break
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
320 looperRings.append(looper)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
321 prev = looperRings[-1]
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
322 else:
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
323 # bug: At this point we've only shown that these are valid
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
324 # starting rings. The rules might be tricky enough that this
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
325 # permutation won't get us to the solution.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
326 return looperRings
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
327 if permsTried > 50000:
1652
dddfa09ea0b9 debug logging and comments
drewp@bigasterisk.com
parents: 1651
diff changeset
328 raise NotImplementedError(f'trying too many permutations {len(chunks)=}')
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
329 permsTried += 1
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
330
1697
88f6e9bf69d1 stats and non-debug mode speedups
drewp@bigasterisk.com
parents: 1696
diff changeset
331 stats['permsTried'] += permsTried
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
332 odolog.debug(f'{INDENT*5} no perms worked- rule cannot match anything')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
333 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
334
1692
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
335 def _unpartitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
336 for perm in itertools.permutations(sorted(list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs)))):
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
337 yield perm
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
338
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
339 def _partitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
340 """always puts function chunks after pattern chunks
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
341
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
342 (and, if we cared, static chunks could go before that. Currently they're
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
343 culled out elsewhere, but that's done as a special case)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
344 """
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
345 tupleOfNoChunks: Tuple[Chunk, ...] = ()
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
346 pats = sorted(self.graph.patternChunks)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
347 funcs = sorted(self.graph.chunksUsedByFuncs)
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
348 for patternPart in itertools.permutations(pats) if pats else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
349 for funcPart in itertools.permutations(funcs) if funcs else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
350 perm = patternPart + funcPart
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
351 yield perm
2883da14847c debugging and cleanup, as i looked for a bug
drewp@bigasterisk.com
parents: 1677
diff changeset
352
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
353 def _advanceTheStack(self, looperRings: List[ChunkLooper]) -> bool:
1696
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
354 toRestart: List[ChunkLooper] = []
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
355 pos = len(looperRings) - 1
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
356 while True:
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
357 looperRings[pos].advance()
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
358 if looperRings[pos].pastEnd():
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
359 if pos == 0:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
360 return True
1696
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
361 toRestart.append(looperRings[pos])
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
362 pos -= 1
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
363 else:
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
364 break
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
365 for ring in reversed(toRestart):
cdf706cf5f82 fix a test (odometer rollover direction)
drewp@bigasterisk.com
parents: 1695
diff changeset
366 ring.restart()
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
367 return False
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
368
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
369 def _assertAllRingsAreValid(self, looperRings):
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
370 if any(ring.pastEnd() for ring in looperRings): # this is an unexpected debug assertion
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
371 odolog.warning(f'{INDENT*4} some rings started at pastEnd {looperRings}')
1633
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
372 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm
drewp@bigasterisk.com
parents: 1632
diff changeset
373
1592
d7b66234064b pure reordering of funcs to make the next diffs smaller
drewp@bigasterisk.com
parents: 1591
diff changeset
374
1622
38bd8ef9ef67 add CandidateTermMatches, unused so far
drewp@bigasterisk.com
parents: 1621
diff changeset
375 @dataclass
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
376 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
377 lhs: Lhs
1610
6fc48ef4c696 mysteriously lost an important line
drewp@bigasterisk.com
parents: 1609
diff changeset
378 binding: CandidateBinding
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
379
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
380
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
381 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
382 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
383 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
384 rhsGraph: Graph
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
385
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
386 def __post_init__(self):
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
387 self.lhs = Lhs(ChunkedGraph(self.lhsGraph, RuleUnboundBnode, functionsFor))
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
388
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
389 self.maps = {}
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
390
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
391 self.rhsGraphConvert: List[Triple] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
392 for s, p, o in self.rhsGraph:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
393 from rdflib import BNode
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
394 if isinstance(s, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
395 s = RhsBnode(s)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
396 if isinstance(p, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
397 p = RhsBnode(p)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
398 if isinstance(o, BNode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
399 o = RhsBnode(o)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
400 self.rhsGraphConvert.append((s, p, o))
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
401
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
402 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict, slog: Optional[StructuredLog],
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
403 ruleStatementsIterationLimit):
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
404 # this does not change for the current applyRule call. The rule will be
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
405 # tried again in an outer loop, in case it can produce more.
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
406 workingSetChunked = ChunkedGraph(workingSet, WorkingSetBnode, functionsFor)
1651
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples
drewp@bigasterisk.com
parents: 1650
diff changeset
407
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
408 for bound in self.lhs.findCandidateBindings(workingSetChunked, stats, slog, ruleStatementsIterationLimit):
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
409 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
410 slog.foundBinding(bound)
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
411 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
412
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
413 newStmts = self.generateImpliedFromRhs(bound.binding)
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
414
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
415 for newStmt in newStmts:
1631
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over
drewp@bigasterisk.com
parents: 1627
diff changeset
416 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
417 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
418 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
419
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
420 def generateImpliedFromRhs(self, binding: CandidateBinding) -> List[Triple]:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
421
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
422 out: List[Triple] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
423
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
424 # Each time the RHS is used (in a rule firing), its own BNodes (which
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
425 # are subtype RhsBnode) need to be turned into distinct ones. Note that
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
426 # bnodes that come from the working set should not be remapped.
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
427 rhsBnodeMap: Dict[RhsBnode, WorkingSetBnode] = {}
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
428
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
429 # but, the iteration loop could come back with the same bindings again
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
430 key = binding.key()
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
431 rhsBnodeMap = self.maps.setdefault(key, {})
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
432
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
433 for stmt in binding.apply(self.rhsGraphConvert):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
434
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
435 outStmt: List[Node] = []
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
436
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
437 for t in stmt:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
438 if isinstance(t, RhsBnode):
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
439 if t not in rhsBnodeMap:
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
440 rhsBnodeMap[t] = WorkingSetBnode()
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
441 t = rhsBnodeMap[t]
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
442
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
443 outStmt.append(t)
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
444
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
445 log.debug(f'{INDENT*6} rhs stmt {stmt} became {outStmt}')
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
446 out.append((outStmt[0], outStmt[1], outStmt[2]))
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
447
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
448 return out
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
449
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
450
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
451 @dataclass
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
452 class Inference:
1673
80f4e741ca4f redo RHS bnode processing
drewp@bigasterisk.com
parents: 1669
diff changeset
453 rulesIterationLimit = 4
1664
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times
drewp@bigasterisk.com
parents: 1661
diff changeset
454 ruleStatementsIterationLimit = 5000
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
455
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
456 def __init__(self) -> None:
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
457 self.rules: List[Rule] = []
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
458 self._nonRuleStmts: List[Triple] = []
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
459
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
460 def setRules(self, g: ConjunctiveGraph):
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
461 self.rules = []
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
462 self._nonRuleStmts = []
1599
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules
drewp@bigasterisk.com
parents: 1598
diff changeset
463 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules
drewp@bigasterisk.com
parents: 1598
diff changeset
464 if stmt[1] == LOG['implies']:
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
465 self.rules.append(Rule(stmt[0], stmt[2]))
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
466 else:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
467 self._nonRuleStmts.append(stmt)
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
468
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
469 def nonRuleStatements(self) -> List[Triple]:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
470 return self._nonRuleStmts
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
471
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
472 @INFER_CALLS.time()
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
473 def infer(self, graph: Graph, htmlLog: Optional[Path] = None):
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
474 """
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
475 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
476 """
1626
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
477 n = graph.__len__()
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
478 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes
drewp@bigasterisk.com
parents: 1623
diff changeset
479 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601
30463df12d89 infer() dumps stats
drewp@bigasterisk.com
parents: 1600
diff changeset
480 startTime = time.time()
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
481 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
482
1589
5c1055be3c36 WIP more debugging, working towards bnode-matching support
drewp@bigasterisk.com
parents: 1588
diff changeset
483 # everything that is true: the input graph, plus every rule conclusion we can make
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
484 workingSet = Graph()
1649
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit
drewp@bigasterisk.com
parents: 1648
diff changeset
485 workingSet += self._nonRuleStmts
1593
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing
drewp@bigasterisk.com
parents: 1592
diff changeset
486 workingSet += graph
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
487
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
488 # just the statements that came from RHS's of rules that fired.
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
489 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
490
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
491 slog = StructuredLog(htmlLog) if htmlLog else None
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
492
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
493 rulesIterations = 0
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
494 delta = 1
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
495 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
496 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
497 slog.workingSet = workingSet
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
498
1674
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
499 while delta > 0:
1620
92f8deb59735 log layout
drewp@bigasterisk.com
parents: 1616
diff changeset
500 log.debug('')
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
501 log.info(f'{INDENT*1}*iteration {rulesIterations}')
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
502 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
503 slog.startIteration(rulesIterations)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
504
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
505 delta = -len(implied)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
506 self._iterateAllRules(workingSet, implied, stats, slog)
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
507 delta += len(implied)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
508 rulesIterations += 1
1597
387a9cb66517 logging adjustments
drewp@bigasterisk.com
parents: 1596
diff changeset
509 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1674
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
510 if rulesIterations >= self.rulesIterationLimit:
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached
drewp@bigasterisk.com
parents: 1673
diff changeset
511 raise ValueError(f"rule too complex after {rulesIterations=}")
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
512 stats['iterations'] = rulesIterations
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
513 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
514 stats['impliedStmts'] = len(implied)
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
515 log.info(f'{INDENT*0} Inference done {dict(stats)}.')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
516 log.debug('Implied:')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
517 log.debug(graphDump(implied))
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
518
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
519 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
520 slog.render()
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
521 log.info(f'wrote {htmlLog}')
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
522
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
523 return implied
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
524
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
525 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats, slog: Optional[StructuredLog]):
1612
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
526 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups.
drewp@bigasterisk.com
parents: 1611
diff changeset
527 self._logRuleApplicationHeader(workingSet, i, rule)
1694
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
528 if slog:
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
529 slog.rule(workingSet, i, rule)
73abfd4cf5d0 new html log and other refactoring as i work on the advanceTheStack problems
drewp@bigasterisk.com
parents: 1693
diff changeset
530 rule.applyRule(workingSet, implied, stats, slog, self.ruleStatementsIterationLimit)
1587
9a3a18c494f9 WIP new inferencer. no vars yet.
drewp@bigasterisk.com
parents:
diff changeset
531
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
532 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
533 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
534 return
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
535
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
536 log.debug('')
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
537 log.debug(f'{INDENT*2} workingSet:')
1648
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
538 # for j, stmt in enumerate(sorted(workingSet)):
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
539 # log.debug(f'{INDENT*3} ({j}) {stmt}')
3059f31b2dfa more performance work
drewp@bigasterisk.com
parents: 1641
diff changeset
540 log.debug(f'{INDENT*3} {graphDump(workingSet, oneLine=False)}')
1594
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
541
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
542 log.debug('')
e58bcfa66093 cleanups and a few fixed cases
drewp@bigasterisk.com
parents: 1593
diff changeset
543 log.debug(f'{INDENT*2}-applying rule {i}')
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
544 log.debug(f'{INDENT*3} rule def lhs:')
1677
aa35ae7a1acc add new bug test (no fix yet)
drewp@bigasterisk.com
parents: 1674
diff changeset
545 for stmt in sorted(r.lhs.graph.allChunks()):
1632
bd79a2941cab just (a lot of) debug changes
drewp@bigasterisk.com
parents: 1631
diff changeset
546 log.debug(f'{INDENT*4} {stmt}')
1607
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs
drewp@bigasterisk.com
parents: 1605
diff changeset
547 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')