homeauto: service/mqtt_to_rdf/inference.py annotate

annotate service/mqtt_to_rdf/inference.py @ 1693:0455a1e18e4f

really rough fix to a broken test. hopefully this gets redone

author	drewp@bigasterisk.com
date	Sat, 25 Sep 2021 22:20:42 -0700
parents	2883da14847c
children	73abfd4cf5d0

rev	line source
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	1 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	4 """
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	5 import itertools
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	6 import logging
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	7 import time
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	8 from collections import defaultdict
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	9 from dataclasses import dataclass
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	10 from typing import Dict, Iterator, List, Optional, Tuple, Union, cast
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	11
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	12 from prometheus_client import Histogram, Summary
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	13 from rdflib import Graph, Namespace
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	14 from rdflib.graph import ConjunctiveGraph
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	15 from rdflib.term import Node, URIRef, Variable
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	16
1638 0ba1625037ae don't crash, just skip the rule if there's a BindingConflict (no clear test case yet) drewp@bigasterisk.com parents: 1637 diff changeset	17 from candidate_binding import BindingConflict, CandidateBinding
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	18 from inference_types import (BindingUnknown, Inconsistent, RhsBnode, RuleOutBnode, RuleUnboundBnode, Triple, WorkingSetBnode)
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	19 from lhs_evaluation import functionsFor
1650 2061df259224 move graphDump (on its way out, since reprs are getting better) drewp@bigasterisk.com parents: 1649 diff changeset	20 from rdf_debug import graphDump
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	21 from stmt_chunk import AlignedRuleChunk, Chunk, ChunkedGraph, applyChunky
1605 449746d1598f WIP move evaluation to new file drewp@bigasterisk.com parents: 1603 diff changeset	22
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	23 log = logging.getLogger('infer')
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	24 odolog = logging.getLogger('infer.odo') # the "odometer" logic
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	25 ringlog = logging.getLogger('infer.ring') # for ChunkLooper
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	26
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	27 INDENT = ' '
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	28
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	29 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	30 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	31
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	32 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	33 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	34 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	35
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	36
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	37 class NoOptions(ValueError):
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	38 """ChunkLooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	39
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	40
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	41 _chunkLooperShortId = itertools.count()
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	42
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	43
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	44 @dataclass
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	45 class ChunkLooper:
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	46 """given one LHS Chunk, iterate through the possible matches for it,
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	47 returning what bindings they would imply. Only distinct bindings are
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	48 returned. The bindings build on any `prev` ChunkLooper's results.
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	49
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	50 In the odometer metaphor used below, this is one of the rings.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	51
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	52 This iterator is restartable."""
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	53 lhsChunk: Chunk
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	54 prev: Optional['ChunkLooper']
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	55 workingSet: 'ChunkedGraph'
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	56
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	57 def __repr__(self):
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	58 return f'{self.__class__.__name__}{self._shortId}{"<pastEnd>" if self.pastEnd() else ""}'
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	59
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	60 def __post_init__(self):
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	61 self._shortId = next(_chunkLooperShortId)
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	62 self._alignedMatches = list(self.lhsChunk.ruleMatchesFrom(self.workingSet))
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	63
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	64 # only ours- do not store prev, since it could change without us
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	65 self._current = CandidateBinding({})
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	66 self._pastEnd = False
1667 a2347393b43e comments, debug, dead code drewp@bigasterisk.com parents: 1664 diff changeset	67 self._seenBindings: List[CandidateBinding] = [] # combined bindings (up to our ring) that we've returned
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	68
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	69 if ringlog.isEnabledFor(logging.DEBUG):
1667 a2347393b43e comments, debug, dead code drewp@bigasterisk.com parents: 1664 diff changeset	70 ringlog.debug('')
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	71 ringlog.debug(f'{INDENT*6} introducing {self!r}({self.lhsChunk}, {self._alignedMatches=})')
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	72
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	73 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	74
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	75 def _prevBindings(self) -> CandidateBinding:
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	76 if not self.prev or self.prev.pastEnd():
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	77 return CandidateBinding({})
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	78
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	79 return self.prev.currentBinding()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	80
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	81 def advance(self):
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	82 """update _current to a new set of valid bindings we haven't seen (since
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	83 last restart), or go into pastEnd mode. Note that _current is just our
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	84 contribution, but returned valid bindings include all prev rings."""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	85 if self._pastEnd:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	86 raise NotImplementedError('need restart')
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	87 ringlog.debug('')
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	88 self._currentIsFromFunc = None
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	89 augmentedWorkingSet: List[AlignedRuleChunk] = []
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	90 if self.prev is None:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	91 augmentedWorkingSet = self._alignedMatches
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	92 else:
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	93 augmentedWorkingSet = list(applyChunky(self.prev.currentBinding(), self._alignedMatches))
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	94
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	95 if self._advanceWithPlainMatches(augmentedWorkingSet):
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	96 ringlog.debug(f'{INDENT*6} <-- {self}.advance finished with plain matches')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	97 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	98
1639 ae5ca4ba8954 rm unused computation drewp@bigasterisk.com parents: 1638 diff changeset	99 if self._advanceWithFunctions():
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	100 ringlog.debug(f'{INDENT*6} <-- {self}.advance finished with function matches')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	101 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	102
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	103 ringlog.debug(f'{INDENT*6} <-- {self}.advance had nothing and is now past end')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	104 self._pastEnd = True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	105
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	106 def _advanceWithPlainMatches(self, augmentedWorkingSet: List[AlignedRuleChunk]) -> bool:
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	107 ringlog.debug(f'{INDENT*7} {self} mines {len(augmentedWorkingSet)} matching augmented statements')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	108 for s in augmentedWorkingSet:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	109 ringlog.debug(f'{INDENT*8} {s}')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	110
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	111 for aligned in augmentedWorkingSet:
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	112 try:
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	113 newBinding = aligned.newBindingIfMatched(self._prevBindings())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	114 except Inconsistent as exc:
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	115 ringlog.debug(
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	116 f'{INDENT*7} ChunkLooper{self._shortId} - {aligned} would be inconsistent with prev bindings ({exc})')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	117 continue
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	118
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	119 if self._testAndKeepNewBinding(newBinding):
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	120 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	121 return False
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	122
1639 ae5ca4ba8954 rm unused computation drewp@bigasterisk.com parents: 1638 diff changeset	123 def _advanceWithFunctions(self) -> bool:
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	124 pred: Node = self.lhsChunk.predicate
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	125 if not isinstance(pred, URIRef):
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	126 raise NotImplementedError
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	127
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	128 ringlog.debug(f'{INDENT*6} advanceWithFunctions {pred!r}')
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	129
1637 ec3f98d0c1d8 refactor rules eval drewp@bigasterisk.com parents: 1636 diff changeset	130 for functionType in functionsFor(pred):
1661 00a5624d1d14 cleanups and optimizations drewp@bigasterisk.com parents: 1656 diff changeset	131 fn = functionType(self.lhsChunk)
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	132 ringlog.debug(f'{INDENT*7} ChunkLooper{self._shortId} advanceWithFunctions, {functionType=}')
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	133
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	134 try:
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	135 log.debug(f'fn.bind {self._prevBindings()} ...')
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	136 #fullBinding = self._prevBindings().copy()
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	137 newBinding = fn.bind(self._prevBindings())
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	138 log.debug(f'...makes {newBinding=}')
1637 ec3f98d0c1d8 refactor rules eval drewp@bigasterisk.com parents: 1636 diff changeset	139 except BindingUnknown:
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	140 pass
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	141 else:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	142 if newBinding is not None:
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	143 self._currentIsFromFunc = fn
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	144 if self._testAndKeepNewBinding(newBinding):
1637 ec3f98d0c1d8 refactor rules eval drewp@bigasterisk.com parents: 1636 diff changeset	145 return True
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	146
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	147 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	148
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	149 def _testAndKeepNewBinding(self, newBinding: CandidateBinding):
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	150 fullBinding: CandidateBinding = self._prevBindings().copy()
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	151 fullBinding.addNewBindings(newBinding)
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	152 isNew = fullBinding not in self._seenBindings
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	153 ringlog.debug(f'{INDENT*7} {self} considering {newBinding=} to make {fullBinding}. {isNew=}')
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	154 if isNew:
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	155 self._seenBindings.append(fullBinding.copy())
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	156 self._current = newBinding
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	157 return True
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	158 return False
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	159
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	160 def _boundOperands(self, operands) -> List[Node]:
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	161 pb: CandidateBinding = self._prevBindings()
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	162
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	163 boundOperands: List[Node] = []
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	164 for op in operands:
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	165 if isinstance(op, (Variable, RuleUnboundBnode)):
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	166 boundOperands.append(pb.applyTerm(op))
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	167 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	168 boundOperands.append(op)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	169 return boundOperands
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	170
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	171 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	172 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	173 raise NotImplementedError()
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	174 together = self._prevBindings().copy()
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	175 together.addNewBindings(self._current)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	176 return together
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	177
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	178 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	179 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	180
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	181 def restart(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	182 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	183 self._seenBindings = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	184 self.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	185 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	186 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	187
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	188 def prevMayHaveChanged(self):
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	189 # This is a total patch for a test failure. This should be generalized
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	190 # to a Looper that can keep itself correct when prev changes.
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	191 if self._currentIsFromFunc:
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	192 self._advanceWithFunctions()
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	193 if self.pastEnd():
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	194 self.restart()
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	195
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	196
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	197 @dataclass
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	198 class Lhs:
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	199 graph: ChunkedGraph # our full LHS graph, as input. See below for the statements partitioned into groups.
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	200
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	201 def __post_init__(self):
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	202
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	203 self.myPreds = self.graph.allPredicatesExceptFunctions()
1602 e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	204
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	205 def __repr__(self):
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	206 return f"Lhs({self.graph!r})"
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	207
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	208 def findCandidateBindings(self, knownTrue: ChunkedGraph, stats, ruleStatementsIterationLimit) -> Iterator['BoundLhs']:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	209 """distinct bindings that fit the LHS of a rule, using statements from
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	210 workingSet and functions from LHS"""
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	211 if not self.graph:
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	212 # special case- no LHS!
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	213 yield BoundLhs(self, CandidateBinding({}))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	214 return
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	215
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	216 if self._checkPredicateCounts(knownTrue):
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	217 stats['_checkPredicateCountsCulls'] += 1
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	218 return
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	219
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	220 if not all(ch in knownTrue for ch in self.graph.staticChunks):
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	221 stats['staticStmtCulls'] += 1
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	222 return
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	223 # After this point we don't need to consider self.graph.staticChunks.
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	224
1656 d8e5b2232474 bugfix for simple (no arg lists) functions not evaluating drewp@bigasterisk.com parents: 1653 diff changeset	225 if not self.graph.patternChunks and not self.graph.chunksUsedByFuncs:
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	226 # static only
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	227 yield BoundLhs(self, CandidateBinding({}))
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	228 return
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	229
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	230 log.debug('')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	231 try:
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	232 chunkStack = self._assembleRings(knownTrue, stats)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	233 except NoOptions:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	234 ringlog.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	235 return
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	236 log.debug('')
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	237 log.debug('')
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	238 self._debugChunkStack('time to spin: initial odometer is', chunkStack)
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	239 self._assertAllRingsAreValid(chunkStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	240
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	241 lastRing = chunkStack[-1]
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	242 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	243 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	244 iterCount += 1
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	245 if iterCount > ruleStatementsIterationLimit:
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	246 raise ValueError('rule too complex')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	247
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	248 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	249
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	250 yield BoundLhs(self, lastRing.currentBinding())
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	251
1667 a2347393b43e comments, debug, dead code drewp@bigasterisk.com parents: 1664 diff changeset	252 # self._debugChunkStack('odometer', chunkStack)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	253
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	254 done = self._advanceTheStack(chunkStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	255
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	256 self._debugChunkStack(f'odometer after ({done=})', chunkStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	257
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	258 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	259 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	260 break
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	261
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	262 def _debugChunkStack(self, label: str, chunkStack: List[ChunkLooper]):
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	263 odolog.debug(f'{INDENT*4} {label}:')
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	264 for i, l in enumerate(chunkStack):
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	265 odolog.debug(f'{INDENT*5} [{i}] {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	266
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	267 def _checkPredicateCounts(self, knownTrue):
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	268 """raise NoOptions quickly in some cases"""
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	269
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	270 if self.graph.noPredicatesAppear(self.myPreds):
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	271 log.debug(f'{INDENT*3} checkPredicateCounts does cull because not all {self.myPreds=} are in knownTrue')
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	272 return True
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	273 log.debug(f'{INDENT*3} checkPredicateCounts does not cull because all {self.myPreds=} are in knownTrue')
1640 4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	274 return False
4bb6f593ebf3 speedups: abort some rules faster drewp@bigasterisk.com parents: 1639 diff changeset	275
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	276 def _assembleRings(self, knownTrue: ChunkedGraph, stats) -> List[ChunkLooper]:
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	277 """make ChunkLooper for each stmt in our LHS graph, but do it in a way that they all
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	278 start out valid (or else raise NoOptions). static chunks have already been confirmed."""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	279
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	280 log.debug(f'{INDENT*4} stats={dict(stats)}')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	281 odolog.debug(f'{INDENT*3} build new ChunkLooper stack')
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	282 chunks = list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs))
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	283 chunks.sort(key=None)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	284 odolog.info(f' {INDENT*3} taking permutations of {len(chunks)=}')
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	285
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	286 permsTried = 0
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	287
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	288 for perm in self._partitionedGraphPermutations():
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	289 looperRings: List[ChunkLooper] = []
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	290 prev: Optional[ChunkLooper] = None
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	291 if odolog.isEnabledFor(logging.DEBUG):
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	292 odolog.debug(
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	293 f'{INDENT*4} [perm {permsTried}] try rule chunks in this order: {" THEN ".join(repr(p) for p in perm)}')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	294
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	295 for ruleChunk in perm:
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	296 try:
1661 00a5624d1d14 cleanups and optimizations drewp@bigasterisk.com parents: 1656 diff changeset	297 # These are getting rebuilt a lot which takes time. It would
00a5624d1d14 cleanups and optimizations drewp@bigasterisk.com parents: 1656 diff changeset	298 # be nice if they could accept a changing `prev` order
00a5624d1d14 cleanups and optimizations drewp@bigasterisk.com parents: 1656 diff changeset	299 # (which might already be ok).
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	300 looper = ChunkLooper(ruleChunk, prev, knownTrue)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	301 except NoOptions:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	302 odolog.debug(f'{INDENT*5} permutation didnt work, try another')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	303 break
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	304 looperRings.append(looper)
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	305 prev = looperRings[-1]
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	306 else:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	307 # bug: At this point we've only shown that these are valid
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	308 # starting rings. The rules might be tricky enough that this
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	309 # permutation won't get us to the solution.
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	310 return looperRings
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	311 if permsTried > 50000:
1652 dddfa09ea0b9 debug logging and comments drewp@bigasterisk.com parents: 1651 diff changeset	312 raise NotImplementedError(f'trying too many permutations {len(chunks)=}')
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	313 permsTried += 1
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	314
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	315 odolog.debug(f'{INDENT*5} no perms worked- rule cannot match anything')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	316 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	317
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	318 def _unpartitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	319 for perm in itertools.permutations(sorted(list(self.graph.patternChunks.union(self.graph.chunksUsedByFuncs)))):
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	320 yield perm
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	321
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	322 def _partitionedGraphPermutations(self) -> Iterator[Tuple[Chunk, ...]]:
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	323 """always puts function chunks after pattern chunks
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	324
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	325 (and, if we cared, static chunks could go before that. Currently they're
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	326 culled out elsewhere, but that's done as a special case)
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	327 """
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	328 tupleOfNoChunks: Tuple[Chunk, ...] = ()
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	329 pats = sorted(self.graph.patternChunks)
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	330 funcs = sorted(self.graph.chunksUsedByFuncs)
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	331 for patternPart in itertools.permutations(pats) if pats else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	332 for funcPart in itertools.permutations(funcs) if funcs else [tupleOfNoChunks]:
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	333 perm = patternPart + funcPart
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	334 yield perm
2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	335
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	336 def _advanceTheStack(self, looperRings: List[ChunkLooper]) -> bool:
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	337
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	338 def freshenRight(i):
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	339 for ring in looperRings[i + 1:]:
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	340 ring.prevMayHaveChanged()
0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	341
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	342 carry = True # last elem always must advance
89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	343 for i, ring in reversed(list(enumerate(looperRings))):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	344 # unlike normal odometer, advancing any earlier ring could invalidate later ones
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	345 if carry:
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	346 odolog.debug(f'{INDENT*4} advanceAll [ring={i}] {ring} carry/advance')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	347 ring.advance()
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	348 freshenRight(i)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	349 carry = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	350 if ring.pastEnd():
1668 89e53cb8a01c fix some harder tests. Mostly, _advanceTheStack needed to spin the odometer rings starting from the other side, to get all the right combos drewp@bigasterisk.com parents: 1667 diff changeset	351 if ring is looperRings[0]:
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	352 allRingsDone = [r.pastEnd() for r in looperRings]
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	353 odolog.debug(f'{INDENT*5} advanceAll [ring={i}] {ring} says we done {allRingsDone=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	354 return True
1692 2883da14847c debugging and cleanup, as i looked for a bug drewp@bigasterisk.com parents: 1677 diff changeset	355 odolog.debug(f'{INDENT*5} advanceAll [ring={i}] {ring} restart')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	356 ring.restart()
1693 0455a1e18e4f really rough fix to a broken test. hopefully this gets redone drewp@bigasterisk.com parents: 1692 diff changeset	357 freshenRight(i)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	358 carry = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	359 return False
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	360
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	361 def _assertAllRingsAreValid(self, looperRings):
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	362 if any(ring.pastEnd() for ring in looperRings): # this is an unexpected debug assertion
1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	363 odolog.warning(f'{INDENT*4} some rings started at pastEnd {looperRings}')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	364 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	365
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	366
1622 38bd8ef9ef67 add CandidateTermMatches, unused so far drewp@bigasterisk.com parents: 1621 diff changeset	367 @dataclass
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	368 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	369 lhs: Lhs
1610 6fc48ef4c696 mysteriously lost an important line drewp@bigasterisk.com parents: 1609 diff changeset	370 binding: CandidateBinding
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	371
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	372
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	373 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	374 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	375 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	376 rhsGraph: Graph
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	377
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	378 def __post_init__(self):
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	379 self.lhs = Lhs(ChunkedGraph(self.lhsGraph, RuleUnboundBnode, functionsFor))
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	380
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	381 self.maps = {}
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	382
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	383 self.rhsGraphConvert: List[Triple] = []
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	384 for s, p, o in self.rhsGraph:
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	385 from rdflib import BNode
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	386 if isinstance(s, BNode):
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	387 s = RhsBnode(s)
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	388 if isinstance(p, BNode):
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	389 p = RhsBnode(p)
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	390 if isinstance(o, BNode):
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	391 o = RhsBnode(o)
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	392 self.rhsGraphConvert.append((s, p, o))
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	393
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	394 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict, ruleStatementsIterationLimit):
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	395 # this does not change for the current applyRule call. The rule will be
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	396 # tried again in an outer loop, in case it can produce more.
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	397 workingSetChunked = ChunkedGraph(workingSet, WorkingSetBnode, functionsFor)
1651 20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	398
20474ad4968e WIP - functions are broken as i move most layers to work in Chunks not Triples drewp@bigasterisk.com parents: 1650 diff changeset	399 for bound in self.lhs.findCandidateBindings(workingSetChunked, stats, ruleStatementsIterationLimit):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	400 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	401
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	402 newStmts = self.generateImpliedFromRhs(bound.binding)
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	403
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	404 for newStmt in newStmts:
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	405 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	406 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	407 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	408
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	409 def generateImpliedFromRhs(self, binding: CandidateBinding) -> List[Triple]:
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	410
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	411 out: List[Triple] = []
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	412
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	413 # Each time the RHS is used (in a rule firing), its own BNodes (which
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	414 # are subtype RhsBnode) need to be turned into distinct ones. Note that
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	415 # bnodes that come from the working set should not be remapped.
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	416 rhsBnodeMap: Dict[RhsBnode, WorkingSetBnode] = {}
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	417
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	418 # but, the iteration loop could come back with the same bindings again
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	419 key = binding.key()
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	420 rhsBnodeMap = self.maps.setdefault(key, {})
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	421
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	422 for stmt in binding.apply(self.rhsGraphConvert):
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	423
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	424 outStmt: List[Node] = []
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	425
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	426 for t in stmt:
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	427 if isinstance(t, RhsBnode):
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	428 if t not in rhsBnodeMap:
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	429 rhsBnodeMap[t] = WorkingSetBnode()
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	430 t = rhsBnodeMap[t]
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	431
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	432 outStmt.append(t)
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	433
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	434 log.debug(f'{INDENT*6} rhs stmt {stmt} became {outStmt}')
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	435 out.append((outStmt[0], outStmt[1], outStmt[2]))
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	436
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	437 return out
80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	438
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	439
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	440 @dataclass
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	441 class Inference:
1673 80f4e741ca4f redo RHS bnode processing drewp@bigasterisk.com parents: 1669 diff changeset	442 rulesIterationLimit = 4
1664 1a7c1261302c logic fix- some bindings were being returned 2+; some 0 times drewp@bigasterisk.com parents: 1661 diff changeset	443 ruleStatementsIterationLimit = 5000
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	444
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	445 def __init__(self) -> None:
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	446 self.rules: List[Rule] = []
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	447 self._nonRuleStmts: List[Triple] = []
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	448
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	449 def setRules(self, g: ConjunctiveGraph):
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	450 self.rules = []
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	451 self._nonRuleStmts = []
1599 abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	452 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	453 if stmt[1] == LOG['implies']:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	454 self.rules.append(Rule(stmt[0], stmt[2]))
1649 bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	455 else:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	456 self._nonRuleStmts.append(stmt)
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	457
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	458 def nonRuleStatements(self) -> List[Triple]:
bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	459 return self._nonRuleStmts
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	460
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	461 @INFER_CALLS.time()
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	462 def infer(self, graph: Graph):
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	463 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	464 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	465 """
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	466 n = graph.__len__()
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	467 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	468 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	469 startTime = time.time()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	470 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1649 bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	471
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	472 # everything that is true: the input graph, plus every rule conclusion we can make
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	473 workingSet = Graph()
1649 bb5d2b5370ac add nonRuleStatments to Inference api. there's already a test in an eariler commit drewp@bigasterisk.com parents: 1648 diff changeset	474 workingSet += self._nonRuleStmts
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	475 workingSet += graph
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	476
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	477 # just the statements that came from RHS's of rules that fired.
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	478 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	479
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	480 rulesIterations = 0
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	481 delta = 1
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	482 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1674 4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached drewp@bigasterisk.com parents: 1673 diff changeset	483 while delta > 0:
1620 92f8deb59735 log layout drewp@bigasterisk.com parents: 1616 diff changeset	484 log.debug('')
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	485 log.info(f'{INDENT1}iteration {rulesIterations}')
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	486
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	487 delta = -len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	488 self._iterateAllRules(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	489 delta += len(implied)
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	490 rulesIterations += 1
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	491 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1674 4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached drewp@bigasterisk.com parents: 1673 diff changeset	492 if rulesIterations >= self.rulesIterationLimit:
4a15b4cd4600 don't jsut quietly succeed if iteration limit is reached drewp@bigasterisk.com parents: 1673 diff changeset	493 raise ValueError(f"rule too complex after {rulesIterations=}")
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	494 stats['iterations'] = rulesIterations
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	495 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	496 stats['impliedStmts'] = len(implied)
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	497 log.info(f'{INDENT*0} Inference done {dict(stats)}.')
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	498 log.debug('Implied:')
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	499 log.debug(graphDump(implied))
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	500 return implied
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	501
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	502 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats):
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	503 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	504 self._logRuleApplicationHeader(workingSet, i, rule)
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	505 rule.applyRule(workingSet, implied, stats, self.ruleStatementsIterationLimit)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	506
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	507 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	508 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	509 return
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	510
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	511 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	512 log.debug(f'{INDENT*2} workingSet:')
1648 3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	513 # for j, stmt in enumerate(sorted(workingSet)):
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	514 # log.debug(f'{INDENT*3} ({j}) {stmt}')
3059f31b2dfa more performance work drewp@bigasterisk.com parents: 1641 diff changeset	515 log.debug(f'{INDENT*3} {graphDump(workingSet, oneLine=False)}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	516
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	517 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	518 log.debug(f'{INDENT*2}-applying rule {i}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	519 log.debug(f'{INDENT*3} rule def lhs:')
1677 aa35ae7a1acc add new bug test (no fix yet) drewp@bigasterisk.com parents: 1674 diff changeset	520 for stmt in sorted(r.lhs.graph.allChunks()):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	521 log.debug(f'{INDENT*4} {stmt}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	522 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')

Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1693:0455a1e18e4f