homeauto: service/mqtt_to_rdf/inference.py annotate

annotate service/mqtt_to_rdf/inference.py @ 1635:22d481f0a924

refactor: use CandidateBinding throughout, not loose dicts

author	drewp@bigasterisk.com
date	Mon, 13 Sep 2021 00:06:00 -0700
parents	ba59cfc3c747
children	3252bdc284bc

rev	line source
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	1 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	4 """
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	5 import itertools
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	6 import logging
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	7 import time
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	8 from collections import defaultdict
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	9 from dataclasses import dataclass
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	10 from typing import (Dict, Iterator, List, Optional, Sequence, Set, Tuple, Union, cast)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	11
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	12 from prometheus_client import Histogram, Summary
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	13 from rdflib import RDF, BNode, Graph, Namespace
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	14 from rdflib.graph import ConjunctiveGraph, ReadOnlyGraphAggregate
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	15 from rdflib.term import Literal, Node, Variable
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	16
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	17 from candidate_binding import CandidateBinding
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	18 from inference_types import (BindableTerm, BindingUnknown, EvaluationFailed, ReadOnlyWorkingSet, Triple)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	19 from lhs_evaluation import Decimal, Evaluation, numericNode, parseList
1605 449746d1598f WIP move evaluation to new file drewp@bigasterisk.com parents: 1603 diff changeset	20
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	21 log = logging.getLogger('infer')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	22 INDENT = ' '
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	23
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	24 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	25 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	26
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	27 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	28 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	29 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	30
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	31
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	32 def stmtTemplate(stmt: Triple) -> Tuple[Optional[Node], Optional[Node], Optional[Node]]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	33 return (
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	34 None if isinstance(stmt[0], (Variable, BNode)) else stmt[0],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	35 None if isinstance(stmt[1], (Variable, BNode)) else stmt[1],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	36 None if isinstance(stmt[2], (Variable, BNode)) else stmt[2],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	37 )
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	38
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	39
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	40 class NoOptions(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	41 """stmtlooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	42
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	43
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	44 class Inconsistent(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	45 """adding this stmt would be inconsistent with an existing binding"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	46
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	47
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	48 _stmtLooperShortId = itertools.count()
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	49
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	50
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	51 @dataclass
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	52 class StmtLooper:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	53 """given one LHS stmt, iterate through the possible matches for it,
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	54 returning what bindings they would imply. Only distinct bindings are
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	55 returned. The bindings build on any `prev` StmtLooper's results.
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	56
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	57 This iterator is restartable."""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	58 lhsStmt: Triple
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	59 prev: Optional['StmtLooper']
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	60 workingSet: ReadOnlyWorkingSet
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	61 parent: 'Lhs' # just for lhs.graph, really
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	62
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	63 def __repr__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	64 return f'StmtLooper{self._shortId}({graphDump([self.lhsStmt])} {"<pastEnd>" if self.pastEnd() else ""})'
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	65
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	66 def __post_init__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	67 self._shortId = next(_stmtLooperShortId)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	68 self._myWorkingSetMatches = self._myMatches(self.workingSet)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	69
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	70 self._current = CandidateBinding({})
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	71 self._pastEnd = False
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	72 self._seenBindings: List[CandidateBinding] = []
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	73 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	74
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	75 def _myMatches(self, g: Graph) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	76 template = stmtTemplate(self.lhsStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	77
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	78 stmts = sorted(cast(Iterator[Triple], list(g.triples(template))))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	79 # plus new lhs possibilties...
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	80 # log.debug(f'{INDENT*6} {self} find {len(stmts)=} in {len(self.workingSet)=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	81
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	82 return stmts
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	83
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	84 def _prevBindings(self) -> CandidateBinding:
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	85 if not self.prev or self.prev.pastEnd():
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	86 return CandidateBinding({})
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	87
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	88 return self.prev.currentBinding()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	89
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	90 def advance(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	91 """update to a new set of bindings we haven't seen (since last restart), or go into pastEnd mode"""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	92 if self._pastEnd:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	93 raise NotImplementedError('need restart')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	94 log.debug('')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	95 augmentedWorkingSet: Sequence[Triple] = []
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	96 if self.prev is None:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	97 augmentedWorkingSet = self._myWorkingSetMatches
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	98 else:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	99 augmentedWorkingSet = list(self.prev.currentBinding().apply(self._myWorkingSetMatches,
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	100 returnBoundStatementsOnly=False))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	101
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	102 log.debug(f'{INDENT*6} {self}.advance has {augmentedWorkingSet=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	103
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	104 if self._advanceWithPlainMatches(augmentedWorkingSet):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	105 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	106
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	107 if self._advanceWithBoolRules():
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	108 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	109
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	110 curBind = self.prev.currentBinding() if self.prev else CandidateBinding({})
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	111 [lhsStmtBound] = curBind.apply([self.lhsStmt], returnBoundStatementsOnly=False)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	112
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	113 fullWorkingSet = self.workingSet + self.parent.graph
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	114 boundFullWorkingSet = list(curBind.apply(fullWorkingSet, returnBoundStatementsOnly=False))
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	115 log.debug(f'{fullWorkingSet.__len__()=} {len(boundFullWorkingSet)=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	116
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	117 if self._advanceWithFunctions(augmentedWorkingSet, boundFullWorkingSet, lhsStmtBound):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	118 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	119
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	120 log.debug(f'{INDENT*6} {self} is past end')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	121 self._pastEnd = True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	122
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	123 def _advanceWithPlainMatches(self, augmentedWorkingSet: Sequence[Triple]) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	124 log.debug(f'{INDENT*7} {self} mines {len(augmentedWorkingSet)} matching augmented statements')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	125 for s in augmentedWorkingSet:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	126 log.debug(f'{INDENT*7} {s}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	127
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	128 for i, stmt in enumerate(augmentedWorkingSet):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	129 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	130 outBinding = self._totalBindingIfThisStmtWereTrue(stmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	131 except Inconsistent:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	132 log.debug(f'{INDENT*7} {self} - {stmt} would be inconsistent with prev bindings')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	133 continue
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	134
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	135 log.debug(f'{INDENT*7} {outBinding=} {self._seenBindings=}')
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	136 if outBinding not in self._seenBindings:
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	137 self._seenBindings.append(outBinding.copy())
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	138 self._current = outBinding
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	139 log.debug(f'{INDENT*7} new binding from {self} -> {outBinding}')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	140 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	141 return False
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	142
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	143 def _advanceWithBoolRules(self) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	144 log.debug(f'{INDENT*7} {self} mines bool rules')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	145 if self.lhsStmt[1] == MATH['greaterThan']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	146 operands = [self.lhsStmt[0], self.lhsStmt[2]]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	147 try:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	148 boundOperands = self._boundOperands(operands)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	149 except BindingUnknown:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	150 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	151 if numericNode(boundOperands[0]) > numericNode(boundOperands[1]):
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	152 binding: CandidateBinding = self._prevBindings().copy() # no new values; just allow matching to keep going
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	153 if binding not in self._seenBindings:
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	154 self._seenBindings.append(binding)
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	155 self._current = binding
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	156 log.debug(f'{INDENT*7} new binding from {self} -> {binding}')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	157 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	158 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	159
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	160 def _advanceWithFunctions(self, augmentedWorkingSet: Sequence[Triple], boundFullWorkingSet, lhsStmtBound) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	161 log.debug(f'{INDENT*7} {self} mines rules')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	162
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	163 if self.lhsStmt[1] == ROOM['asFarenheit']:
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	164 pb: CandidateBinding = self._prevBindings()
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	165 log.debug(f'{INDENT*7} {self} consider ?x faren ?y where ?x={self.lhsStmt[0]} and {pb=}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	166
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	167 if isinstance(self.lhsStmt[0], (Variable, BNode)) and pb.contains(self.lhsStmt[0]):
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	168 operands = [pb.applyTerm(self.lhsStmt[0])]
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	169 f = cast(Literal, Literal(Decimal(numericNode(operands[0])) * 9 / 5 + 32))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	170 objVar = self.lhsStmt[2]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	171 if not isinstance(objVar, Variable):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	172 raise TypeError(f'expected Variable, got {objVar!r}')
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	173 newBindings = CandidateBinding({cast(BindableTerm, objVar): cast(Node, f)})
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	174 self._current.addNewBindings(newBindings)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	175 if newBindings not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	176 self._seenBindings.append(newBindings)
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	177 self._current = newBindings
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	178 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	179 elif self.lhsStmt[1] == MATH['sum']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	180
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	181 g = Graph()
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	182 for s in boundFullWorkingSet:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	183 g.add(s)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	184 log.debug(f' boundWorkingSet graph: {s}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	185 log.debug(f'_parseList subj = {lhsStmtBound[0]}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	186 operands, _ = parseList(g, lhsStmtBound[0])
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	187 log.debug(f'********* {INDENT*7} {self} found list {operands=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	188 try:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	189 obj = Literal(sum(map(numericNode, operands)))
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	190 except TypeError:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	191 log.debug('typeerr in operands')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	192 pass
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	193 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	194 objVar = lhsStmtBound[2]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	195 log.debug(f'{objVar=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	196
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	197 if not isinstance(objVar, Variable):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	198 raise TypeError(f'expected Variable, got {objVar!r}')
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	199 newBindings = CandidateBinding({objVar: obj})
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	200 log.debug(f'{newBindings=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	201
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	202 self._current.addNewBindings(newBindings)
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	203 log.debug(f'{self._seenBindings=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	204 if newBindings not in self._seenBindings:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	205 self._seenBindings.append(newBindings)
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	206 self._current = newBindings
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	207 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	208
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	209 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	210
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	211 def _boundOperands(self, operands) -> List[Node]:
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	212 pb: CandidateBinding = self._prevBindings()
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	213
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	214 boundOperands: List[Node] = []
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	215 for op in operands:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	216 if isinstance(op, (Variable, BNode)):
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	217 boundOperands.append(pb.applyTerm(op))
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	218 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	219 boundOperands.append(op)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	220 return boundOperands
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	221
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	222 def _totalBindingIfThisStmtWereTrue(self, newStmt: Triple) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	223 outBinding = self._prevBindings().copy()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	224 for rt, ct in zip(self.lhsStmt, newStmt):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	225 if isinstance(rt, (Variable, BNode)):
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	226 if outBinding.contains(rt) and outBinding.applyTerm(rt) != ct:
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	227 raise Inconsistent(f'{rt=} {ct=} {outBinding=}')
1635 22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	228 outBinding.addNewBindings(CandidateBinding({rt: ct}))
22d481f0a924 refactor: use CandidateBinding throughout, not loose dicts drewp@bigasterisk.com parents: 1634 diff changeset	229 return outBinding
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	230
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	231 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	232 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	233 raise NotImplementedError()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	234 return self._current
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	235
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	236 def newLhsStmts(self) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	237 """under the curent bindings, what new stmts beyond workingSet are also true? includes all `prev`"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	238 return []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	239
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	240 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	241 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	242
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	243 def restart(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	244 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	245 self._seenBindings = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	246 self.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	247 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	248 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	249
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	250
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	251 @dataclass
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	252 class Lhs:
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	253 graph: Graph
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	254
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	255 def __post_init__(self):
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	256 # do precomputation in here that's not specific to the workingSet
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	257 # self.staticRuleStmts = Graph()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	258 # self.nonStaticRuleStmts = Graph()
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	259
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	260 # self.lhsBindables: Set[BindableTerm] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	261 # self.lhsBnodes: Set[BNode] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	262 # for ruleStmt in self.graph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	263 # varsAndBnodesInStmt = [term for term in ruleStmt if isinstance(term, (Variable, BNode))]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	264 # self.lhsBindables.update(varsAndBnodesInStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	265 # self.lhsBnodes.update(x for x in varsAndBnodesInStmt if isinstance(x, BNode))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	266 # if not varsAndBnodesInStmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	267 # self.staticRuleStmts.add(ruleStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	268 # else:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	269 # self.nonStaticRuleStmts.add(ruleStmt)
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	270
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	271 # self.nonStaticRuleStmtsSet = set(self.nonStaticRuleStmts)
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	272
1602 e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	273 self.evaluations = list(Evaluation.findEvals(self.graph))
e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	274
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	275 def __repr__(self):
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	276 return f"Lhs({graphDump(self.graph)})"
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	277
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	278 def findCandidateBindings(self, knownTrue: ReadOnlyWorkingSet, stats) -> Iterator['BoundLhs']:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	279 """bindings that fit the LHS of a rule, using statements from workingSet and functions
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	280 from LHS"""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	281 if self.graph.__len__() == 0:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	282 # special case- no LHS!
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	283 yield BoundLhs(self, CandidateBinding({}))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	284 return
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	285
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	286 log.debug(f'{INDENT*4} build new StmtLooper stack')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	287
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	288 try:
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	289 stmtStack = self._assembleRings(knownTrue)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	290 except NoOptions:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	291 log.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	292 return
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	293 self._debugStmtStack('initial odometer', stmtStack)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	294 self._assertAllRingsAreValid(stmtStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	295
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	296 lastRing = stmtStack[-1]
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	297 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	298 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	299 iterCount += 1
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	300 if iterCount > 10:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	301 raise ValueError('stuck')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	302
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	303 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	304
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	305 yield BoundLhs(self, lastRing.currentBinding())
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	306
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	307 self._debugStmtStack('odometer', stmtStack)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	308
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	309 done = self._advanceAll(stmtStack)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	310
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	311 self._debugStmtStack('odometer after ({done=})', stmtStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	312
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	313 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	314 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	315 break
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	316
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	317 def _debugStmtStack(self, label, stmtStack):
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	318 log.debug(f'{INDENT*5} {label}:')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	319 for l in stmtStack:
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	320 log.debug(f'{INDENT*6} {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	321
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	322 def _assembleRings(self, knownTrue: ReadOnlyWorkingSet) -> List[StmtLooper]:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	323 """make StmtLooper for each stmt in our LHS graph, but do it in a way that they all
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	324 start out valid (or else raise NoOptions)"""
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	325
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	326 usedByFuncs: Set[Triple] = set() # don't worry about matching these
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	327 stmtsToResolve = list(self.graph)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	328 for i, s in enumerate(stmtsToResolve):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	329 if s[1] == MATH['sum']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	330 _, used = parseList(self.graph, s[0])
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	331 usedByFuncs.update(used)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	332
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	333 stmtsToAdd = [stmt for stmt in stmtsToResolve if not stmt in usedByFuncs]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	334
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	335 # sort them by variable dependencies; don't just try all perms!
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	336 def lightSortKey(stmt): # Not this. Though it helps performance on the big rdf list cases.
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	337 (s, p, o) = stmt
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	338 return p == MATH['sum'], p, s, o
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	339
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	340 stmtsToAdd.sort(key=lightSortKey)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	341
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	342 for perm in itertools.permutations(stmtsToAdd):
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	343 stmtStack: List[StmtLooper] = []
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	344 prev: Optional[StmtLooper] = None
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	345 log.debug(f'{INDENT*5} try stmts in this order: {" -> ".join(graphDump([p]) for p in perm)}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	346
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	347 for s in perm:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	348 try:
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	349 elem = StmtLooper(s, prev, knownTrue, parent=self)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	350 except NoOptions:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	351 log.debug(f'{INDENT*6} permutation didnt work, try another')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	352 break
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	353 stmtStack.append(elem)
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	354 prev = stmtStack[-1]
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	355 else:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	356 return stmtStack
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	357 log.debug(f'{INDENT*6} no perms worked- rule cannot match anything')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	358
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	359 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	360
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	361 def _advanceAll(self, stmtStack: List[StmtLooper]) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	362 carry = True # 1st elem always must advance
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	363 for i, ring in enumerate(stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	364 # unlike normal odometer, advancing any earlier ring could invalidate later ones
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	365 if carry:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	366 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} carry/advance')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	367 ring.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	368 carry = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	369 if ring.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	370 if ring is stmtStack[-1]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	371 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} says we done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	372 return True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	373 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} restart')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	374 ring.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	375 carry = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	376 return False
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	377
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	378 def _assertAllRingsAreValid(self, stmtStack):
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	379 if any(ring.pastEnd() for ring in stmtStack): # this is an unexpected debug assertion
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	380 log.debug(f'{INDENT*5} some rings started at pastEnd {stmtStack}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	381 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	382
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	383 def _allStaticStatementsMatch(self, knownTrue: ReadOnlyWorkingSet) -> bool:
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	384 # bug: see TestSelfFulfillingRule.test3 for a case where this rule's
03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	385 # static stmt is matched by a non-static stmt in the rule itself
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	386 for ruleStmt in self.staticRuleStmts:
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	387 if ruleStmt not in knownTrue:
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	388 log.debug(f'{INDENT*3} {ruleStmt} not in working set- skip rule')
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	389 return False
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	390 return True
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	391
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	392 def _possibleBindings(self, workingSet, stats) -> Iterator['BoundLhs']:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	393 """this yields at least the working bindings, and possibly others"""
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	394 candidateTermMatches: Dict[BindableTerm, Set[Node]] = self._allCandidateTermMatches(workingSet)
1614 97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	395 for bindRow in self._product(candidateTermMatches):
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	396 try:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	397 yield BoundLhs(self, bindRow)
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	398 except EvaluationFailed:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	399 stats['permCountFailingEval'] += 1
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	400
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	401 def _allCandidateTermMatches(self, workingSet: ReadOnlyWorkingSet) -> Dict[BindableTerm, Set[Node]]:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	402 """the total set of terms each variable could possibly match"""
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	403
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	404 candidateTermMatches: Dict[BindableTerm, Set[Node]] = defaultdict(set)
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	405 for lhsStmt in self.graph:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	406 log.debug(f'{INDENT*4} possibles for this lhs stmt: {lhsStmt}')
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	407 for i, trueStmt in enumerate(workingSet):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	408 # log.debug(f'{INDENT*5} consider this true stmt ({i}): {trueStmt}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	409
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	410 for v, vals in self._bindingsFromStatement(lhsStmt, trueStmt):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	411 candidateTermMatches[v].update(vals)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	412
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	413 return candidateTermMatches
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	414
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	415 def _bindingsFromStatement(self, stmt1: Triple, stmt2: Triple) -> Iterator[Tuple[Variable, Set[Node]]]:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	416 """if these stmts match otherwise, what BNode or Variable mappings do we learn?
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	417
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	418 e.g. stmt1=(?x B ?y) and stmt2=(A B C), then we yield (?x, {A}) and (?y, {C})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	419 or stmt1=(_:x B C) and stmt2=(A B C), then we yield (_:x, {A})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	420 or stmt1=(?x B C) and stmt2=(A B D), then we yield nothing
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	421 """
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	422 bindingsFromStatement = {}
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	423 for term1, term2 in zip(stmt1, stmt2):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	424 if isinstance(term1, (BNode, Variable)):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	425 bindingsFromStatement.setdefault(term1, set()).add(term2)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	426 elif term1 != term2:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	427 break
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	428 else:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	429 for v, vals in bindingsFromStatement.items():
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	430 log.debug(f'{INDENT*5} {v=} {vals=}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	431 yield v, vals
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	432
1627 ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	433 def _product(self, candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Iterator[CandidateBinding]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	434 orderedVars, orderedValueSets = _organize(candidateTermMatches)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	435 self._logCandidates(orderedVars, orderedValueSets)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	436 log.debug(f'{INDENT*3} trying all permutations:')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	437 if not orderedValueSets:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	438 yield CandidateBinding({})
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	439 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	440
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	441 if not orderedValueSets or not all(orderedValueSets):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	442 # some var or bnode has no options at all
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	443 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	444 rings: List[Iterator[Node]] = [itertools.cycle(valSet) for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	445 currentSet: List[Node] = [next(ring) for ring in rings]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	446 starts = [valSet[-1] for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	447 while True:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	448 for col, curr in enumerate(currentSet):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	449 currentSet[col] = next(rings[col])
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	450 log.debug(f'{INDENT*4} currentSet: {repr(currentSet)}')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	451 yield CandidateBinding(dict(zip(orderedVars, currentSet)))
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	452 if curr is not starts[col]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	453 break
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	454 if col == len(orderedValueSets) - 1:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	455 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	456
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	457 def _logCandidates(self, orderedVars, orderedValueSets):
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	458 if not log.isEnabledFor(logging.DEBUG):
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	459 return
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	460 log.debug(f'{INDENT*3} resulting candidate terms:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	461 for v, vals in zip(orderedVars, orderedValueSets):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	462 log.debug(f'{INDENT*4} {v!r} could be:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	463 for val in vals:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	464 log.debug(f'{INDENT*5}{val!r}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	465
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	466
1622 38bd8ef9ef67 add CandidateTermMatches, unused so far drewp@bigasterisk.com parents: 1621 diff changeset	467 @dataclass
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	468 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	469 lhs: Lhs
1610 6fc48ef4c696 mysteriously lost an important line drewp@bigasterisk.com parents: 1609 diff changeset	470 binding: CandidateBinding
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	471
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	472 def __post_init__(self):
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	473 self.usedByFuncs = Graph()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	474 # self._applyFunctions()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	475
1616 3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	476 def lhsStmtsWithoutEvals(self):
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	477 for stmt in self.lhs.graph:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	478 if stmt in self.usedByFuncs:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	479 continue
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	480 yield stmt
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	481
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	482 def _applyFunctions(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	483 """may grow the binding with some results"""
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	484 while True:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	485 delta = self._applyFunctionsIteration()
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	486 if delta == 0:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	487 break
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	488
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	489 def _applyFunctionsIteration(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	490 before = len(self.binding.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	491 delta = 0
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	492 for ev in self.lhs.evaluations:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	493 newBindings, usedGraph = ev.resultBindings(self.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	494 self.usedByFuncs += usedGraph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	495 self.binding.addNewBindings(newBindings)
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	496 delta = len(self.binding.binding) - before
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	497 log.debug(f'{INDENT*4} eval rules made {delta} new bindings')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	498 return delta
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	499
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	500 def verify(self, workingSet: ReadOnlyWorkingSet) -> bool:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	501 """Can this bound lhs be true all at once in workingSet?"""
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	502 rem = cast(Set[Triple], self.lhs.nonStaticRuleStmtsSet.difference(self.usedByFuncs))
bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	503 boundLhs = self.binding.apply(rem)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	504
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	505 if log.isEnabledFor(logging.DEBUG):
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	506 boundLhs = list(boundLhs)
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	507 self._logVerifyBanner(boundLhs, workingSet)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	508
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	509 for stmt in boundLhs:
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	510 log.debug(f'{INDENT*4} check for %s', stmt)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	511
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	512 if stmt not in workingSet:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	513 log.debug(f'{INDENT*5} stmt not known to be true')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	514 return False
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	515 return True
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	516
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	517 def _logVerifyBanner(self, boundLhs, workingSet: ReadOnlyWorkingSet):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	518 log.debug(f'{INDENT*4}/ verify all bindings against this boundLhs:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	519 for stmt in sorted(boundLhs):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	520 log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	521
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	522 # log.debug(f'{INDENT*4}\| and against this workingSet:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	523 # for stmt in sorted(workingSet):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	524 # log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	525
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	526 log.debug(f'{INDENT*4}\\')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	527
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	528
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	529 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	530 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	531 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	532 rhsGraph: Graph
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	533
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	534 def __post_init__(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	535 self.lhs = Lhs(self.lhsGraph)
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	536 #
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	537 self.rhsBnodeMap = {}
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	538
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	539 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	540 for bound in self.lhs.findCandidateBindings(ReadOnlyGraphAggregate([workingSet]), stats):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	541 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	542
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	543 # rhs could have more bnodes, and they just need to be distinct per rule-firing that we do
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	544 existingRhsBnodes = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	545 for stmt in self.rhsGraph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	546 for t in stmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	547 if isinstance(t, BNode):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	548 existingRhsBnodes.add(t)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	549 # if existingRhsBnodes:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	550 # log.debug(f'{INDENT*6} mapping rhs bnodes {existingRhsBnodes} to new ones')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	551
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	552 for b in existingRhsBnodes:
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	553
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	554 key = tuple(sorted(bound.binding.binding.items())), b
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	555 self.rhsBnodeMap.setdefault(key, BNode())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	556
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	557 bound.binding.addNewBindings(CandidateBinding({b: self.rhsBnodeMap[key]}))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	558
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	559 # for lhsBoundStmt in bound.binding.apply(bound.lhsStmtsWithoutEvals()):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	560 # log.debug(f'{INDENT*6} adding to workingSet {lhsBoundStmt=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	561 # workingSet.add(lhsBoundStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	562 # log.debug(f'{INDENT*6} rhsGraph is good: {list(self.rhsGraph)}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	563
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	564 for newStmt in bound.binding.apply(self.rhsGraph):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	565 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	566 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	567 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	568
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	569
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	570 class Inference:
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	571
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	572 def __init__(self) -> None:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	573 self.rules = []
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	574
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	575 def setRules(self, g: ConjunctiveGraph):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	576 self.rules: List[Rule] = []
1599 abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	577 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	578 if stmt[1] == LOG['implies']:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	579 self.rules.append(Rule(stmt[0], stmt[2]))
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	580 # other stmts should go to a default working set?
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	581
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	582 @INFER_CALLS.time()
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	583 def infer(self, graph: Graph):
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	584 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	585 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	586 """
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	587 n = graph.__len__()
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	588 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	589 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	590 startTime = time.time()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	591 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	592 # everything that is true: the input graph, plus every rule conclusion we can make
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	593 workingSet = Graph()
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	594 workingSet += graph
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	595
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	596 # just the statements that came from RHS's of rules that fired.
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	597 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	598
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	599 bailout_iterations = 100
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	600 delta = 1
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	601 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	602 while delta > 0 and bailout_iterations > 0:
1620 92f8deb59735 log layout drewp@bigasterisk.com parents: 1616 diff changeset	603 log.debug('')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	604 log.info(f'{INDENT1}iteration ({bailout_iterations} left)')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	605 bailout_iterations -= 1
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	606 delta = -len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	607 self._iterateAllRules(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	608 delta += len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	609 stats['iterations'] += 1
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	610 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	611 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	612 stats['impliedStmts'] = len(implied)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	613 log.info(f'{INDENT*0} Inference done {dict(stats)}. Implied:')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	614 for st in implied:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	615 log.info(f'{INDENT*1} {st}')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	616 return implied
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	617
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	618 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats):
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	619 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	620 self._logRuleApplicationHeader(workingSet, i, rule)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	621 rule.applyRule(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	622
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	623 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	624 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	625 return
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	626
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	627 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	628 log.debug(f'{INDENT*2} workingSet:')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	629 for j, stmt in enumerate(sorted(workingSet)):
387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	630 log.debug(f'{INDENT*3} ({j}) {stmt}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	631
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	632 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	633 log.debug(f'{INDENT*2}-applying rule {i}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	634 log.debug(f'{INDENT*3} rule def lhs:')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	635 for stmt in sorted(r.lhsGraph, reverse=True):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	636 log.debug(f'{INDENT*4} {stmt}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	637 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	638
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	639
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	640 def graphDump(g: Union[Graph, List[Triple]]):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	641 if not isinstance(g, Graph):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	642 g2 = Graph()
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	643 g2 += g
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	644 g = g2
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	645 g.bind('', ROOM)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	646 g.bind('ex', Namespace('http://example.com/'))
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	647 lines = cast(bytes, g.serialize(format='n3')).decode('utf8').splitlines()
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	648 lines = [line.strip() for line in lines if not line.startswith('@prefix')]
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	649 return ' '.join(lines)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	650
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	651
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	652 def _organize(candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Tuple[List[BindableTerm], List[List[Node]]]:
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	653 items = list(candidateTermMatches.items())
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	654 items.sort()
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	655 orderedVars: List[BindableTerm] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	656 orderedValueSets: List[List[Node]] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	657 for v, vals in items:
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	658 orderedVars.append(v)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	659 orderedValues: List[Node] = list(vals)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	660 orderedValues.sort(key=str)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	661 orderedValueSets.append(orderedValues)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	662
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	663 return orderedVars, orderedValueSets

Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1635:22d481f0a924