homeauto: service/mqtt_to_rdf/inference.py annotate

annotate service/mqtt_to_rdf/inference.py @ 1634:ba59cfc3c747

hack math:sum in there. Test suite is passing except some slow performers

author	drewp@bigasterisk.com
date	Sun, 12 Sep 2021 23:48:43 -0700
parents	6107603ed455
children	22d481f0a924

rev	line source
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	1 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	4 """
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	5 import itertools
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	6 import logging
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	7 import time
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	8 from collections import defaultdict
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	9 from dataclasses import dataclass
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	10 from typing import (Dict, Iterator, List, Optional, Sequence, Set, Tuple, Union, cast)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	11
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	12 from prometheus_client import Histogram, Summary
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	13 from rdflib import RDF, BNode, Graph, Namespace
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	14 from rdflib.graph import ConjunctiveGraph, ReadOnlyGraphAggregate
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	15 from rdflib.term import Literal, Node, Variable
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	16
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	17 from candidate_binding import CandidateBinding
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	18 from inference_types import (BindableTerm, BindingUnknown, EvaluationFailed, ReadOnlyWorkingSet, Triple)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	19 from lhs_evaluation import Decimal, Evaluation, numericNode, parseList
1605 449746d1598f WIP move evaluation to new file drewp@bigasterisk.com parents: 1603 diff changeset	20
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	21 log = logging.getLogger('infer')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	22 INDENT = ' '
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	23
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	24 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	25 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	26
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	27 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	28 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	29 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	30
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	31
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	32 def stmtTemplate(stmt: Triple) -> Tuple[Optional[Node], Optional[Node], Optional[Node]]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	33 return (
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	34 None if isinstance(stmt[0], (Variable, BNode)) else stmt[0],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	35 None if isinstance(stmt[1], (Variable, BNode)) else stmt[1],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	36 None if isinstance(stmt[2], (Variable, BNode)) else stmt[2],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	37 )
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	38
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	39
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	40 class NoOptions(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	41 """stmtlooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	42
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	43
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	44 class Inconsistent(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	45 """adding this stmt would be inconsistent with an existing binding"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	46
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	47
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	48 _stmtLooperShortId = itertools.count()
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	49
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	50
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	51 @dataclass
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	52 class StmtLooper:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	53 """given one LHS stmt, iterate through the possible matches for it,
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	54 returning what bindings they would imply. Only distinct bindings are
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	55 returned. The bindings build on any `prev` StmtLooper's results.
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	56
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	57 This iterator is restartable."""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	58 lhsStmt: Triple
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	59 prev: Optional['StmtLooper']
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	60 workingSet: ReadOnlyWorkingSet
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	61 parent: 'Lhs' # just for lhs.graph, really
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	62
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	63 def __repr__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	64 return f'StmtLooper{self._shortId}({graphDump([self.lhsStmt])} {"<pastEnd>" if self.pastEnd() else ""})'
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	65
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	66 def __post_init__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	67 self._shortId = next(_stmtLooperShortId)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	68 self._myWorkingSetMatches = self._myMatches(self.workingSet)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	69
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	70 self._current = CandidateBinding({})
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	71 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	72 self._seenBindings: List[Dict[BindableTerm, Node]] = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	73 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	74
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	75 def _myMatches(self, g: Graph) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	76 template = stmtTemplate(self.lhsStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	77
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	78 stmts = sorted(cast(Iterator[Triple], list(g.triples(template))))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	79 # plus new lhs possibilties...
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	80 # log.debug(f'{INDENT*6} {self} find {len(stmts)=} in {len(self.workingSet)=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	81
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	82 return stmts
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	83
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	84 def _prevBindings(self) -> Dict[BindableTerm, Node]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	85 if not self.prev or self.prev.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	86 return {}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	87
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	88 return self.prev.currentBinding().binding
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	89
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	90 def advance(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	91 """update to a new set of bindings we haven't seen (since last restart), or go into pastEnd mode"""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	92 if self._pastEnd:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	93 raise NotImplementedError('need restart')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	94 log.debug('')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	95 augmentedWorkingSet: Sequence[Triple] = []
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	96 if self.prev is None:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	97 augmentedWorkingSet = self._myWorkingSetMatches
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	98 else:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	99 augmentedWorkingSet = list(self.prev.currentBinding().apply(self._myWorkingSetMatches,
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	100 returnBoundStatementsOnly=False))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	101
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	102 log.debug(f'{INDENT*6} {self}.advance has {augmentedWorkingSet=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	103
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	104 if self._advanceWithPlainMatches(augmentedWorkingSet):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	105 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	106
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	107 if self._advanceWithBoolRules():
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	108 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	109
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	110 curBind = self.prev.currentBinding() if self.prev else CandidateBinding({})
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	111 [lhsStmtBound] = curBind.apply([self.lhsStmt], returnBoundStatementsOnly=False)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	112
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	113 fullWorkingSet = self.workingSet + self.parent.graph
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	114 boundFullWorkingSet = list(curBind.apply(fullWorkingSet, returnBoundStatementsOnly=False))
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	115 log.debug(f'{fullWorkingSet.__len__()=} {len(boundFullWorkingSet)=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	116
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	117 if self._advanceWithFunctions(augmentedWorkingSet, boundFullWorkingSet, lhsStmtBound):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	118 return
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	119
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	120 log.debug(f'{INDENT*6} {self} is past end')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	121 self._pastEnd = True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	122
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	123 def _advanceWithPlainMatches(self, augmentedWorkingSet: Sequence[Triple]) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	124 log.debug(f'{INDENT*7} {self} mines {len(augmentedWorkingSet)} matching augmented statements')
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	125 for s in augmentedWorkingSet:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	126 log.debug(f'{INDENT*7} {s}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	127
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	128 for i, stmt in enumerate(augmentedWorkingSet):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	129 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	130 outBinding = self._totalBindingIfThisStmtWereTrue(stmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	131 except Inconsistent:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	132 log.debug(f'{INDENT*7} {self} - {stmt} would be inconsistent with prev bindings')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	133 continue
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	134
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	135 log.debug(f'{INDENT*7} {outBinding=} {self._seenBindings=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	136 if outBinding.binding not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	137 self._seenBindings.append(outBinding.binding.copy())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	138 self._current = outBinding
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	139 log.debug(f'{INDENT*7} new binding from {self} -> {outBinding}')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	140 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	141 return False
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	142
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	143 def _advanceWithBoolRules(self) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	144 log.debug(f'{INDENT*7} {self} mines bool rules')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	145 if self.lhsStmt[1] == MATH['greaterThan']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	146 operands = [self.lhsStmt[0], self.lhsStmt[2]]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	147 try:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	148 boundOperands = self._boundOperands(operands)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	149 except BindingUnknown:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	150 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	151 if numericNode(boundOperands[0]) > numericNode(boundOperands[1]):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	152 bindingDict: Dict[BindableTerm,
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	153 Node] = self._prevBindings().copy() # no new values; just allow matching to keep going
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	154 if bindingDict not in self._seenBindings:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	155 self._seenBindings.append(bindingDict)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	156 self._current = CandidateBinding(bindingDict)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	157 log.debug(f'{INDENT*7} new binding from {self} -> {bindingDict}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	158 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	159 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	160
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	161 def _advanceWithFunctions(self, augmentedWorkingSet: Sequence[Triple], boundFullWorkingSet, lhsStmtBound) -> bool:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	162 log.debug(f'{INDENT*7} {self} mines rules')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	163
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	164 if self.lhsStmt[1] == ROOM['asFarenheit']:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	165 pb: Dict[BindableTerm, Node] = self._prevBindings()
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	166 log.debug(f'{INDENT*7} {self} consider ?x faren ?y where ?x={self.lhsStmt[0]} and {pb=}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	167
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	168 if self.lhsStmt[0] in pb:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	169 operands = [pb[cast(BindableTerm, self.lhsStmt[0])]]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	170 f = cast(Literal, Literal(Decimal(numericNode(operands[0])) * 9 / 5 + 32))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	171 objVar = self.lhsStmt[2]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	172 if not isinstance(objVar, Variable):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	173 raise TypeError(f'expected Variable, got {objVar!r}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	174 newBindings = {cast(BindableTerm, objVar): cast(Node, f)}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	175 self._current.addNewBindings(CandidateBinding(newBindings))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	176 if newBindings not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	177 self._seenBindings.append(newBindings)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	178 self._current = CandidateBinding(newBindings)
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	179 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	180 elif self.lhsStmt[1] == MATH['sum']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	181
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	182 g = Graph()
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	183 for s in boundFullWorkingSet:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	184 g.add(s)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	185 log.debug(f' boundWorkingSet graph: {s}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	186 log.debug(f'_parseList subj = {lhsStmtBound[0]}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	187 operands, _ = parseList(g, lhsStmtBound[0])
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	188 log.debug(f'********* {INDENT*7} {self} found list {operands=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	189 try:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	190 obj = Literal(sum(map(numericNode, operands)))
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	191 except TypeError:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	192 log.debug('typeerr in operands')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	193 pass
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	194 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	195 objVar = lhsStmtBound[2]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	196 log.debug(f'{objVar=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	197
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	198 if not isinstance(objVar, Variable):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	199 raise TypeError(f'expected Variable, got {objVar!r}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	200 newBindings: Dict[BindableTerm, Node] = {objVar: obj}
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	201 log.debug(f'{newBindings=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	202
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	203 self._current.addNewBindings(CandidateBinding(newBindings))
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	204 log.debug(f'{self._seenBindings=}')
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	205 if newBindings not in self._seenBindings:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	206 self._seenBindings.append(newBindings)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	207 self._current = CandidateBinding(newBindings)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	208 return True
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	209
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	210 return False
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	211
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	212 def _boundOperands(self, operands) -> List[Node]:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	213 pb: Dict[BindableTerm, Node] = self._prevBindings()
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	214
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	215 boundOperands: List[Node] = []
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	216 for op in operands:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	217 if isinstance(op, (Variable, BNode)):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	218 if op in pb:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	219 boundOperands.append(pb[op])
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	220 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	221 raise BindingUnknown()
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	222 else:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	223 boundOperands.append(op)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	224 return boundOperands
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	225
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	226 def _totalBindingIfThisStmtWereTrue(self, newStmt: Triple) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	227 outBinding = self._prevBindings().copy()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	228 for rt, ct in zip(self.lhsStmt, newStmt):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	229 if isinstance(rt, (Variable, BNode)):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	230 if rt in outBinding and outBinding[rt] != ct:
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	231 raise Inconsistent(f'{rt=} {ct=} {outBinding=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	232 outBinding[rt] = ct
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	233 return CandidateBinding(outBinding)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	234
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	235 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	236 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	237 raise NotImplementedError()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	238 return self._current
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	239
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	240 def newLhsStmts(self) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	241 """under the curent bindings, what new stmts beyond workingSet are also true? includes all `prev`"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	242 return []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	243
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	244 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	245 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	246
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	247 def restart(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	248 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	249 self._seenBindings = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	250 self.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	251 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	252 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	253
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	254
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	255 @dataclass
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	256 class Lhs:
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	257 graph: Graph
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	258
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	259 def __post_init__(self):
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	260 # do precomputation in here that's not specific to the workingSet
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	261 # self.staticRuleStmts = Graph()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	262 # self.nonStaticRuleStmts = Graph()
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	263
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	264 # self.lhsBindables: Set[BindableTerm] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	265 # self.lhsBnodes: Set[BNode] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	266 # for ruleStmt in self.graph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	267 # varsAndBnodesInStmt = [term for term in ruleStmt if isinstance(term, (Variable, BNode))]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	268 # self.lhsBindables.update(varsAndBnodesInStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	269 # self.lhsBnodes.update(x for x in varsAndBnodesInStmt if isinstance(x, BNode))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	270 # if not varsAndBnodesInStmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	271 # self.staticRuleStmts.add(ruleStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	272 # else:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	273 # self.nonStaticRuleStmts.add(ruleStmt)
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	274
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	275 # self.nonStaticRuleStmtsSet = set(self.nonStaticRuleStmts)
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	276
1602 e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	277 self.evaluations = list(Evaluation.findEvals(self.graph))
e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	278
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	279 def __repr__(self):
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	280 return f"Lhs({graphDump(self.graph)})"
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	281
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	282 def findCandidateBindings(self, knownTrue: ReadOnlyWorkingSet, stats) -> Iterator['BoundLhs']:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	283 """bindings that fit the LHS of a rule, using statements from workingSet and functions
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	284 from LHS"""
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	285 if self.graph.__len__() == 0:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	286 # special case- no LHS!
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	287 yield BoundLhs(self, CandidateBinding({}))
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	288 return
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	289
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	290 log.debug(f'{INDENT*4} build new StmtLooper stack')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	291
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	292 try:
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	293 stmtStack = self._assembleRings(knownTrue)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	294 except NoOptions:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	295 log.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	296 return
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	297 self._debugStmtStack('initial odometer', stmtStack)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	298 self._assertAllRingsAreValid(stmtStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	299
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	300 lastRing = stmtStack[-1]
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	301 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	302 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	303 iterCount += 1
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	304 if iterCount > 10:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	305 raise ValueError('stuck')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	306
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	307 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	308
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	309 yield BoundLhs(self, lastRing.currentBinding())
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	310
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	311 self._debugStmtStack('odometer', stmtStack)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	312
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	313 done = self._advanceAll(stmtStack)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	314
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	315 self._debugStmtStack('odometer after ({done=})', stmtStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	316
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	317 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	318 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	319 break
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	320
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	321 def _debugStmtStack(self, label, stmtStack):
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	322 log.debug(f'{INDENT*5} {label}:')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	323 for l in stmtStack:
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	324 log.debug(f'{INDENT*6} {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	325
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	326 def _assembleRings(self, knownTrue: ReadOnlyWorkingSet) -> List[StmtLooper]:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	327 """make StmtLooper for each stmt in our LHS graph, but do it in a way that they all
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	328 start out valid (or else raise NoOptions)"""
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	329
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	330 usedByFuncs: Set[Triple] = set() # don't worry about matching these
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	331 stmtsToResolve = list(self.graph)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	332 for i, s in enumerate(stmtsToResolve):
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	333 if s[1] == MATH['sum']:
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	334 _, used = parseList(self.graph, s[0])
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	335 usedByFuncs.update(used)
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	336
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	337 stmtsToAdd = [stmt for stmt in stmtsToResolve if not stmt in usedByFuncs]
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	338
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	339 # sort them by variable dependencies; don't just try all perms!
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	340 def lightSortKey(stmt): # Not this. Though it helps performance on the big rdf list cases.
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	341 (s, p, o) = stmt
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	342 return p == MATH['sum'], p, s, o
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	343
ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	344 stmtsToAdd.sort(key=lightSortKey)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	345
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	346 for perm in itertools.permutations(stmtsToAdd):
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	347 stmtStack: List[StmtLooper] = []
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	348 prev: Optional[StmtLooper] = None
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	349 log.debug(f'{INDENT*5} try stmts in this order: {" -> ".join(graphDump([p]) for p in perm)}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	350
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	351 for s in perm:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	352 try:
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	353 elem = StmtLooper(s, prev, knownTrue, parent=self)
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	354 except NoOptions:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	355 log.debug(f'{INDENT*6} permutation didnt work, try another')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	356 break
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	357 stmtStack.append(elem)
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	358 prev = stmtStack[-1]
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	359 else:
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	360 return stmtStack
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	361 log.debug(f'{INDENT*6} no perms worked- rule cannot match anything')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	362
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	363 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	364
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	365 def _advanceAll(self, stmtStack: List[StmtLooper]) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	366 carry = True # 1st elem always must advance
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	367 for i, ring in enumerate(stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	368 # unlike normal odometer, advancing any earlier ring could invalidate later ones
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	369 if carry:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	370 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} carry/advance')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	371 ring.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	372 carry = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	373 if ring.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	374 if ring is stmtStack[-1]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	375 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} says we done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	376 return True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	377 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} restart')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	378 ring.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	379 carry = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	380 return False
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	381
1633 6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	382 def _assertAllRingsAreValid(self, stmtStack):
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	383 if any(ring.pastEnd() for ring in stmtStack): # this is an unexpected debug assertion
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	384 log.debug(f'{INDENT*5} some rings started at pastEnd {stmtStack}')
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	385 raise NoOptions()
6107603ed455 fix farenheit rule case, fix some others that depend on rings order, but this breaks some performance because of itertools.perm drewp@bigasterisk.com parents: 1632 diff changeset	386
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	387 def _allStaticStatementsMatch(self, knownTrue: ReadOnlyWorkingSet) -> bool:
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	388 # bug: see TestSelfFulfillingRule.test3 for a case where this rule's
03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	389 # static stmt is matched by a non-static stmt in the rule itself
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	390 for ruleStmt in self.staticRuleStmts:
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	391 if ruleStmt not in knownTrue:
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	392 log.debug(f'{INDENT*3} {ruleStmt} not in working set- skip rule')
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	393 return False
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	394 return True
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	395
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	396 def _possibleBindings(self, workingSet, stats) -> Iterator['BoundLhs']:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	397 """this yields at least the working bindings, and possibly others"""
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	398 candidateTermMatches: Dict[BindableTerm, Set[Node]] = self._allCandidateTermMatches(workingSet)
1614 97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	399 for bindRow in self._product(candidateTermMatches):
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	400 try:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	401 yield BoundLhs(self, bindRow)
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	402 except EvaluationFailed:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	403 stats['permCountFailingEval'] += 1
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	404
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	405 def _allCandidateTermMatches(self, workingSet: ReadOnlyWorkingSet) -> Dict[BindableTerm, Set[Node]]:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	406 """the total set of terms each variable could possibly match"""
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	407
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	408 candidateTermMatches: Dict[BindableTerm, Set[Node]] = defaultdict(set)
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	409 for lhsStmt in self.graph:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	410 log.debug(f'{INDENT*4} possibles for this lhs stmt: {lhsStmt}')
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	411 for i, trueStmt in enumerate(workingSet):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	412 # log.debug(f'{INDENT*5} consider this true stmt ({i}): {trueStmt}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	413
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	414 for v, vals in self._bindingsFromStatement(lhsStmt, trueStmt):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	415 candidateTermMatches[v].update(vals)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	416
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	417 return candidateTermMatches
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	418
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	419 def _bindingsFromStatement(self, stmt1: Triple, stmt2: Triple) -> Iterator[Tuple[Variable, Set[Node]]]:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	420 """if these stmts match otherwise, what BNode or Variable mappings do we learn?
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	421
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	422 e.g. stmt1=(?x B ?y) and stmt2=(A B C), then we yield (?x, {A}) and (?y, {C})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	423 or stmt1=(_:x B C) and stmt2=(A B C), then we yield (_:x, {A})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	424 or stmt1=(?x B C) and stmt2=(A B D), then we yield nothing
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	425 """
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	426 bindingsFromStatement = {}
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	427 for term1, term2 in zip(stmt1, stmt2):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	428 if isinstance(term1, (BNode, Variable)):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	429 bindingsFromStatement.setdefault(term1, set()).add(term2)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	430 elif term1 != term2:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	431 break
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	432 else:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	433 for v, vals in bindingsFromStatement.items():
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	434 log.debug(f'{INDENT*5} {v=} {vals=}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	435 yield v, vals
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	436
1627 ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	437 def _product(self, candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Iterator[CandidateBinding]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	438 orderedVars, orderedValueSets = _organize(candidateTermMatches)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	439 self._logCandidates(orderedVars, orderedValueSets)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	440 log.debug(f'{INDENT*3} trying all permutations:')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	441 if not orderedValueSets:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	442 yield CandidateBinding({})
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	443 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	444
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	445 if not orderedValueSets or not all(orderedValueSets):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	446 # some var or bnode has no options at all
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	447 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	448 rings: List[Iterator[Node]] = [itertools.cycle(valSet) for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	449 currentSet: List[Node] = [next(ring) for ring in rings]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	450 starts = [valSet[-1] for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	451 while True:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	452 for col, curr in enumerate(currentSet):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	453 currentSet[col] = next(rings[col])
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	454 log.debug(f'{INDENT*4} currentSet: {repr(currentSet)}')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	455 yield CandidateBinding(dict(zip(orderedVars, currentSet)))
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	456 if curr is not starts[col]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	457 break
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	458 if col == len(orderedValueSets) - 1:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	459 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	460
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	461 def _logCandidates(self, orderedVars, orderedValueSets):
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	462 if not log.isEnabledFor(logging.DEBUG):
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	463 return
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	464 log.debug(f'{INDENT*3} resulting candidate terms:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	465 for v, vals in zip(orderedVars, orderedValueSets):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	466 log.debug(f'{INDENT*4} {v!r} could be:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	467 for val in vals:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	468 log.debug(f'{INDENT*5}{val!r}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	469
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	470
1622 38bd8ef9ef67 add CandidateTermMatches, unused so far drewp@bigasterisk.com parents: 1621 diff changeset	471 @dataclass
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	472 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	473 lhs: Lhs
1610 6fc48ef4c696 mysteriously lost an important line drewp@bigasterisk.com parents: 1609 diff changeset	474 binding: CandidateBinding
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	475
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	476 def __post_init__(self):
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	477 self.usedByFuncs = Graph()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	478 # self._applyFunctions()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	479
1616 3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	480 def lhsStmtsWithoutEvals(self):
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	481 for stmt in self.lhs.graph:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	482 if stmt in self.usedByFuncs:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	483 continue
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	484 yield stmt
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	485
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	486 def _applyFunctions(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	487 """may grow the binding with some results"""
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	488 while True:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	489 delta = self._applyFunctionsIteration()
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	490 if delta == 0:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	491 break
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	492
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	493 def _applyFunctionsIteration(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	494 before = len(self.binding.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	495 delta = 0
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	496 for ev in self.lhs.evaluations:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	497 newBindings, usedGraph = ev.resultBindings(self.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	498 self.usedByFuncs += usedGraph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	499 self.binding.addNewBindings(newBindings)
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	500 delta = len(self.binding.binding) - before
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	501 log.debug(f'{INDENT*4} eval rules made {delta} new bindings')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	502 return delta
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	503
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	504 def verify(self, workingSet: ReadOnlyWorkingSet) -> bool:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	505 """Can this bound lhs be true all at once in workingSet?"""
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	506 rem = cast(Set[Triple], self.lhs.nonStaticRuleStmtsSet.difference(self.usedByFuncs))
bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	507 boundLhs = self.binding.apply(rem)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	508
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	509 if log.isEnabledFor(logging.DEBUG):
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	510 boundLhs = list(boundLhs)
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	511 self._logVerifyBanner(boundLhs, workingSet)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	512
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	513 for stmt in boundLhs:
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	514 log.debug(f'{INDENT*4} check for %s', stmt)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	515
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	516 if stmt not in workingSet:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	517 log.debug(f'{INDENT*5} stmt not known to be true')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	518 return False
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	519 return True
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	520
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	521 def _logVerifyBanner(self, boundLhs, workingSet: ReadOnlyWorkingSet):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	522 log.debug(f'{INDENT*4}/ verify all bindings against this boundLhs:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	523 for stmt in sorted(boundLhs):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	524 log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	525
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	526 # log.debug(f'{INDENT*4}\| and against this workingSet:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	527 # for stmt in sorted(workingSet):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	528 # log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	529
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	530 log.debug(f'{INDENT*4}\\')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	531
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	532
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	533 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	534 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	535 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	536 rhsGraph: Graph
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	537
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	538 def __post_init__(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	539 self.lhs = Lhs(self.lhsGraph)
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	540 #
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	541 self.rhsBnodeMap = {}
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	542
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	543 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	544 for bound in self.lhs.findCandidateBindings(ReadOnlyGraphAggregate([workingSet]), stats):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	545 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	546
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	547 # rhs could have more bnodes, and they just need to be distinct per rule-firing that we do
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	548 existingRhsBnodes = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	549 for stmt in self.rhsGraph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	550 for t in stmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	551 if isinstance(t, BNode):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	552 existingRhsBnodes.add(t)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	553 # if existingRhsBnodes:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	554 # log.debug(f'{INDENT*6} mapping rhs bnodes {existingRhsBnodes} to new ones')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	555
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	556 for b in existingRhsBnodes:
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	557
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	558 key = tuple(sorted(bound.binding.binding.items())), b
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	559 self.rhsBnodeMap.setdefault(key, BNode())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	560
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	561 bound.binding.addNewBindings(CandidateBinding({b: self.rhsBnodeMap[key]}))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	562
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	563 # for lhsBoundStmt in bound.binding.apply(bound.lhsStmtsWithoutEvals()):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	564 # log.debug(f'{INDENT*6} adding to workingSet {lhsBoundStmt=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	565 # workingSet.add(lhsBoundStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	566 # log.debug(f'{INDENT*6} rhsGraph is good: {list(self.rhsGraph)}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	567
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	568 for newStmt in bound.binding.apply(self.rhsGraph):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	569 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	570 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	571 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	572
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	573
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	574 class Inference:
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	575
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	576 def __init__(self) -> None:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	577 self.rules = []
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	578
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	579 def setRules(self, g: ConjunctiveGraph):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	580 self.rules: List[Rule] = []
1599 abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	581 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	582 if stmt[1] == LOG['implies']:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	583 self.rules.append(Rule(stmt[0], stmt[2]))
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	584 # other stmts should go to a default working set?
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	585
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	586 @INFER_CALLS.time()
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	587 def infer(self, graph: Graph):
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	588 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	589 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	590 """
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	591 n = graph.__len__()
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	592 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	593 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	594 startTime = time.time()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	595 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	596 # everything that is true: the input graph, plus every rule conclusion we can make
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	597 workingSet = Graph()
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	598 workingSet += graph
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	599
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	600 # just the statements that came from RHS's of rules that fired.
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	601 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	602
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	603 bailout_iterations = 100
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	604 delta = 1
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	605 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	606 while delta > 0 and bailout_iterations > 0:
1620 92f8deb59735 log layout drewp@bigasterisk.com parents: 1616 diff changeset	607 log.debug('')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	608 log.info(f'{INDENT1}iteration ({bailout_iterations} left)')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	609 bailout_iterations -= 1
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	610 delta = -len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	611 self._iterateAllRules(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	612 delta += len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	613 stats['iterations'] += 1
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	614 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	615 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	616 stats['impliedStmts'] = len(implied)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	617 log.info(f'{INDENT*0} Inference done {dict(stats)}. Implied:')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	618 for st in implied:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	619 log.info(f'{INDENT*1} {st}')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	620 return implied
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	621
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	622 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats):
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	623 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	624 self._logRuleApplicationHeader(workingSet, i, rule)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	625 rule.applyRule(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	626
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	627 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	628 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	629 return
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	630
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	631 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	632 log.debug(f'{INDENT*2} workingSet:')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	633 for j, stmt in enumerate(sorted(workingSet)):
387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	634 log.debug(f'{INDENT*3} ({j}) {stmt}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	635
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	636 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	637 log.debug(f'{INDENT*2}-applying rule {i}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	638 log.debug(f'{INDENT*3} rule def lhs:')
1634 ba59cfc3c747 hack math:sum in there. Test suite is passing except some slow performers drewp@bigasterisk.com parents: 1633 diff changeset	639 for stmt in sorted(r.lhsGraph, reverse=True):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	640 log.debug(f'{INDENT*4} {stmt}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	641 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	642
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	643
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	644 def graphDump(g: Union[Graph, List[Triple]]):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	645 if not isinstance(g, Graph):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	646 g2 = Graph()
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	647 g2 += g
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	648 g = g2
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	649 g.bind('', ROOM)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	650 g.bind('ex', Namespace('http://example.com/'))
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	651 lines = cast(bytes, g.serialize(format='n3')).decode('utf8').splitlines()
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	652 lines = [line.strip() for line in lines if not line.startswith('@prefix')]
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	653 return ' '.join(lines)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	654
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	655
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	656 def _organize(candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Tuple[List[BindableTerm], List[List[Node]]]:
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	657 items = list(candidateTermMatches.items())
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	658 items.sort()
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	659 orderedVars: List[BindableTerm] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	660 orderedValueSets: List[List[Node]] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	661 for v, vals in items:
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	662 orderedVars.append(v)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	663 orderedValues: List[Node] = list(vals)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	664 orderedValues.sort(key=str)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	665 orderedValueSets.append(orderedValues)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	666
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	667 return orderedVars, orderedValueSets

Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1634:ba59cfc3c747