homeauto: service/mqtt_to_rdf/inference.py annotate

annotate service/mqtt_to_rdf/inference.py @ 1632:bd79a2941cab

just (a lot of) debug changes

author	drewp@bigasterisk.com
date	Sun, 12 Sep 2021 21:46:39 -0700
parents	2c85a4f5dd9c
children	6107603ed455

rev	line source
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	1 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	4 """
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	5 import itertools
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	6 import logging
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	7 import time
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	8 from collections import defaultdict
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	9 from dataclasses import dataclass
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	10 from typing import Dict, Iterator, List, Optional, Set, Tuple, Union, cast
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	11
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	12 from prometheus_client import Histogram, Summary
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	13 from rdflib import BNode, Graph, Namespace
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	14 from rdflib.graph import ConjunctiveGraph, ReadOnlyGraphAggregate
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	15 from rdflib.term import Literal, Node, Variable
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	16
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	17 from candidate_binding import CandidateBinding
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	18 from inference_types import (BindableTerm, EvaluationFailed, ReadOnlyWorkingSet, Triple)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	19 from lhs_evaluation import Decimal, Evaluation, numericNode
1605 449746d1598f WIP move evaluation to new file drewp@bigasterisk.com parents: 1603 diff changeset	20
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	21 log = logging.getLogger('infer')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	22 INDENT = ' '
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	23
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	24 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	25 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	26
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	27 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	28 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	29 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	30
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	31
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	32 def stmtTemplate(stmt: Triple) -> Tuple[Optional[Node], Optional[Node], Optional[Node]]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	33 return (
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	34 None if isinstance(stmt[0], (Variable, BNode)) else stmt[0],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	35 None if isinstance(stmt[1], (Variable, BNode)) else stmt[1],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	36 None if isinstance(stmt[2], (Variable, BNode)) else stmt[2],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	37 )
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	38
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	39
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	40 class NoOptions(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	41 """stmtlooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	42
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	43
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	44 class Inconsistent(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	45 """adding this stmt would be inconsistent with an existing binding"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	46
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	47
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	48 _stmtLooperShortId = itertools.count()
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	49
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	50
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	51 @dataclass
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	52 class StmtLooper:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	53 """given one LHS stmt, iterate through the possible matches for it,
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	54 returning what bindings they would imply. Only distinct bindings are
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	55 returned. The bindings build on any `prev` StmtLooper's results.
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	56
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	57 This iterator is restartable."""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	58 lhsStmt: Triple
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	59 prev: Optional['StmtLooper']
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	60 workingSet: ReadOnlyWorkingSet
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	61
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	62 def __repr__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	63 return f'StmtLooper{self._shortId}({graphDump([self.lhsStmt])} {"<pastEnd>" if self.pastEnd() else ""})'
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	64
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	65 def __post_init__(self):
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	66 self._shortId = next(_stmtLooperShortId)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	67 self._myWorkingSetMatches = self._myMatches(self.workingSet)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	68
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	69 self._current = CandidateBinding({})
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	70 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	71 self._seenBindings: List[Dict[BindableTerm, Node]] = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	72 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	73
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	74 def _myMatches(self, g: Graph) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	75 template = stmtTemplate(self.lhsStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	76
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	77 stmts = sorted(cast(Iterator[Triple], list(g.triples(template))))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	78 # plus new lhs possibilties...
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	79 # log.debug(f'{INDENT*6} {self} find {len(stmts)=} in {len(self.workingSet)=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	80
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	81 return stmts
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	82
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	83 def _prevBindings(self) -> Dict[BindableTerm, Node]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	84 if not self.prev or self.prev.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	85 return {}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	86
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	87 return self.prev.currentBinding().binding
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	88
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	89 def advance(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	90 """update to a new set of bindings we haven't seen (since last restart), or go into pastEnd mode"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	91 log.debug(f'{INDENT*6} {self} mines {len(self._myWorkingSetMatches)} matching statements')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	92 for i, stmt in enumerate(self._myWorkingSetMatches):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	93 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	94 outBinding = self._totalBindingIfThisStmtWereTrue(stmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	95 except Inconsistent:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	96 log.debug(f'{INDENT*7} {self} - {stmt} would be inconsistent with prev bindings')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	97 continue
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	98
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	99 log.debug(f'{INDENT*6} {outBinding=} {self._seenBindings=}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	100 if outBinding.binding not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	101 self._seenBindings.append(outBinding.binding.copy())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	102 self._current = outBinding
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	103 log.debug(f'{INDENT*7} new binding from {self} -> {outBinding}')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	104 return
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	105 log.debug(f'yes we saw')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	106
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	107 log.debug(f'{INDENT*6} {self} mines rules')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	108
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	109 if self.lhsStmt[1] == ROOM['asFarenheit']:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	110 pb: Dict[BindableTerm, Node] = self._prevBindings()
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	111 log.debug(f'{INDENT*6} {self} consider ?x faren ?y where ?x={self.lhsStmt[0]} and {pb=}')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	112
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	113 if self.lhsStmt[0] in pb:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	114 operands = [pb[cast(BindableTerm, self.lhsStmt[0])]]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	115 f = cast(Literal, Literal(Decimal(numericNode(operands[0])) * 9 / 5 + 32))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	116 objVar = self.lhsStmt[2]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	117 if not isinstance(objVar, Variable):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	118 raise TypeError(f'expected Variable, got {objVar!r}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	119 newBindings = {cast(BindableTerm, objVar): cast(Node, f)}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	120 self._current.addNewBindings(CandidateBinding(newBindings))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	121 if newBindings not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	122 self._seenBindings.append(newBindings)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	123 self._current = CandidateBinding(newBindings)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	124
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	125 log.debug(f'{INDENT*6} {self} is past end')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	126 self._pastEnd = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	127
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	128 def _totalBindingIfThisStmtWereTrue(self, newStmt: Triple) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	129 outBinding = self._prevBindings().copy()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	130 for rt, ct in zip(self.lhsStmt, newStmt):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	131 if isinstance(rt, (Variable, BNode)):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	132 if rt in outBinding and outBinding[rt] != ct:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	133 raise Inconsistent()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	134 outBinding[rt] = ct
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	135 return CandidateBinding(outBinding)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	136
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	137 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	138 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	139 raise NotImplementedError()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	140 return self._current
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	141
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	142 def newLhsStmts(self) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	143 """under the curent bindings, what new stmts beyond workingSet are also true? includes all `prev`"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	144 return []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	145
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	146 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	147 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	148
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	149 def restart(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	150 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	151 self._seenBindings = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	152 self.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	153 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	154 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	155
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	156
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	157 @dataclass
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	158 class Lhs:
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	159 graph: Graph
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	160
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	161 def __post_init__(self):
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	162 # do precomputation in here that's not specific to the workingSet
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	163 # self.staticRuleStmts = Graph()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	164 # self.nonStaticRuleStmts = Graph()
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	165
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	166 # self.lhsBindables: Set[BindableTerm] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	167 # self.lhsBnodes: Set[BNode] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	168 # for ruleStmt in self.graph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	169 # varsAndBnodesInStmt = [term for term in ruleStmt if isinstance(term, (Variable, BNode))]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	170 # self.lhsBindables.update(varsAndBnodesInStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	171 # self.lhsBnodes.update(x for x in varsAndBnodesInStmt if isinstance(x, BNode))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	172 # if not varsAndBnodesInStmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	173 # self.staticRuleStmts.add(ruleStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	174 # else:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	175 # self.nonStaticRuleStmts.add(ruleStmt)
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	176
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	177 # self.nonStaticRuleStmtsSet = set(self.nonStaticRuleStmts)
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	178
1602 e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	179 self.evaluations = list(Evaluation.findEvals(self.graph))
e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	180
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	181 def __repr__(self):
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	182 return f"Lhs({graphDump(self.graph)})"
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	183
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	184 def findCandidateBindings(self, knownTrue: ReadOnlyWorkingSet, stats) -> Iterator['BoundLhs']:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	185 """bindings that fit the LHS of a rule, using statements from workingSet and functions
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	186 from LHS"""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	187 log.debug(f'{INDENT*4} build new StmtLooper stack')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	188
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	189 stmtStack: List[StmtLooper] = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	190 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	191 prev: Optional[StmtLooper] = None
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	192 for s in sorted(self.graph): # order of this matters! :(
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	193 stmtStack.append(StmtLooper(s, prev, knownTrue))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	194 prev = stmtStack[-1]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	195 except NoOptions:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	196 log.debug(f'{INDENT*5} start up with no options; 0 bindings')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	197 return
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	198 self._debugStmtStack('initial odometer', stmtStack)
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	199
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	200
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	201 if any(ring.pastEnd() for ring in stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	202 log.debug(f'{INDENT*5} some rings started at pastEnd {stmtStack}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	203
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	204 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	205 sl = stmtStack[-1]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	206 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	207 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	208 iterCount += 1
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	209 if iterCount > 10:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	210 raise ValueError('stuck')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	211
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	212 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	213
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	214 yield BoundLhs(self, sl.currentBinding())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	215
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	216 self._debugStmtStack('odometer', stmtStack)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	217
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	218 done = self._advanceAll(stmtStack)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	219
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	220 self._debugStmtStack('odometer after ({done=})', stmtStack)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	221
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	222 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	223 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	224 break
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	225
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	226 def _debugStmtStack(self, label, stmtStack):
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	227 log.debug(f'{INDENT*5} {label}:')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	228 for l in stmtStack:
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	229 log.debug(f'{INDENT*6} {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	230
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	231 def _advanceAll(self, stmtStack: List[StmtLooper]) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	232 carry = True # 1st elem always must advance
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	233 for i, ring in enumerate(stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	234 # unlike normal odometer, advancing any earlier ring could invalidate later ones
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	235 if carry:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	236 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} carry/advance')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	237 ring.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	238 carry = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	239 if ring.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	240 if ring is stmtStack[-1]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	241 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} says we done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	242 return True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	243 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} restart')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	244 ring.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	245 carry = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	246 return False
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	247
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	248 def _allStaticStatementsMatch(self, knownTrue: ReadOnlyWorkingSet) -> bool:
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	249 # bug: see TestSelfFulfillingRule.test3 for a case where this rule's
03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	250 # static stmt is matched by a non-static stmt in the rule itself
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	251 for ruleStmt in self.staticRuleStmts:
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	252 if ruleStmt not in knownTrue:
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	253 log.debug(f'{INDENT*3} {ruleStmt} not in working set- skip rule')
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	254 return False
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	255 return True
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	256
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	257 def _possibleBindings(self, workingSet, stats) -> Iterator['BoundLhs']:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	258 """this yields at least the working bindings, and possibly others"""
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	259 candidateTermMatches: Dict[BindableTerm, Set[Node]] = self._allCandidateTermMatches(workingSet)
1614 97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	260 for bindRow in self._product(candidateTermMatches):
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	261 try:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	262 yield BoundLhs(self, bindRow)
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	263 except EvaluationFailed:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	264 stats['permCountFailingEval'] += 1
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	265
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	266 def _allCandidateTermMatches(self, workingSet: ReadOnlyWorkingSet) -> Dict[BindableTerm, Set[Node]]:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	267 """the total set of terms each variable could possibly match"""
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	268
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	269 candidateTermMatches: Dict[BindableTerm, Set[Node]] = defaultdict(set)
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	270 for lhsStmt in self.graph:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	271 log.debug(f'{INDENT*4} possibles for this lhs stmt: {lhsStmt}')
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	272 for i, trueStmt in enumerate(workingSet):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	273 # log.debug(f'{INDENT*5} consider this true stmt ({i}): {trueStmt}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	274
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	275 for v, vals in self._bindingsFromStatement(lhsStmt, trueStmt):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	276 candidateTermMatches[v].update(vals)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	277
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	278 return candidateTermMatches
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	279
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	280 def _bindingsFromStatement(self, stmt1: Triple, stmt2: Triple) -> Iterator[Tuple[Variable, Set[Node]]]:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	281 """if these stmts match otherwise, what BNode or Variable mappings do we learn?
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	282
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	283 e.g. stmt1=(?x B ?y) and stmt2=(A B C), then we yield (?x, {A}) and (?y, {C})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	284 or stmt1=(_:x B C) and stmt2=(A B C), then we yield (_:x, {A})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	285 or stmt1=(?x B C) and stmt2=(A B D), then we yield nothing
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	286 """
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	287 bindingsFromStatement = {}
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	288 for term1, term2 in zip(stmt1, stmt2):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	289 if isinstance(term1, (BNode, Variable)):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	290 bindingsFromStatement.setdefault(term1, set()).add(term2)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	291 elif term1 != term2:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	292 break
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	293 else:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	294 for v, vals in bindingsFromStatement.items():
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	295 log.debug(f'{INDENT*5} {v=} {vals=}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	296 yield v, vals
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	297
1627 ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	298 def _product(self, candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Iterator[CandidateBinding]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	299 orderedVars, orderedValueSets = _organize(candidateTermMatches)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	300 self._logCandidates(orderedVars, orderedValueSets)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	301 log.debug(f'{INDENT*3} trying all permutations:')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	302 if not orderedValueSets:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	303 yield CandidateBinding({})
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	304 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	305
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	306 if not orderedValueSets or not all(orderedValueSets):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	307 # some var or bnode has no options at all
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	308 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	309 rings: List[Iterator[Node]] = [itertools.cycle(valSet) for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	310 currentSet: List[Node] = [next(ring) for ring in rings]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	311 starts = [valSet[-1] for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	312 while True:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	313 for col, curr in enumerate(currentSet):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	314 currentSet[col] = next(rings[col])
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	315 log.debug(f'{INDENT*4} currentSet: {repr(currentSet)}')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	316 yield CandidateBinding(dict(zip(orderedVars, currentSet)))
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	317 if curr is not starts[col]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	318 break
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	319 if col == len(orderedValueSets) - 1:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	320 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	321
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	322 def _logCandidates(self, orderedVars, orderedValueSets):
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	323 if not log.isEnabledFor(logging.DEBUG):
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	324 return
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	325 log.debug(f'{INDENT*3} resulting candidate terms:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	326 for v, vals in zip(orderedVars, orderedValueSets):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	327 log.debug(f'{INDENT*4} {v!r} could be:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	328 for val in vals:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	329 log.debug(f'{INDENT*5}{val!r}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	330
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	331
1622 38bd8ef9ef67 add CandidateTermMatches, unused so far drewp@bigasterisk.com parents: 1621 diff changeset	332 @dataclass
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	333 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	334 lhs: Lhs
1610 6fc48ef4c696 mysteriously lost an important line drewp@bigasterisk.com parents: 1609 diff changeset	335 binding: CandidateBinding
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	336
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	337 def __post_init__(self):
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	338 self.usedByFuncs = Graph()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	339 # self._applyFunctions()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	340
1616 3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	341 def lhsStmtsWithoutEvals(self):
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	342 for stmt in self.lhs.graph:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	343 if stmt in self.usedByFuncs:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	344 continue
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	345 yield stmt
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	346
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	347 def _applyFunctions(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	348 """may grow the binding with some results"""
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	349 while True:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	350 delta = self._applyFunctionsIteration()
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	351 if delta == 0:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	352 break
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	353
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	354 def _applyFunctionsIteration(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	355 before = len(self.binding.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	356 delta = 0
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	357 for ev in self.lhs.evaluations:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	358 newBindings, usedGraph = ev.resultBindings(self.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	359 self.usedByFuncs += usedGraph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	360 self.binding.addNewBindings(newBindings)
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	361 delta = len(self.binding.binding) - before
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	362 log.debug(f'{INDENT*4} eval rules made {delta} new bindings')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	363 return delta
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	364
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	365 def verify(self, workingSet: ReadOnlyWorkingSet) -> bool:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	366 """Can this bound lhs be true all at once in workingSet?"""
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	367 rem = cast(Set[Triple], self.lhs.nonStaticRuleStmtsSet.difference(self.usedByFuncs))
bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	368 boundLhs = self.binding.apply(rem)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	369
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	370 if log.isEnabledFor(logging.DEBUG):
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	371 boundLhs = list(boundLhs)
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	372 self._logVerifyBanner(boundLhs, workingSet)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	373
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	374 for stmt in boundLhs:
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	375 log.debug(f'{INDENT*4} check for %s', stmt)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	376
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	377 if stmt not in workingSet:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	378 log.debug(f'{INDENT*5} stmt not known to be true')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	379 return False
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	380 return True
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	381
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	382 def _logVerifyBanner(self, boundLhs, workingSet: ReadOnlyWorkingSet):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	383 log.debug(f'{INDENT*4}/ verify all bindings against this boundLhs:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	384 for stmt in sorted(boundLhs):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	385 log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	386
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	387 # log.debug(f'{INDENT*4}\| and against this workingSet:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	388 # for stmt in sorted(workingSet):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	389 # log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	390
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	391 log.debug(f'{INDENT*4}\\')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	392
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	393
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	394 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	395 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	396 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	397 rhsGraph: Graph
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	398
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	399 def __post_init__(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	400 self.lhs = Lhs(self.lhsGraph)
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	401 #
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	402 self.rhsBnodeMap = {}
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	403
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	404 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	405 for bound in self.lhs.findCandidateBindings(ReadOnlyGraphAggregate([workingSet]), stats):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	406 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	407
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	408 # rhs could have more bnodes, and they just need to be distinct per rule-firing that we do
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	409 existingRhsBnodes = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	410 for stmt in self.rhsGraph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	411 for t in stmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	412 if isinstance(t, BNode):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	413 existingRhsBnodes.add(t)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	414 # if existingRhsBnodes:
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	415 # log.debug(f'{INDENT*6} mapping rhs bnodes {existingRhsBnodes} to new ones')
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	416
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	417 for b in existingRhsBnodes:
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	418
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	419 key = tuple(sorted(bound.binding.binding.items())), b
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	420 self.rhsBnodeMap.setdefault(key, BNode())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	421
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	422 bound.binding.addNewBindings(CandidateBinding({b: self.rhsBnodeMap[key]}))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	423
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	424 # for lhsBoundStmt in bound.binding.apply(bound.lhsStmtsWithoutEvals()):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	425 # log.debug(f'{INDENT*6} adding to workingSet {lhsBoundStmt=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	426 # workingSet.add(lhsBoundStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	427 # log.debug(f'{INDENT*6} rhsGraph is good: {list(self.rhsGraph)}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	428
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	429 for newStmt in bound.binding.apply(self.rhsGraph):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	430 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	431 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	432 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	433
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	434
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	435 class Inference:
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	436
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	437 def __init__(self) -> None:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	438 self.rules = []
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	439
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	440 def setRules(self, g: ConjunctiveGraph):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	441 self.rules: List[Rule] = []
1599 abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	442 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	443 if stmt[1] == LOG['implies']:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	444 self.rules.append(Rule(stmt[0], stmt[2]))
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	445 # other stmts should go to a default working set?
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	446
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	447 @INFER_CALLS.time()
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	448 def infer(self, graph: Graph):
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	449 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	450 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	451 """
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	452 n = graph.__len__()
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	453 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	454 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	455 startTime = time.time()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	456 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	457 # everything that is true: the input graph, plus every rule conclusion we can make
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	458 workingSet = Graph()
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	459 workingSet += graph
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	460
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	461 # just the statements that came from RHS's of rules that fired.
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	462 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	463
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	464 bailout_iterations = 100
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	465 delta = 1
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	466 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	467 while delta > 0 and bailout_iterations > 0:
1620 92f8deb59735 log layout drewp@bigasterisk.com parents: 1616 diff changeset	468 log.debug('')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	469 log.info(f'{INDENT1}iteration ({bailout_iterations} left)')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	470 bailout_iterations -= 1
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	471 delta = -len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	472 self._iterateAllRules(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	473 delta += len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	474 stats['iterations'] += 1
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	475 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	476 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	477 stats['impliedStmts'] = len(implied)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	478 log.info(f'{INDENT*0} Inference done {dict(stats)}. Implied:')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	479 for st in implied:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	480 log.info(f'{INDENT*1} {st}')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	481 return implied
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	482
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	483 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats):
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	484 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	485 self._logRuleApplicationHeader(workingSet, i, rule)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	486 rule.applyRule(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	487
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	488 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	489 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	490 return
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	491
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	492 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	493 log.debug(f'{INDENT*2} workingSet:')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	494 for j, stmt in enumerate(sorted(workingSet)):
387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	495 log.debug(f'{INDENT*3} ({j}) {stmt}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	496
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	497 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	498 log.debug(f'{INDENT*2}-applying rule {i}')
1632 bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	499 log.debug(f'{INDENT*3} rule def lhs:')
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	500 for stmt in r.lhsGraph:
bd79a2941cab just (a lot of) debug changes drewp@bigasterisk.com parents: 1631 diff changeset	501 log.debug(f'{INDENT*4} {stmt}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	502 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	503
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	504
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	505 def graphDump(g: Union[Graph, List[Triple]]):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	506 if not isinstance(g, Graph):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	507 g2 = Graph()
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	508 g2 += g
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	509 g = g2
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	510 g.bind('', ROOM)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	511 g.bind('ex', Namespace('http://example.com/'))
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	512 lines = cast(bytes, g.serialize(format='n3')).decode('utf8').splitlines()
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	513 lines = [line.strip() for line in lines if not line.startswith('@prefix')]
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	514 return ' '.join(lines)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	515
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	516
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	517 def _organize(candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Tuple[List[BindableTerm], List[List[Node]]]:
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	518 items = list(candidateTermMatches.items())
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	519 items.sort()
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	520 orderedVars: List[BindableTerm] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	521 orderedValueSets: List[List[Node]] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	522 for v, vals in items:
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	523 orderedVars.append(v)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	524 orderedValues: List[Node] = list(vals)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	525 orderedValues.sort(key=str)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	526 orderedValueSets.append(orderedValues)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	527
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	528 return orderedVars, orderedValueSets

Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1632:bd79a2941cab