homeauto: service/mqtt_to_rdf/inference.py annotate

annotate service/mqtt_to_rdf/inference.py @ 1631:2c85a4f5dd9c

big rewrite of infer() using statements not variables as the things to iterate over

author	drewp@bigasterisk.com
date	Sun, 12 Sep 2021 04:32:52 -0700
parents	ea559a846714
children	bd79a2941cab

rev	line source
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	1 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	2 copied from reasoning 2021-08-29. probably same api. should
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	3 be able to lib/ this out
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	4 """
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	5 import itertools
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	6 import logging
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	7 import time
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	8 from collections import defaultdict
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	9 from dataclasses import dataclass
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	10 from typing import Dict, Iterator, List, Optional, Set, Tuple, Union, cast
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	11
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	12 from prometheus_client import Histogram, Summary
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	13 from rdflib import BNode, Graph, Namespace
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	14 from rdflib.graph import ConjunctiveGraph, ReadOnlyGraphAggregate
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	15 from rdflib.term import Literal, Node, Variable
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	16
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	17 from candidate_binding import CandidateBinding
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	18 from inference_types import (BindableTerm, EvaluationFailed, ReadOnlyWorkingSet, Triple)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	19 from lhs_evaluation import Decimal, Evaluation, numericNode
1605 449746d1598f WIP move evaluation to new file drewp@bigasterisk.com parents: 1603 diff changeset	20
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	21 log = logging.getLogger('infer')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	22 INDENT = ' '
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	23
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	24 INFER_CALLS = Summary('inference_infer_calls', 'calls')
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	25 INFER_GRAPH_SIZE = Histogram('inference_graph_size', 'statements', buckets=[2**x for x in range(2, 20, 2)])
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	26
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	27 ROOM = Namespace("http://projects.bigasterisk.com/room/")
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	28 LOG = Namespace('http://www.w3.org/2000/10/swap/log#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	29 MATH = Namespace('http://www.w3.org/2000/10/swap/math#')
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	30
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	31
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	32 def stmtTemplate(stmt: Triple) -> Tuple[Optional[Node], Optional[Node], Optional[Node]]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	33 return (
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	34 None if isinstance(stmt[0], (Variable, BNode)) else stmt[0],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	35 None if isinstance(stmt[1], (Variable, BNode)) else stmt[1],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	36 None if isinstance(stmt[2], (Variable, BNode)) else stmt[2],
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	37 )
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	38
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	39
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	40 class NoOptions(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	41 """stmtlooper has no possibilites to add to the binding; the whole rule must therefore not apply"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	42
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	43
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	44 class Inconsistent(ValueError):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	45 """adding this stmt would be inconsistent with an existing binding"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	46
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	47
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	48 @dataclass
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	49 class StmtLooper:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	50 lhsStmt: Triple
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	51 prev: Optional['StmtLooper']
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	52 workingSet: ReadOnlyWorkingSet
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	53
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	54 def __repr__(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	55 return f'StmtLooper({graphDump([self.lhsStmt])} {"<pastEnd>" if self.pastEnd() else ""})'
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	56
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	57 def __post_init__(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	58 self._myWorkingSetMatches = self._myMatches(self.workingSet)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	59
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	60 self._current = CandidateBinding({})
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	61 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	62 self._seenBindings: List[Dict[BindableTerm, Node]] = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	63 self.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	64
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	65 def _myMatches(self, g: Graph) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	66 template = stmtTemplate(self.lhsStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	67
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	68 stmts = sorted(cast(Iterator[Triple], list(g.triples(template))))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	69 # plus new lhs possibilties...
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	70 # log.debug(f'{INDENT*6} {self} find {len(stmts)=} in {len(self.workingSet)=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	71
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	72 return stmts
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	73
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	74 def _prevBindings(self) -> Dict[BindableTerm, Node]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	75 if not self.prev or self.prev.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	76 return {}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	77
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	78 return self.prev.currentBinding().binding
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	79
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	80 def advance(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	81 """update to a new set of bindings we haven't seen (since last restart), or go into pastEnd mode"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	82 log.debug(f'{INDENT*6} {self} mines {len(self._myWorkingSetMatches)} matching statements')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	83 for i, stmt in enumerate(self._myWorkingSetMatches):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	84 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	85 outBinding = self._totalBindingIfThisStmtWereTrue(stmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	86 except Inconsistent:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	87 log.debug(f'{INDENT*7} {self} - {stmt} would be inconsistent with prev bindings')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	88 continue
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	89 log.debug(f'seen {outBinding.binding} in {self._seenBindings}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	90 if outBinding.binding not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	91 self._seenBindings.append(outBinding.binding.copy())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	92 log.debug(f'no, adding')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	93 self._current = outBinding
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	94 log.debug(f'{INDENT*7} {self} - Looper matches {stmt} which tells us {outBinding}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	95 return
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	96 log.debug(f'yes we saw')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	97
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	98 log.debug(f'{INDENT*6} {self} mines rules')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	99
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	100 if self.lhsStmt[1] == ROOM['asFarenheit']:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	101 pb: Dict[BindableTerm, Node] = self._prevBindings()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	102 if self.lhsStmt[0] in pb:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	103 operands = [pb[cast(BindableTerm, self.lhsStmt[0])]]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	104 f = cast(Literal, Literal(Decimal(numericNode(operands[0])) * 9 / 5 + 32))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	105 objVar = self.lhsStmt[2]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	106 if not isinstance(objVar, Variable):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	107 raise TypeError(f'expected Variable, got {objVar!r}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	108 newBindings = {cast(BindableTerm, objVar): cast(Node, f)}
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	109 self._current.addNewBindings(CandidateBinding(newBindings))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	110 if newBindings not in self._seenBindings:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	111 self._seenBindings.append(newBindings)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	112 self._current = CandidateBinding(newBindings)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	113
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	114 log.debug(f'{INDENT*6} {self} is past end')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	115 self._pastEnd = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	116
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	117 def _totalBindingIfThisStmtWereTrue(self, newStmt: Triple) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	118 outBinding = self._prevBindings().copy()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	119 for rt, ct in zip(self.lhsStmt, newStmt):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	120 if isinstance(rt, (Variable, BNode)):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	121 if rt in outBinding and outBinding[rt] != ct:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	122 raise Inconsistent()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	123 outBinding[rt] = ct
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	124 return CandidateBinding(outBinding)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	125
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	126 def currentBinding(self) -> CandidateBinding:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	127 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	128 raise NotImplementedError()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	129 return self._current
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	130
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	131 def newLhsStmts(self) -> List[Triple]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	132 """under the curent bindings, what new stmts beyond workingSet are also true? includes all `prev`"""
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	133 return []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	134
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	135 def pastEnd(self) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	136 return self._pastEnd
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	137
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	138 def restart(self):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	139 self._pastEnd = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	140 self._seenBindings = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	141 self.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	142 if self.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	143 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	144
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	145
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	146 @dataclass
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	147 class Lhs:
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	148 graph: Graph
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	149
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	150 def __post_init__(self):
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	151 # do precomputation in here that's not specific to the workingSet
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	152 # self.staticRuleStmts = Graph()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	153 # self.nonStaticRuleStmts = Graph()
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	154
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	155 # self.lhsBindables: Set[BindableTerm] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	156 # self.lhsBnodes: Set[BNode] = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	157 # for ruleStmt in self.graph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	158 # varsAndBnodesInStmt = [term for term in ruleStmt if isinstance(term, (Variable, BNode))]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	159 # self.lhsBindables.update(varsAndBnodesInStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	160 # self.lhsBnodes.update(x for x in varsAndBnodesInStmt if isinstance(x, BNode))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	161 # if not varsAndBnodesInStmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	162 # self.staticRuleStmts.add(ruleStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	163 # else:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	164 # self.nonStaticRuleStmts.add(ruleStmt)
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	165
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	166 # self.nonStaticRuleStmtsSet = set(self.nonStaticRuleStmts)
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	167
1602 e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	168 self.evaluations = list(Evaluation.findEvals(self.graph))
e3c44ac6d3c5 do findEvals once at setRules time drewp@bigasterisk.com parents: 1601 diff changeset	169
1609 34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	170 def __repr__(self):
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	171 return f"Lhs({graphDump(self.graph)})"
34f2817320cc new tests for a smaller part of the inner loop drewp@bigasterisk.com parents: 1608 diff changeset	172
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	173 def findCandidateBindings(self, knownTrue: ReadOnlyWorkingSet, stats) -> Iterator['BoundLhs']:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	174 """bindings that fit the LHS of a rule, using statements from workingSet and functions
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	175 from LHS"""
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	176 log.debug(f'{INDENT*4} build new StmtLooper stack')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	177
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	178 stmtStack: List[StmtLooper] = []
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	179 try:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	180 prev: Optional[StmtLooper] = None
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	181 for s in sorted(self.graph): # order of this matters! :(
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	182 stmtStack.append(StmtLooper(s, prev, knownTrue))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	183 prev = stmtStack[-1]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	184 except NoOptions:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	185 log.debug(f'{INDENT*5} no options; 0 bindings')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	186 return
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	187
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	188 log.debug(f'{INDENT*5} initial odometer:')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	189 for l in stmtStack:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	190 log.debug(f'{INDENT*6} {l}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	191
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	192 if any(ring.pastEnd() for ring in stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	193 log.debug(f'{INDENT*5} some rings started at pastEnd {stmtStack}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	194
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	195 raise NoOptions()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	196 sl = stmtStack[-1]
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	197 iterCount = 0
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	198 while True:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	199 iterCount += 1
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	200 if iterCount > 10:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	201 raise ValueError('stuck')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	202
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	203 log.debug(f'{INDENT*4} vv findCandBindings iteration {iterCount}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	204
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	205 log.debug(f'{INDENT*5} <<<')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	206 yield BoundLhs(self, sl.currentBinding())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	207 log.debug(f'{INDENT*5} >>>')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	208
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	209 log.debug(f'{INDENT*5} odometer:')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	210 for l in stmtStack:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	211 log.debug(f'{INDENT*6} {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	212
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	213 done = self._advanceAll(stmtStack)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	214
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	215 log.debug(f'{INDENT*5} odometer after ({done=}):')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	216 for l in stmtStack:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	217 log.debug(f'{INDENT*6} {l} curbind={l.currentBinding() if not l.pastEnd() else "<end>"}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	218
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	219 log.debug(f'{INDENT*4} ^^ findCandBindings iteration done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	220 if done:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	221 break
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	222
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	223 def _advanceAll(self, stmtStack: List[StmtLooper]) -> bool:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	224 carry = True # 1st elem always must advance
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	225 for i, ring in enumerate(stmtStack):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	226 # unlike normal odometer, advancing any earlier ring could invalidate later ones
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	227 if carry:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	228 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} carry/advance')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	229 ring.advance()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	230 carry = False
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	231 if ring.pastEnd():
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	232 if ring is stmtStack[-1]:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	233 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} says we done')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	234 return True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	235 log.debug(f'{INDENT*5} advanceAll [{i}] {ring} restart')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	236 ring.restart()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	237 carry = True
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	238 return False
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	239
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	240 def _allStaticStatementsMatch(self, knownTrue: ReadOnlyWorkingSet) -> bool:
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	241 # bug: see TestSelfFulfillingRule.test3 for a case where this rule's
03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	242 # static stmt is matched by a non-static stmt in the rule itself
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	243 for ruleStmt in self.staticRuleStmts:
1621 da235054caa0 rename workingSet, sometimes drewp@bigasterisk.com parents: 1620 diff changeset	244 if ruleStmt not in knownTrue:
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	245 log.debug(f'{INDENT*3} {ruleStmt} not in working set- skip rule')
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	246 return False
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	247 return True
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	248
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	249 def _possibleBindings(self, workingSet, stats) -> Iterator['BoundLhs']:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	250 """this yields at least the working bindings, and possibly others"""
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	251 candidateTermMatches: Dict[BindableTerm, Set[Node]] = self._allCandidateTermMatches(workingSet)
1614 97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	252 for bindRow in self._product(candidateTermMatches):
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	253 try:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	254 yield BoundLhs(self, bindRow)
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	255 except EvaluationFailed:
97b2c3cfdb83 refactor: inline an odometer algorithm in place of itertools.product drewp@bigasterisk.com parents: 1613 diff changeset	256 stats['permCountFailingEval'] += 1
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	257
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	258 def _allCandidateTermMatches(self, workingSet: ReadOnlyWorkingSet) -> Dict[BindableTerm, Set[Node]]:
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	259 """the total set of terms each variable could possibly match"""
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	260
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	261 candidateTermMatches: Dict[BindableTerm, Set[Node]] = defaultdict(set)
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	262 for lhsStmt in self.graph:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	263 log.debug(f'{INDENT*4} possibles for this lhs stmt: {lhsStmt}')
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	264 for i, trueStmt in enumerate(workingSet):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	265 # log.debug(f'{INDENT*5} consider this true stmt ({i}): {trueStmt}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	266
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	267 for v, vals in self._bindingsFromStatement(lhsStmt, trueStmt):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	268 candidateTermMatches[v].update(vals)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	269
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	270 return candidateTermMatches
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	271
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	272 def _bindingsFromStatement(self, stmt1: Triple, stmt2: Triple) -> Iterator[Tuple[Variable, Set[Node]]]:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	273 """if these stmts match otherwise, what BNode or Variable mappings do we learn?
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	274
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	275 e.g. stmt1=(?x B ?y) and stmt2=(A B C), then we yield (?x, {A}) and (?y, {C})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	276 or stmt1=(_:x B C) and stmt2=(A B C), then we yield (_:x, {A})
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	277 or stmt1=(?x B C) and stmt2=(A B D), then we yield nothing
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	278 """
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	279 bindingsFromStatement = {}
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	280 for term1, term2 in zip(stmt1, stmt2):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	281 if isinstance(term1, (BNode, Variable)):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	282 bindingsFromStatement.setdefault(term1, set()).add(term2)
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	283 elif term1 != term2:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	284 break
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	285 else:
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	286 for v, vals in bindingsFromStatement.items():
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	287 log.debug(f'{INDENT*5} {v=} {vals=}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	288 yield v, vals
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	289
1627 ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	290 def _product(self, candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Iterator[CandidateBinding]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	291 orderedVars, orderedValueSets = _organize(candidateTermMatches)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	292 self._logCandidates(orderedVars, orderedValueSets)
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	293 log.debug(f'{INDENT*3} trying all permutations:')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	294 if not orderedValueSets:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	295 yield CandidateBinding({})
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	296 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	297
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	298 if not orderedValueSets or not all(orderedValueSets):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	299 # some var or bnode has no options at all
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	300 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	301 rings: List[Iterator[Node]] = [itertools.cycle(valSet) for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	302 currentSet: List[Node] = [next(ring) for ring in rings]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	303 starts = [valSet[-1] for valSet in orderedValueSets]
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	304 while True:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	305 for col, curr in enumerate(currentSet):
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	306 currentSet[col] = next(rings[col])
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	307 log.debug(f'{INDENT*4} currentSet: {repr(currentSet)}')
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	308 yield CandidateBinding(dict(zip(orderedVars, currentSet)))
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	309 if curr is not starts[col]:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	310 break
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	311 if col == len(orderedValueSets) - 1:
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	312 return
ea559a846714 some shuffling, i don't know- i'm about to rewrite again drewp@bigasterisk.com parents: 1626 diff changeset	313
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	314 def _logCandidates(self, orderedVars, orderedValueSets):
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	315 if not log.isEnabledFor(logging.DEBUG):
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	316 return
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	317 log.debug(f'{INDENT*3} resulting candidate terms:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	318 for v, vals in zip(orderedVars, orderedValueSets):
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	319 log.debug(f'{INDENT*4} {v!r} could be:')
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	320 for val in vals:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	321 log.debug(f'{INDENT*5}{val!r}')
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	322
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	323
1622 38bd8ef9ef67 add CandidateTermMatches, unused so far drewp@bigasterisk.com parents: 1621 diff changeset	324 @dataclass
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	325 class BoundLhs:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	326 lhs: Lhs
1610 6fc48ef4c696 mysteriously lost an important line drewp@bigasterisk.com parents: 1609 diff changeset	327 binding: CandidateBinding
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	328
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	329 def __post_init__(self):
1613 03ed8c9abd5b forget GRAPH_ID optimization in this case drewp@bigasterisk.com parents: 1612 diff changeset	330 self.usedByFuncs = Graph()
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	331 # self._applyFunctions()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	332
1616 3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	333 def lhsStmtsWithoutEvals(self):
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	334 for stmt in self.lhs.graph:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	335 if stmt in self.usedByFuncs:
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	336 continue
3a6ed545357f optimization: stream stmts instead of building a Graph drewp@bigasterisk.com parents: 1615 diff changeset	337 yield stmt
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	338
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	339 def _applyFunctions(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	340 """may grow the binding with some results"""
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	341 while True:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	342 delta = self._applyFunctionsIteration()
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	343 if delta == 0:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	344 break
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	345
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	346 def _applyFunctionsIteration(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	347 before = len(self.binding.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	348 delta = 0
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	349 for ev in self.lhs.evaluations:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	350 newBindings, usedGraph = ev.resultBindings(self.binding)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	351 self.usedByFuncs += usedGraph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	352 self.binding.addNewBindings(newBindings)
1608 f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	353 delta = len(self.binding.binding) - before
f928eb06a4f6 cleaning up inner loop drewp@bigasterisk.com parents: 1607 diff changeset	354 log.debug(f'{INDENT*4} eval rules made {delta} new bindings')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	355 return delta
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	356
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	357 def verify(self, workingSet: ReadOnlyWorkingSet) -> bool:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	358 """Can this bound lhs be true all at once in workingSet?"""
1615 bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	359 rem = cast(Set[Triple], self.lhs.nonStaticRuleStmtsSet.difference(self.usedByFuncs))
bcfa368e5498 change a Graph.__sub__ to Set.difference in verify() for a big speedup drewp@bigasterisk.com parents: 1614 diff changeset	360 boundLhs = self.binding.apply(rem)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	361
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	362 if log.isEnabledFor(logging.DEBUG):
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	363 boundLhs = list(boundLhs)
a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	364 self._logVerifyBanner(boundLhs, workingSet)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	365
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	366 for stmt in boundLhs:
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	367 log.debug(f'{INDENT*4} check for %s', stmt)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	368
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	369 if stmt not in workingSet:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	370 log.debug(f'{INDENT*5} stmt not known to be true')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	371 return False
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	372 return True
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	373
1611 a794a150a89b a little inner-loop performace. same number of iterations, but less time on logging and some stmt filtering drewp@bigasterisk.com parents: 1610 diff changeset	374 def _logVerifyBanner(self, boundLhs, workingSet: ReadOnlyWorkingSet):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	375 log.debug(f'{INDENT*4}/ verify all bindings against this boundLhs:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	376 for stmt in sorted(boundLhs):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	377 log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	378
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	379 # log.debug(f'{INDENT*4}\| and against this workingSet:')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	380 # for stmt in sorted(workingSet):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	381 # log.debug(f'{INDENT*4}\|{INDENT} {stmt}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	382
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	383 log.debug(f'{INDENT*4}\\')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	384
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	385
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	386 @dataclass
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	387 class Rule:
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	388 lhsGraph: Graph
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	389 rhsGraph: Graph
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	390
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	391 def __post_init__(self):
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	392 self.lhs = Lhs(self.lhsGraph)
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	393 #
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	394 self.rhsBnodeMap = {}
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	395
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	396 def applyRule(self, workingSet: Graph, implied: Graph, stats: Dict):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	397 for bound in self.lhs.findCandidateBindings(ReadOnlyGraphAggregate([workingSet]), stats):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	398 log.debug(f'{INDENT*5} +rule has a working binding: {bound}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	399
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	400 # rhs could have more bnodes, and they just need to be distinct per rule-firing that we do
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	401 existingRhsBnodes = set()
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	402 for stmt in self.rhsGraph:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	403 for t in stmt:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	404 if isinstance(t, BNode):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	405 existingRhsBnodes.add(t)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	406 # if existingRhsBnodes:
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	407 # log.debug(f'{INDENT*6} mapping rhs bnodes {existingRhsBnodes} to new ones')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	408
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	409 for b in existingRhsBnodes:
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	410
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	411 key = tuple(sorted(bound.binding.binding.items())), b
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	412 self.rhsBnodeMap.setdefault(key, BNode())
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	413
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	414
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	415 bound.binding.addNewBindings(CandidateBinding({b: self.rhsBnodeMap[key]}))
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	416
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	417 # for lhsBoundStmt in bound.binding.apply(bound.lhsStmtsWithoutEvals()):
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	418 # log.debug(f'{INDENT*6} adding to workingSet {lhsBoundStmt=}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	419 # workingSet.add(lhsBoundStmt)
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	420 # log.debug(f'{INDENT*6} rhsGraph is good: {list(self.rhsGraph)}')
2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	421
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	422 for newStmt in bound.binding.apply(self.rhsGraph):
1631 2c85a4f5dd9c big rewrite of infer() using statements not variables as the things to iterate over drewp@bigasterisk.com parents: 1627 diff changeset	423 # log.debug(f'{INDENT*6} adding {newStmt=}')
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	424 workingSet.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	425 implied.add(newStmt)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	426
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	427
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	428 class Inference:
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	429
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	430 def __init__(self) -> None:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	431 self.rules = []
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	432
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	433 def setRules(self, g: ConjunctiveGraph):
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	434 self.rules: List[Rule] = []
1599 abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	435 for stmt in g:
abbf0eb0e640 fix a bug with a slightly moer complicated set of rules drewp@bigasterisk.com parents: 1598 diff changeset	436 if stmt[1] == LOG['implies']:
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	437 self.rules.append(Rule(stmt[0], stmt[2]))
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	438 # other stmts should go to a default working set?
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	439
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	440 @INFER_CALLS.time()
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	441 def infer(self, graph: Graph):
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	442 """
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	443 returns new graph of inferred statements.
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	444 """
1626 7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	445 n = graph.__len__()
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	446 INFER_GRAPH_SIZE.observe(n)
7b3656867185 metrics on input graph sizes drewp@bigasterisk.com parents: 1623 diff changeset	447 log.info(f'{INDENT*0} Begin inference of graph len={n} with rules len={len(self.rules)}:')
1601 30463df12d89 infer() dumps stats drewp@bigasterisk.com parents: 1600 diff changeset	448 startTime = time.time()
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	449 stats: Dict[str, Union[int, float]] = defaultdict(lambda: 0)
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	450 # everything that is true: the input graph, plus every rule conclusion we can make
1593 b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	451 workingSet = Graph()
b0df43d5494c big rewrite- more classes, smaller methods, more typesafe, all current tests passing drewp@bigasterisk.com parents: 1592 diff changeset	452 workingSet += graph
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	453
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	454 # just the statements that came from RHS's of rules that fired.
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	455 implied = ConjunctiveGraph()
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	456
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	457 bailout_iterations = 100
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	458 delta = 1
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	459 stats['initWorkingSet'] = cast(int, workingSet.__len__())
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	460 while delta > 0 and bailout_iterations > 0:
1620 92f8deb59735 log layout drewp@bigasterisk.com parents: 1616 diff changeset	461 log.debug('')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	462 log.info(f'{INDENT1}iteration ({bailout_iterations} left)')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	463 bailout_iterations -= 1
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	464 delta = -len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	465 self._iterateAllRules(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	466 delta += len(implied)
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	467 stats['iterations'] += 1
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	468 log.info(f'{INDENT*2} this inference iteration added {delta} more implied stmts')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	469 stats['timeSpent'] = round(time.time() - startTime, 3)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	470 stats['impliedStmts'] = len(implied)
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	471 log.info(f'{INDENT*0} Inference done {dict(stats)}. Implied:')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	472 for st in implied:
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	473 log.info(f'{INDENT*1} {st}')
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	474 return implied
9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	475
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	476 def _iterateAllRules(self, workingSet: Graph, implied: Graph, stats):
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	477 for i, rule in enumerate(self.rules):
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	478 self._logRuleApplicationHeader(workingSet, i, rule)
272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	479 rule.applyRule(workingSet, implied, stats)
1587 9a3a18c494f9 WIP new inferencer. no vars yet. drewp@bigasterisk.com parents: diff changeset	480
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	481 def _logRuleApplicationHeader(self, workingSet, i, r: Rule):
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	482 if not log.isEnabledFor(logging.DEBUG):
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	483 return
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	484
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	485 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	486 log.debug(f'{INDENT*2} workingSet:')
1597 387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	487 for j, stmt in enumerate(sorted(workingSet)):
387a9cb66517 logging adjustments drewp@bigasterisk.com parents: 1596 diff changeset	488 log.debug(f'{INDENT*3} ({j}) {stmt}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	489
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	490 log.debug('')
e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	491 log.debug(f'{INDENT*2}-applying rule {i}')
1607 b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	492 log.debug(f'{INDENT*3} rule def lhs: {graphDump(r.lhsGraph)}')
b21885181e35 more modules, types. Maybe less repeated computation on BoundLhs drewp@bigasterisk.com parents: 1605 diff changeset	493 log.debug(f'{INDENT*3} rule def rhs: {graphDump(r.rhsGraph)}')
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	494
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	495
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	496 def graphDump(g: Union[Graph, List[Triple]]):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	497 if not isinstance(g, Graph):
327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	498 g2 = Graph()
1594 e58bcfa66093 cleanups and a few fixed cases drewp@bigasterisk.com parents: 1593 diff changeset	499 g2 += g
1590 327202020892 WIP inference- getting into more degenerate test cases drewp@bigasterisk.com parents: 1589 diff changeset	500 g = g2
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	501 g.bind('', ROOM)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	502 g.bind('ex', Namespace('http://example.com/'))
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	503 lines = cast(bytes, g.serialize(format='n3')).decode('utf8').splitlines()
1612 272f78d4671a mark skipped tests. move applyRule into Rule. minor cleanups. drewp@bigasterisk.com parents: 1611 diff changeset	504 lines = [line.strip() for line in lines if not line.startswith('@prefix')]
1589 5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	505 return ' '.join(lines)
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	506
5c1055be3c36 WIP more debugging, working towards bnode-matching support drewp@bigasterisk.com parents: 1588 diff changeset	507
1600 89a50242cb5e cleanup internal names, imports drewp@bigasterisk.com parents: 1599 diff changeset	508 def _organize(candidateTermMatches: Dict[BindableTerm, Set[Node]]) -> Tuple[List[BindableTerm], List[List[Node]]]:
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	509 items = list(candidateTermMatches.items())
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	510 items.sort()
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	511 orderedVars: List[BindableTerm] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	512 orderedValueSets: List[List[Node]] = []
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	513 for v, vals in items:
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	514 orderedVars.append(v)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	515 orderedValues: List[Node] = list(vals)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	516 orderedValues.sort(key=str)
d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	517 orderedValueSets.append(orderedValues)
1588 0757fafbfdab WIP inferencer - partial var and function support drewp@bigasterisk.com parents: 1587 diff changeset	518
1592 d7b66234064b pure reordering of funcs to make the next diffs smaller drewp@bigasterisk.com parents: 1591 diff changeset	519 return orderedVars, orderedValueSets

Mercurial > code > home > repos > homeauto

annotate service/mqtt_to_rdf/inference.py @ 1631:2c85a4f5dd9c