# HG changeset patch # User Drew Perttula # Date 1558940331 0 # Node ID 8d4822ae58bc040f256768ed00c60e069ce4168b # Parent 541010f99d6142b44a251c6c3c9ec2bef46478dd more types, as needed on light9 keyboardcomposer Ignore-this: 9bc5d28702e6684fcce62104fc50966d diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/autodepgraphapi.py --- a/rdfdb/autodepgraphapi.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/autodepgraphapi.py Mon May 27 06:58:51 2019 +0000 @@ -1,5 +1,5 @@ import logging -from typing import Callable, Dict, Set, Tuple +from typing import Callable, Dict, Set, Tuple, List from rdflib import RDF, RDFS, URIRef from rdfdb.currentstategraphapi import contextsForStatementNoWildcards log = logging.getLogger('autodepgraphapi') @@ -21,9 +21,9 @@ def __init__(self): self._watchers = _GraphWatchers() - self.currentFuncs: Callable[[], None] = [] # stack of addHandler callers + self.currentFuncs: List[Callable[[], None]] = [] # stack of addHandler callers - def addHandler(self, func): + def addHandler(self, func: Callable[[], None]) -> None: """ run this (idempotent) func, noting what graph values it uses. Run it again in the future if there are changes to those diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/currentstategraphapi_test.py --- a/rdfdb/currentstategraphapi_test.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/currentstategraphapi_test.py Mon May 27 06:58:51 2019 +0000 @@ -4,6 +4,6 @@ class TestSequentialUri(unittest.TestCase): def test_returnsSequentialUris(self): - g = SyncedGraph('http://example.com/db/', label='test') + g = SyncedGraph(URIRef('http://example.com/db/'), label='test') self.assertEqual(g.sequentialUri(URIRef('http://example.com/foo')), URIRef('http://example.com/foo1')) self.assertEqual(g.sequentialUri(URIRef('http://example.com/foo')), URIRef('http://example.com/foo2')) diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/grapheditapi.py --- a/rdfdb/grapheditapi.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/grapheditapi.py Mon May 27 06:58:51 2019 +0000 @@ -1,6 +1,7 @@ import random, logging from itertools import chain from rdflib import URIRef, RDF +from rdflib.term import Node from rdfdb.patch import Patch, quadsWithContextUris log = logging.getLogger('graphedit') @@ -25,11 +26,11 @@ if newObject is not None else []) return Patch(delQuads=existing, addQuads=toAdd).simplify() - def patchObject(self, context, subject, predicate, newObject): + def patchObject(self, context: URIRef, subject: Node, predicate: URIRef, newObject: Node): p = self.getObjectPatch(context, subject, predicate, newObject) if not p.isNoop(): log.debug("patchObject %r" % p.jsonRepr) - self.patch(p) + self.patch(p) # type: ignore def patchSubgraph(self, context, newGraph): """ diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/graphfile.py --- a/rdfdb/graphfile.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/graphfile.py Mon May 27 06:58:51 2019 +0000 @@ -1,11 +1,13 @@ import logging, traceback, os, time from twisted.python.filepath import FilePath from twisted.internet import reactor -from twisted.internet.inotify import humanReadableMask +from twisted.internet.interfaces import IDelayedCall +from twisted.internet.inotify import INotify, humanReadableMask from rdflib import Graph, RDF, URIRef from rdfdb.patch import Patch from rdfdb.rdflibpatch import inContext -from typing import Dict +from typing import Dict, Optional +from typing_extensions import Protocol log = logging.getLogger('graphfile') iolog = logging.getLogger('io') @@ -60,11 +62,25 @@ patchN3SerializerToUseLessWhitespace() + +class PatchCb(Protocol): + def __call__(self, patch: Patch, dueToFileChange: bool=False) -> None: ... + +class GetSubgraph(Protocol): + def __call__(self, uri: URIRef) -> Graph: ... + class GraphFile(object): """ one rdf file that we read from, write to, and notice external changes to """ - def __init__(self, notifier, path, uri, patch, getSubgraph, globalPrefixes, ctxPrefixes): + def __init__(self, + notifier: INotify, + path: bytes, + uri: URIRef, + patch: PatchCb, + getSubgraph: GetSubgraph, + globalPrefixes: Dict[str, URIRef], + ctxPrefixes: Dict[str, URIRef]): """ uri is the context for the triples in this file. We assume sometimes that we're the only ones with triples in this @@ -77,7 +93,7 @@ self.path, self.uri = path, uri self.patch, self.getSubgraph = patch, getSubgraph - self.lastWriteTimestamp = 0 # mtime from the last time _we_ wrote + self.lastWriteTimestamp = 0.0 # mtime from the last time _we_ wrote self.globalPrefixes = globalPrefixes self.ctxPrefixes = ctxPrefixes @@ -99,12 +115,12 @@ self.flushDelay = 2 # seconds until we have to call flush() when dirty - self.writeCall = None # or DelayedCall + self.writeCall: Optional[IDelayedCall] = None self.notifier = notifier self.addWatch() - def addWatch(self): + def addWatch(self) -> None: # emacs save comes in as IN_MOVE_SELF, maybe @@ -120,7 +136,7 @@ log.info("add watch on %s", self.path) self.notifier.watch(FilePath(self.path), callbacks=[self.notify]) - def notify(self, notifier, filepath, mask): + def notify(self, notifier: INotify, filepath: FilePath, mask: int) -> None: try: maskNames = humanReadableMask(mask) if maskNames[0] == 'delete_self': @@ -156,7 +172,7 @@ except Exception: traceback.print_exc() - def fileGone(self): + def fileGone(self) -> None: """ our file is gone; remove the statements from that context """ @@ -165,7 +181,7 @@ if myQuads: self.patch(Patch(delQuads=myQuads), dueToFileChange=True) - def reread(self): + def reread(self) -> None: """update the graph with any diffs from this file n3 parser fails on "1.e+0" even though rdflib was emitting that itself @@ -205,7 +221,7 @@ else: log.debug("old == new after reread of %s", self.path) - def dirty(self, graph): + def dirty(self, graph: Graph) -> None: """ there are new contents to write to our file @@ -227,12 +243,13 @@ if self.writeCall: self.writeCall.reset(self.flushDelay) else: - self.writeCall = reactor.callLater(self.flushDelay, self.flush) + # This awkward assignment is just to hide from mypy. + setattr(self, 'writeCall', reactor.callLater(self.flushDelay, self.flush)) - def flush(self): + def flush(self) -> None: self.writeCall = None - tmpOut = self.path + ".rdfdb-temp" + tmpOut = self.path + b".rdfdb-temp" f = open(tmpOut, 'wb') t1 = time.time() for p, n in (list(self.globalPrefixes.items()) + @@ -247,7 +264,7 @@ iolog.info("%s rewrote in %.1f ms", self.path, serializeTime * 1000) - def __repr__(self): + def __repr__(self) -> str: return "%s(path=%r, uri=%r, ...)" % ( self.__class__.__name__, self.path, self.uri) diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/graphfile_test.py --- a/rdfdb/graphfile_test.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/graphfile_test.py Mon May 27 06:58:51 2019 +0000 @@ -16,7 +16,7 @@ def getSubgraph(uri): return Graph() - gf = GraphFile(mock.Mock(), tf.name, URIRef('uri'), mock.Mock(), getSubgraph, {}, {}) + gf = GraphFile(mock.Mock(), tf.name.encode('ascii'), URIRef('uri'), mock.Mock(), getSubgraph, {}, {}) gf.reread() newGraph = Graph() diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/patchsender.py --- a/rdfdb/patchsender.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/patchsender.py Mon May 27 06:58:51 2019 +0000 @@ -1,5 +1,5 @@ import logging, time -from typing import List +from typing import List, Tuple, Optional import cyclone.httpclient from rdflib import URIRef from twisted.internet import defer @@ -7,13 +7,15 @@ log = logging.getLogger('syncedgraph') +SendResult = defer.Deferred# to None + class PatchSender(object): """ SyncedGraph may generate patches faster than we can send them. This object buffers and may even collapse patches before they go the server """ - def __init__(self, target, myUpdateResource): + def __init__(self, target: URIRef, myUpdateResource): """ target is the URI we'll send patches to @@ -23,11 +25,11 @@ """ self.target = target self.myUpdateResource = myUpdateResource - self._patchesToSend: List[Patch] = [] - self._currentSendPatchRequest = None + self._patchesToSend: List[Tuple[Patch, SendResult]] = [] + self._currentSendPatchRequest: Optional[SendResult] = None - def sendPatch(self, p): - sendResult: defer.Deferred[None] = defer.Deferred() + def sendPatch(self, p: Patch) -> SendResult: + sendResult: SendResult = defer.Deferred() self._patchesToSend.append((p, sendResult)) self._continueSending() return sendResult @@ -41,7 +43,7 @@ # 2. or, other code could deal for the fact that cancelAll # isn't perfect - def _continueSending(self): + def _continueSending(self) -> None: if not self._patchesToSend or self._currentSendPatchRequest: return if len(self._patchesToSend) > 1: diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/service.py --- a/rdfdb/service.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/service.py Mon May 27 06:58:51 2019 +0000 @@ -8,7 +8,7 @@ from typing import Dict, List, Set, Optional, Union from rdflib import ConjunctiveGraph, URIRef, Graph -from rdfdb.graphfile import GraphFile +from rdfdb.graphfile import GraphFile, PatchCb, GetSubgraph from rdfdb.patch import Patch, ALLSTMTS from rdfdb.rdflibpatch import patchQuads from rdfdb.file_vs_uri import correctToTopdirPrefix, fileForUri, uriFromFile, DirUriMap @@ -63,14 +63,14 @@ client.sendPatch(Patch( addQuads=graph.quads(ALLSTMTS), delQuads=[])) - + class WatchedFiles(object): """ find files, notice new files. This object watches directories. Each GraphFile watches its own file. """ - def __init__(self, dirUriMap: DirUriMap, patch, getSubgraph, addlPrefixes): + def __init__(self, dirUriMap: DirUriMap, patch: PatchCb, getSubgraph: GetSubgraph, addlPrefixes: Dict[str, URIRef]): self.dirUriMap = dirUriMap # {abspath : uri prefix} self.patch, self.getSubgraph = patch, getSubgraph self.addlPrefixes = addlPrefixes @@ -143,7 +143,7 @@ log.info("%s do initial read", inFile) gf.reread() - def aboutToPatch(self, ctx): + def aboutToPatch(self, ctx: URIRef): """ warn us that a patch is about to come to this context. it's more straightforward to create the new file now @@ -156,7 +156,10 @@ """ if ctx not in self.graphFiles: outFile = fileForUri(self.dirUriMap, ctx) - assert '//' not in outFile, (outFile, self.dirUriMap, ctx) + # mypy missed the next line because of + # https://github.com/python/typeshed/issues/2937 ('str in + # bytes' isn't an error) + assert b'//' not in outFile, (outFile, self.dirUriMap, ctx) log.info("starting new file %r", outFile) self._addGraphFile(ctx, outFile) @@ -199,7 +202,7 @@ self.summarizeToLog() - def patch(self, p, dueToFileChange=False): + def patch(self, patch: Patch, dueToFileChange: bool=False) -> None: """ apply this patch to the master graph then notify everyone about it @@ -209,18 +212,18 @@ if p has a senderUpdateUri attribute, we won't send this patch back to the sender with that updateUri """ - ctx = p.getContext() + ctx = patch.getContext() log.info("patching graph %s -%d +%d" % ( - ctx, len(p.delQuads), len(p.addQuads))) + ctx, len(patch.delQuads), len(patch.addQuads))) if hasattr(self, 'watchedFiles'): # not available during startup self.watchedFiles.aboutToPatch(ctx) - patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True) - self._sendPatch(p) + patchQuads(self.graph, patch.delQuads, patch.addQuads, perfect=True) + self._sendPatch(patch) if not dueToFileChange: self.watchedFiles.dirtyFiles([ctx]) - sendToLiveClients(asJson=p.jsonRepr) + sendToLiveClients(asJson=patch.jsonRepr) def _sendPatch(self, p): senderUpdateUri = getattr(p, 'senderUpdateUri', None) @@ -245,7 +248,7 @@ log.info(" %s: %s statements" % (c.identifier, len(self.getSubgraph(c.identifier)))) - def getSubgraph(self, uri): + def getSubgraph(self, uri: URIRef) -> Graph: """ this is meant to return a live view of the given subgraph, but if i'm still working around an rdflib bug, it might return a diff -r 541010f99d61 -r 8d4822ae58bc rdfdb/syncedgraph.py --- a/rdfdb/syncedgraph.py Mon May 27 00:39:16 2019 +0000 +++ b/rdfdb/syncedgraph.py Mon May 27 06:58:51 2019 +0000 @@ -16,19 +16,21 @@ PatchSender - collects and transmits your graph edits """ -from rdflib import ConjunctiveGraph +from rdflib import ConjunctiveGraph, URIRef import logging, cyclone.httpclient, traceback from twisted.internet import defer import socket import treq, json log = logging.getLogger('syncedgraph') from rdfdb.rdflibpatch import patchQuads +from typing import Optional from rdfdb.patchsender import PatchSender from rdfdb.patchreceiver import PatchReceiver from rdfdb.currentstategraphapi import CurrentStateGraphApi from rdfdb.autodepgraphapi import AutoDepGraphApi from rdfdb.grapheditapi import GraphEditApi +from rdfdb.patch import Patch # everybody who writes literals needs to get this from rdfdb.rdflibpatch_literal import patch @@ -53,7 +55,7 @@ pending local changes) and get the data again from the server. """ - def __init__(self, rdfdbRoot, label, receiverHost=None): + def __init__(self, rdfdbRoot: URIRef, label: str, receiverHost: Optional[str]=None): """ label is a string that the server will display in association with your connection @@ -104,7 +106,7 @@ #diff against old entire graph #broadcast that change - def patch(self, p): + def patch(self, p: Patch) -> None: """send this patch to the server and apply it to our local graph and run handlers"""