Mercurial > code > home > repos > homeauto

annotate lib/patchablegraph.py @ 233:4ebb5cc30002

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code. Ignore-this: cf7d20d54e134e8ff33a9ee405610846
author drewp@bigasterisk.com
date Sat, 30 Jan 2016 06:40:00 -0800
parents c1b98006f56e
children 8d89da1915df
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
1 """
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
2 Design:
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
3
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
4 1. Services each have (named) graphs, which they patch as things
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
5 change. PatchableGraph is an object for holding this graph.
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
6 2. You can http GET that graph, or ...
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
7 3. You can http GET/SSE that graph and hear about modifications to it
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
8 4. The client that got the graph holds and maintains a copy. The
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
9 client may merge together multiple graphs.
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
10 5. Client queries its graph with low-level APIs or client-side sparql.
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
11 6. When the graph changes, the client knows and can update itself at
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
12 low or high granularity.
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
13
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
14
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
15 See also:
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
16 * http://iswc2007.semanticweb.org/papers/533.pdf RDFSync: efficient remote synchronization of RDF
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
17 models
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
18 * https://www.w3.org/2009/12/rdf-ws/papers/ws07 Supporting Change Propagation in RDF
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
19 * https://www.w3.org/DesignIssues/lncs04/Diff.pdf Delta: an ontology for the distribution of
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
20 differences between RDF graphs
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
21
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
22 """
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
23 import sys, json, logging
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
24 import cyclone.sse
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
25 sys.path.append("/my/proj/light9")
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
26 from light9.rdfdb.grapheditapi import GraphEditApi
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
27 from rdflib import ConjunctiveGraph
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
28 from light9.rdfdb.rdflibpatch import patchQuads
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
29 from rdflib_jsonld.serializer import from_rdf
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
30 from cycloneerr import PrettyErrorHandler
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
31
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
32 log = logging.getLogger('patchablegraph')
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
33
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
34 def writeGraphResponse(req, graph, acceptHeader):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
35 if acceptHeader == 'application/nquads':
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
36 req.set_header('Content-type', 'application/nquads')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
37 graph.serialize(req, format='nquads')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
38 elif acceptHeader == 'application/ld+json':
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
39 req.set_header('Content-type', 'application/ld+json')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
40 graph.serialize(req, format='json-ld', indent=2)
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
41 else:
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
42 req.set_header('Content-type', 'application/x-trig')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
43 graph.serialize(req, format='trig')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
44
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
45 # forked from /my/proj/light9/light9/rdfdb/rdflibpatch.py
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
46 def _graphFromQuads2(q):
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
47 g = ConjunctiveGraph()
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
48 #g.addN(q) # no effect on nquad output
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
49 for s,p,o,c in q:
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
50 g.get_context(c).add((s,p,o)) # kind of works with broken rdflib nquad serializer code
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
51 #g.store.add((s,p,o), c) # no effect on nquad output
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
52 return g
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
53
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
54 def patchAsJson(p):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
55 return json.dumps({'patch': {
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
56 'adds': from_rdf(_graphFromQuads2(p.addQuads)),
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
57 'deletes': from_rdf(_graphFromQuads2(p.delQuads)),
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
58 }})
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
59
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
60 def graphAsJson(g):
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
61 # This is not the same as g.serialize(format='json-ld')! That
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
62 # version omits literal datatypes.
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
63 return json.dumps(from_rdf(g))
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
64
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
65 class PatchableGraph(GraphEditApi):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
66 """
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
67 Master graph that you modify with self.patch, and we get the
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
68 updates to all current listeners.
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
69 """
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
70 def __init__(self):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
71 self._graph = ConjunctiveGraph()
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
72 self._observers = []
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
73
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
74 def serialize(self, to, **kw):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
75 return self._graph.serialize(to, **kw)
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
76
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
77 def patch(self, p):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
78 if p.isNoop():
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
79 return
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
80 patchQuads(self._graph,
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
81 deleteQuads=p.delQuads,
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
82 addQuads=p.addQuads,
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
83 perfect=False) # true?
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
84 for ob in self._observers:
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
85 ob(patchAsJson(p))
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
86
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
87 def asJsonLd(self):
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
88 return graphAsJson(self._graph)
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
89
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
90 def addObserver(self, onPatch):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
91 self._observers.append(onPatch)
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
92
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
93 def removeObserver(self, onPatch):
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
94 try:
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
95 self._observers.remove(onPatch)
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
96 except ValueError:
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
97 pass
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
98
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
99
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
100 class CycloneGraphHandler(PrettyErrorHandler, cyclone.web.RequestHandler):
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
101 def initialize(self, masterGraph):
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
102 self.masterGraph = masterGraph
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
103
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
104 def get(self):
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
105 writeGraphResponse(self, self.masterGraph,
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
106 self.request.headers.get('accept'))
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
107
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
108 class CycloneGraphEventsHandler(cyclone.sse.SSEHandler):
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
109 """
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
110 One session with one client.
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
111
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
112 returns current graph plus future patches to keep remote version
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
113 in sync with ours.
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
114
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
115 intsead of turning off buffering all over, it may work for this
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
116 response to send 'x-accel-buffering: no', per
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
117 http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_buffering
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
118 """
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
119 def __init__(self, application, request, masterGraph):
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
120 cyclone.sse.SSEHandler.__init__(self, application, request)
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
121 self.masterGraph = masterGraph
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
122
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
123 def bind(self):
233
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
124 graphJson = self.masterGraph.asJsonLd()
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
125 log.debug("send fullGraph event: %s", graphJson)
4ebb5cc30002 server/browser graph sync. cut dependency on the WS version. merge some changes between arduino/pi code.
drewp@bigasterisk.com
parents: 227
diff changeset
126 self.sendEvent(message=graphJson, event='fullGraph')
224
596c645a1fc5 refactor /graph and /graph/events handlers to lib/
drewp@bigasterisk.com
parents: 223
diff changeset
127 self.masterGraph.addObserver(self.onPatch)
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
128
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
129 def onPatch(self, patchJson):
227
c1b98006f56e fix patchablegraph unbind event
drewp@bigasterisk.com
parents: 224
diff changeset
130 # throttle and combine patches here- ideally we could see how
c1b98006f56e fix patchablegraph unbind event
drewp@bigasterisk.com
parents: 224
diff changeset
131 # long the latency to the client is to make a better rate choice
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
132 self.sendEvent(message=patchJson, event='patch')
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
133
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
134 def unbind(self):
227
c1b98006f56e fix patchablegraph unbind event
drewp@bigasterisk.com
parents: 224
diff changeset
135 self.masterGraph.removeObserver(self.onPatch)
223
9236b736bc34 add new jsonld/SSE support to environment service as a test
drewp@bigasterisk.com
parents:
diff changeset
136