Mercurial > code > home > repos > light9
annotate bin/rdfdb @ 811:b19cd005a491
just comments
Ignore-this: 850b0da90240ab6f764430505809fd6d
author | drewp@bigasterisk.com |
---|---|
date | Sat, 29 Sep 2012 20:59:13 +0000 |
parents | a631e075a5bf |
children | 1ae8e6b287e3 |
rev | line source |
---|---|
796 | 1 #!bin/python |
2 """ | |
3 other tools POST themselves to here as subscribers to the graph. They | |
811 | 4 are providing a URL we can PUT to with graph updates. |
796 | 5 |
6 we immediately PUT them back all the contents of the graph as a bunch | |
7 of adds. | |
8 | |
811 | 9 later we PUT them back with patches (del/add lists) when there are |
796 | 10 changes. |
11 | |
12 If we fail to reach a registered caller, we forget about it for future | |
811 | 13 calls. We could PUT empty diffs as a heartbeat to notice disappearing |
796 | 14 callers faster. |
15 | |
811 | 16 A caller can submit a patch which we'll persist and broadcast to every |
17 other client. | |
796 | 18 |
19 Global data undo should probably happen within this service. | |
20 | |
21 Maybe some subgraphs are for transient data (e.g. current timecode, | |
22 mouse position in curvecalc) that only some listeners want to hear about. | |
23 | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
24 Deletes are graph-specific, so callers may be surprised to delete a |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
25 stmt from one graph but then find that statement is still true. |
796 | 26 |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
27 Alternate plan: would it help to insist that every patch is within |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
28 only one subgraph? I think it's ok for them to span multiple ones. |
796 | 29 |
30 Inserts can be made on any subgraphs, and each subgraph is saved in | |
31 its own file. The file might not be in a format that can express | |
32 graphs, so I'm just going to not store the subgraph URI in any file. | |
33 | |
34 I don't support wildcard deletes, and there are race conditions where a | |
35 s-p could end up with unexpected multiple objects. Every client needs | |
36 to be ready for this. | |
37 | |
38 We watch the files and push their own changes back to the clients. | |
39 | |
40 Persist our client list, to survive restarts. In another rdf file? A | |
41 random json one? memcache? Also hold the recent changes. We're not | |
42 logging everything forever, though, since the output files and a VCS | |
43 shall be used for that | |
44 | |
45 Bnodes: this rdfdb graph might be able to track bnodes correctly, and | |
46 they make for more compact n3 files. I'm not sure if it's going to be | |
47 hard to keep the client bnodes in sync though. File rereads would be | |
811 | 48 hard, if ever a bnode was used across graphs, so that probably should |
796 | 49 not be allowed. |
50 | |
51 Our API: | |
52 | |
53 GET / ui | |
811 | 54 GET /graph the whole graph, or a query from it (needed? just for ui browsing?) |
796 | 55 PUT /patches clients submit changes |
56 GET /patches (recent) patches from clients | |
57 POST /graphClients clientUpdate={uri} to subscribe | |
58 GET /graphClients current clients | |
59 | |
60 format: | |
61 json {"adds" : [[quads]...], | |
62 "deletes": [[quads]], | |
811 | 63 "senderUpdateUri" : tooluri, |
64 "created":tttt // maybe to help resolve some conflicts | |
796 | 65 } |
66 maybe use some http://json-ld.org/ in there. | |
67 | |
806 | 68 proposed rule feature: |
69 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and | |
70 rerun the rules when either one changes. Should the sourceFile be able | |
71 to specify its own rules file? That would be easier | |
72 configuration. How do edits work? Not allowed? Patch the source only? | |
73 Also see the source graph loaded into a different ctx, and you can | |
74 edit that one and see the results in the output context? | |
75 | |
796 | 76 Our web ui: |
77 | |
811 | 78 sections |
79 | |
80 registered clients | |
796 | 81 |
811 | 82 recent patches, each one says what client it came from. You can reverse |
83 them here. We should be able to take patches that are close in time | |
84 and keep updating the same data (e.g. a stream of changes as the user | |
85 drags a slider) and collapse them into a single edit for clarity. | |
86 | |
87 Ways to display patches, using labels and creator/subj icons | |
88 where possible: | |
796 | 89 |
811 | 90 <creator> set <subj>'s <p> to <o> |
91 <creator> changed <subj>'s <pred> from <o1> to <o2> | |
92 <creator> added <o> to <s> <p> | |
93 | |
94 raw messages for debugging this client | |
806 | 95 |
811 | 96 ctx urls take you to-> |
97 files, who's dirty, have we seen external changes, notice big | |
98 files that are taking a long time to save | |
806 | 99 |
811 | 100 graph contents. plain rdf browser like an outliner or |
101 something. clicking any resource from the other displays takes you | |
102 to this, focused on that resource | |
803 | 103 |
796 | 104 """ |
105 from twisted.internet import reactor | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
106 import twisted.internet.error |
796 | 107 import sys, optparse, logging, json, os |
108 import cyclone.web, cyclone.httpclient, cyclone.websocket | |
109 sys.path.append(".") | |
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
110 from light9 import networking, showconfig, prof |
796 | 111 from rdflib import ConjunctiveGraph, URIRef, Graph |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
112 from light9.rdfdb.graphfile import GraphFile |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
113 from light9.rdfdb.patch import Patch, ALLSTMTS |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
114 from light9.rdfdb.rdflibpatch import patchQuads |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
115 from light9.rdfdb import syncedgraph |
796 | 116 |
117 from twisted.internet.inotify import INotify | |
118 logging.basicConfig(level=logging.DEBUG) | |
119 log = logging.getLogger() | |
120 | |
121 try: | |
122 import sys | |
123 sys.path.append("../homeauto/lib") | |
124 from cycloneerr import PrettyErrorHandler | |
125 except ImportError: | |
126 class PrettyErrorHandler(object): | |
127 pass | |
128 | |
129 class Client(object): | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
130 """ |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
131 one of our syncedgraph clients |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
132 """ |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
133 def __init__(self, updateUri, label, db): |
796 | 134 self.db = db |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
135 self.label = label |
796 | 136 self.updateUri = updateUri |
137 self.sendAll() | |
138 | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
139 def __repr__(self): |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
140 return "<%s client at %s>" % (self.label, self.updateUri) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
141 |
796 | 142 def sendAll(self): |
143 """send the client the whole graph contents""" | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
144 log.info("sending all graphs to %s at %s" % |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
145 (self.label, self.updateUri)) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
146 self.sendPatch(Patch( |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
147 addQuads=self.db.graph.quads(ALLSTMTS), |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
148 delQuads=[])) |
796 | 149 |
150 def sendPatch(self, p): | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
151 return syncedgraph.sendPatch(self.updateUri, p) |
796 | 152 |
153 class Db(object): | |
811 | 154 """ |
155 the master graph, all the connected clients, all the files we're watching | |
156 """ | |
796 | 157 def __init__(self): |
158 self.clients = [] | |
159 self.graph = ConjunctiveGraph() | |
160 | |
161 notifier = INotify() | |
162 notifier.startReading() | |
163 | |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
164 for inFile in [#"show/dance2012/config.n3", |
801 | 165 "show/dance2012/patch.n3", |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
166 "show/dance2012/subs/bcools", |
799
fcf95ff23cc5
PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents:
798
diff
changeset
|
167 "show/dance2012/subs/bwarm", |
fcf95ff23cc5
PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents:
798
diff
changeset
|
168 "show/dance2012/subs/house", |
fcf95ff23cc5
PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents:
798
diff
changeset
|
169 "demo.n3", |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
170 ]: |
796 | 171 self.g = GraphFile(notifier, |
172 inFile, | |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
173 URIRef("http://example.com/file/%s" % |
796 | 174 os.path.basename(inFile)), |
175 self.patch, | |
176 self.getSubgraph) | |
177 | |
178 def patch(self, p): | |
179 """ | |
180 apply this patch to the master graph then notify everyone about it | |
811 | 181 |
182 dueToFileChange if this is a patch describing an edit we read | |
183 *from* the file (such that we shouldn't write it back to the file) | |
184 | |
185 if p has a senderUpdateUri attribute, we won't send this patch | |
186 back to the sender with that updateUri | |
796 | 187 """ |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
188 log.info("patching graph -%d +%d" % (len(p.delQuads), len(p.addQuads))) |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
189 |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
190 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True) |
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
191 senderUpdateUri = getattr(p, 'senderUpdateUri', None) |
796 | 192 self.summarizeToLog() |
193 for c in self.clients: | |
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
194 print "send to %s? %s %s" % (c, c.updateUri, senderUpdateUri) |
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
195 if c.updateUri == senderUpdateUri: |
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
196 # this client has self-applied the patch already |
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
197 continue |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
198 d = c.sendPatch(p) |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
199 d.addErrback(self.clientErrored, c) |
796 | 200 sendToLiveClients(asJson=p.jsonRepr) |
201 | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
202 def clientErrored(self, err, c): |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
203 err.trap(twisted.internet.error.ConnectError) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
204 log.info("connection error- dropping client %r" % c) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
205 self.clients.remove(c) |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
206 self.sendClientsToAllLivePages() |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
207 |
796 | 208 def summarizeToLog(self): |
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
209 log.info("contexts in graph (%s total stmts):" % len(self.graph)) |
796 | 210 for c in self.graph.contexts(): |
211 log.info(" %s: %s statements" % | |
212 (c.identifier, len(self.getSubgraph(c.identifier)))) | |
213 | |
214 def getSubgraph(self, uri): | |
811 | 215 """ |
216 this is meant to return a live view of the given subgraph, but | |
217 if i'm still working around an rdflib bug, it might return a | |
218 copy | |
219 | |
220 and it's returning triples, but I think quads would be better | |
221 """ | |
796 | 222 # this is returning an empty Graph :( |
223 #return self.graph.get_context(uri) | |
224 | |
225 g = Graph() | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
226 for s in self.graph.triples(ALLSTMTS, uri): |
796 | 227 g.add(s) |
228 return g | |
229 | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
230 def addClient(self, updateUri, label): |
796 | 231 [self.clients.remove(c) |
232 for c in self.clients if c.updateUri == updateUri] | |
233 | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
234 log.info("new client %s at %s" % (label, updateUri)) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
235 self.clients.append(Client(updateUri, label, self)) |
796 | 236 self.sendClientsToAllLivePages() |
237 | |
238 def sendClientsToAllLivePages(self): | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
239 sendToLiveClients({"clients":[ |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
240 dict(updateUri=c.updateUri, label=c.label) |
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
241 for c in self.clients]}) |
796 | 242 |
243 class Index(PrettyErrorHandler, cyclone.web.RequestHandler): | |
244 def get(self): | |
245 self.set_header("Content-Type", "application/xhtml+xml") | |
246 self.write(open("light9/rdfdb.xhtml").read()) | |
247 | |
248 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler): | |
249 def get(self): | |
250 pass | |
251 | |
252 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler): | |
253 def __init__(self, *args, **kw): | |
254 cyclone.web.RequestHandler.__init__(self, *args, **kw) | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
255 p = syncedgraph.makePatchEndpointPutMethod(self.settings.db.patch) |
796 | 256 self.put = lambda: p(self) |
257 | |
258 def get(self): | |
259 pass | |
260 | |
261 | |
262 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler): | |
263 def get(self): | |
264 pass | |
265 | |
266 def post(self): | |
267 upd = self.get_argument("clientUpdate") | |
268 try: | |
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
269 self.settings.db.addClient(upd, self.get_argument("label")) |
796 | 270 except: |
271 import traceback | |
272 traceback.print_exc() | |
273 raise | |
274 | |
275 liveClients = set() | |
276 def sendToLiveClients(d=None, asJson=None): | |
277 j = asJson or json.dumps(d) | |
278 for c in liveClients: | |
279 c.sendMessage(j) | |
280 | |
281 class Live(cyclone.websocket.WebSocketHandler): | |
282 | |
283 def connectionMade(self, *args, **kwargs): | |
811 | 284 log.info("websocket opened") |
796 | 285 liveClients.add(self) |
286 self.settings.db.sendClientsToAllLivePages() | |
287 | |
288 def connectionLost(self, reason): | |
811 | 289 log.info("websocket closed") |
796 | 290 liveClients.remove(self) |
291 | |
292 def messageReceived(self, message): | |
293 log.info("got message %s" % message) | |
294 self.sendMessage(message) | |
295 | |
296 if __name__ == "__main__": | |
297 logging.basicConfig() | |
298 log = logging.getLogger() | |
299 | |
300 parser = optparse.OptionParser() | |
301 parser.add_option('--show', | |
302 help='show URI, like http://light9.bigasterisk.com/show/dance2008', | |
303 default=showconfig.showUri()) | |
304 parser.add_option("-v", "--verbose", action="store_true", | |
305 help="logging.DEBUG") | |
306 (options, args) = parser.parse_args() | |
307 | |
308 log.setLevel(logging.DEBUG if options.verbose else logging.INFO) | |
309 | |
310 if not options.show: | |
311 raise ValueError("missing --show http://...") | |
312 | |
313 db = Db() | |
314 | |
315 port = 8051 | |
316 reactor.listenTCP(port, cyclone.web.Application(handlers=[ | |
317 (r'/', Index), | |
318 (r'/live', Live), | |
319 (r'/graph', GraphResource), | |
320 (r'/patches', Patches), | |
321 (r'/graphClients', GraphClients), | |
322 | |
323 (r"/(jquery-1\.7\.2\.min\.js)", cyclone.web.StaticFileHandler, | |
324 dict(path='lib')), | |
325 | |
326 ], db=db)) | |
327 log.info("serving on %s" % port) | |
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
328 prof.run(reactor.run, profile=False) |