Mercurial > code > home > repos > light9
annotate bin/rdfdb @ 814:1ae8e6b287e3
improvements to file watching. outline of how resync will work
Ignore-this: 501c4f2076099364645cc27e9fe48f61
| author | drewp@bigasterisk.com |
|---|---|
| date | Sun, 30 Sep 2012 07:11:49 +0000 |
| parents | b19cd005a491 |
| children | d7f1f868eb6c |
| rev | line source |
|---|---|
| 796 | 1 #!bin/python |
| 2 """ | |
| 3 other tools POST themselves to here as subscribers to the graph. They | |
| 811 | 4 are providing a URL we can PUT to with graph updates. |
| 796 | 5 |
| 6 we immediately PUT them back all the contents of the graph as a bunch | |
| 7 of adds. | |
| 8 | |
| 811 | 9 later we PUT them back with patches (del/add lists) when there are |
| 796 | 10 changes. |
| 11 | |
| 12 If we fail to reach a registered caller, we forget about it for future | |
| 811 | 13 calls. We could PUT empty diffs as a heartbeat to notice disappearing |
| 796 | 14 callers faster. |
| 15 | |
| 811 | 16 A caller can submit a patch which we'll persist and broadcast to every |
| 17 other client. | |
| 796 | 18 |
| 19 Global data undo should probably happen within this service. | |
| 20 | |
| 21 Maybe some subgraphs are for transient data (e.g. current timecode, | |
| 22 mouse position in curvecalc) that only some listeners want to hear about. | |
| 23 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
24 Deletes are graph-specific, so callers may be surprised to delete a |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
25 stmt from one graph but then find that statement is still true. |
| 796 | 26 |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
27 Alternate plan: would it help to insist that every patch is within |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
28 only one subgraph? I think it's ok for them to span multiple ones. |
| 796 | 29 |
| 30 Inserts can be made on any subgraphs, and each subgraph is saved in | |
| 31 its own file. The file might not be in a format that can express | |
| 32 graphs, so I'm just going to not store the subgraph URI in any file. | |
| 33 | |
| 34 I don't support wildcard deletes, and there are race conditions where a | |
| 35 s-p could end up with unexpected multiple objects. Every client needs | |
| 36 to be ready for this. | |
| 37 | |
| 38 We watch the files and push their own changes back to the clients. | |
| 39 | |
| 40 Persist our client list, to survive restarts. In another rdf file? A | |
| 41 random json one? memcache? Also hold the recent changes. We're not | |
| 42 logging everything forever, though, since the output files and a VCS | |
| 43 shall be used for that | |
| 44 | |
| 45 Bnodes: this rdfdb graph might be able to track bnodes correctly, and | |
| 46 they make for more compact n3 files. I'm not sure if it's going to be | |
| 47 hard to keep the client bnodes in sync though. File rereads would be | |
| 811 | 48 hard, if ever a bnode was used across graphs, so that probably should |
| 796 | 49 not be allowed. |
| 50 | |
| 51 Our API: | |
| 52 | |
| 53 GET / ui | |
| 811 | 54 GET /graph the whole graph, or a query from it (needed? just for ui browsing?) |
| 796 | 55 PUT /patches clients submit changes |
| 56 GET /patches (recent) patches from clients | |
| 57 POST /graphClients clientUpdate={uri} to subscribe | |
| 58 GET /graphClients current clients | |
| 59 | |
| 60 format: | |
| 61 json {"adds" : [[quads]...], | |
| 62 "deletes": [[quads]], | |
| 811 | 63 "senderUpdateUri" : tooluri, |
| 64 "created":tttt // maybe to help resolve some conflicts | |
| 796 | 65 } |
| 66 maybe use some http://json-ld.org/ in there. | |
| 67 | |
| 806 | 68 proposed rule feature: |
| 69 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and | |
| 70 rerun the rules when either one changes. Should the sourceFile be able | |
| 71 to specify its own rules file? That would be easier | |
| 72 configuration. How do edits work? Not allowed? Patch the source only? | |
| 73 Also see the source graph loaded into a different ctx, and you can | |
| 74 edit that one and see the results in the output context? | |
| 75 | |
| 796 | 76 Our web ui: |
| 77 | |
| 811 | 78 sections |
| 79 | |
| 80 registered clients | |
| 796 | 81 |
| 811 | 82 recent patches, each one says what client it came from. You can reverse |
| 83 them here. We should be able to take patches that are close in time | |
| 84 and keep updating the same data (e.g. a stream of changes as the user | |
| 85 drags a slider) and collapse them into a single edit for clarity. | |
| 86 | |
| 87 Ways to display patches, using labels and creator/subj icons | |
| 88 where possible: | |
| 796 | 89 |
| 811 | 90 <creator> set <subj>'s <p> to <o> |
| 91 <creator> changed <subj>'s <pred> from <o1> to <o2> | |
| 92 <creator> added <o> to <s> <p> | |
| 93 | |
| 94 raw messages for debugging this client | |
| 806 | 95 |
| 811 | 96 ctx urls take you to-> |
| 97 files, who's dirty, have we seen external changes, notice big | |
| 98 files that are taking a long time to save | |
| 806 | 99 |
| 811 | 100 graph contents. plain rdf browser like an outliner or |
| 101 something. clicking any resource from the other displays takes you | |
| 102 to this, focused on that resource | |
| 803 | 103 |
| 796 | 104 """ |
| 105 from twisted.internet import reactor | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
106 import twisted.internet.error |
| 796 | 107 import sys, optparse, logging, json, os |
| 108 import cyclone.web, cyclone.httpclient, cyclone.websocket | |
| 109 sys.path.append(".") | |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
110 from light9 import networking, showconfig, prof |
| 796 | 111 from rdflib import ConjunctiveGraph, URIRef, Graph |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
112 from light9.rdfdb.graphfile import GraphFile |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
113 from light9.rdfdb.patch import Patch, ALLSTMTS |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
114 from light9.rdfdb.rdflibpatch import patchQuads |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
115 from light9.rdfdb import syncedgraph |
| 796 | 116 |
| 117 from twisted.internet.inotify import INotify | |
| 118 logging.basicConfig(level=logging.DEBUG) | |
| 119 log = logging.getLogger() | |
| 120 | |
| 121 try: | |
| 122 import sys | |
| 123 sys.path.append("../homeauto/lib") | |
| 124 from cycloneerr import PrettyErrorHandler | |
| 125 except ImportError: | |
| 126 class PrettyErrorHandler(object): | |
| 127 pass | |
| 128 | |
| 129 class Client(object): | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
130 """ |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
131 one of our syncedgraph clients |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
132 """ |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
133 def __init__(self, updateUri, label, db): |
| 796 | 134 self.db = db |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
135 self.label = label |
| 796 | 136 self.updateUri = updateUri |
| 137 self.sendAll() | |
| 138 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
139 def __repr__(self): |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
140 return "<%s client at %s>" % (self.label, self.updateUri) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
141 |
| 796 | 142 def sendAll(self): |
| 143 """send the client the whole graph contents""" | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
144 log.info("sending all graphs to %s at %s" % |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
145 (self.label, self.updateUri)) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
146 self.sendPatch(Patch( |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
147 addQuads=self.db.graph.quads(ALLSTMTS), |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
148 delQuads=[])) |
| 796 | 149 |
| 150 def sendPatch(self, p): | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
151 return syncedgraph.sendPatch(self.updateUri, p) |
| 796 | 152 |
| 153 class Db(object): | |
| 811 | 154 """ |
| 155 the master graph, all the connected clients, all the files we're watching | |
| 156 """ | |
| 796 | 157 def __init__(self): |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
158 # files from cwd become uris starting with this. *should* be |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
159 # building uris from the show uri in $LIGHT9_SHOW/URI |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
160 # instead. Who wants to keep their data in the same dir tree |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
161 # as the source code?! |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
162 self.topUri = URIRef("http://light9.bigasterisk.com/") |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
163 |
| 796 | 164 self.clients = [] |
| 165 self.graph = ConjunctiveGraph() | |
| 166 | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
167 self.notifier = INotify() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
168 self.notifier.startReading() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
169 self.graphFiles = {} # context uri : GraphFile |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
170 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
171 self.findAndLoadFiles() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
172 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
173 def findAndLoadFiles(self): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
174 self.initialLoad = True |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
175 try: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
176 dirs = [ |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
177 "show/dance2012/sessions", |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
178 "show/dance2012/subs", |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
179 "show/dance2012/subterms", |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
180 ] |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
181 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
182 for topdir in dirs: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
183 for dirpath, dirnames, filenames in os.walk(topdir): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
184 for base in filenames: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
185 self.watchFile(os.path.join(dirpath, base)) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
186 # todo: also notice new files in this dir |
| 796 | 187 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
188 self.watchFile("show/dance2012/config.n3") |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
189 self.watchFile("show/dance2012/patch.n3") |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
190 finally: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
191 self.initialLoad = False |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
192 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
193 self.summarizeToLog() |
| 796 | 194 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
195 def uriFromFile(self, filename): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
196 if filename.endswith('.n3'): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
197 # some legacy files don't end with n3. when we write them |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
198 # back this might not go so well |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
199 filename = filename[:-len('.n3')] |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
200 return URIRef(self.topUri + filename) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
201 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
202 def fileForUri(self, ctx): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
203 if not ctx.startswith(self.topUri): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
204 raise ValueError("don't know what filename to use for %s" % ctx) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
205 return ctx[len(self.topUri):] + ".n3" |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
206 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
207 def watchFile(self, inFile): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
208 ctx = self.uriFromFile(inFile) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
209 gf = GraphFile(self.notifier, inFile, ctx, self.patch, self.getSubgraph) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
210 self.graphFiles[ctx] = gf |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
211 gf.reread() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
212 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
213 def patch(self, p, dueToFileChange=False): |
| 796 | 214 """ |
| 215 apply this patch to the master graph then notify everyone about it | |
| 811 | 216 |
| 217 dueToFileChange if this is a patch describing an edit we read | |
| 218 *from* the file (such that we shouldn't write it back to the file) | |
| 219 | |
| 220 if p has a senderUpdateUri attribute, we won't send this patch | |
| 221 back to the sender with that updateUri | |
| 796 | 222 """ |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
223 ctx = p.getContext() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
224 log.info("patching graph %s -%d +%d" % ( |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
225 ctx, len(p.delQuads), len(p.addQuads))) |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
226 |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
227 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True) |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
228 senderUpdateUri = getattr(p, 'senderUpdateUri', None) |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
229 #if not self.initialLoad: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
230 # self.summarizeToLog() |
| 796 | 231 for c in self.clients: |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
232 if c.updateUri == senderUpdateUri: |
|
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
233 # this client has self-applied the patch already |
|
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
234 continue |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
235 d = c.sendPatch(p) |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
236 d.addErrback(self.clientErrored, c) |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
237 if not dueToFileChange: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
238 self.dirtyFiles([ctx]) |
| 796 | 239 sendToLiveClients(asJson=p.jsonRepr) |
| 240 | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
241 def dirtyFiles(self, ctxs): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
242 """mark dirty the files that we watch in these contexts. |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
243 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
244 the ctx might not be a file that we already read; it might be |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
245 for a new file we have to create, or it might be for a |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
246 transient context that we're not going to save |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
247 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
248 if it's a ctx with no file, error |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
249 """ |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
250 for ctx in ctxs: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
251 g = self.getSubgraph(ctx) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
252 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
253 if ctx not in self.graphFiles: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
254 outFile = self.fileForUri(ctx) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
255 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx, |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
256 self.patch, self.getSubgraph) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
257 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
258 self.graphFiles[ctx].dirty(g) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
259 |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
260 def clientErrored(self, err, c): |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
261 err.trap(twisted.internet.error.ConnectError) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
262 log.info("connection error- dropping client %r" % c) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
263 self.clients.remove(c) |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
264 self.sendClientsToAllLivePages() |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
265 |
| 796 | 266 def summarizeToLog(self): |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
267 log.info("contexts in graph (%s total stmts):" % len(self.graph)) |
| 796 | 268 for c in self.graph.contexts(): |
| 269 log.info(" %s: %s statements" % | |
| 270 (c.identifier, len(self.getSubgraph(c.identifier)))) | |
| 271 | |
| 272 def getSubgraph(self, uri): | |
| 811 | 273 """ |
| 274 this is meant to return a live view of the given subgraph, but | |
| 275 if i'm still working around an rdflib bug, it might return a | |
| 276 copy | |
| 277 | |
| 278 and it's returning triples, but I think quads would be better | |
| 279 """ | |
| 796 | 280 # this is returning an empty Graph :( |
| 281 #return self.graph.get_context(uri) | |
| 282 | |
| 283 g = Graph() | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
284 for s in self.graph.triples(ALLSTMTS, uri): |
| 796 | 285 g.add(s) |
| 286 return g | |
| 287 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
288 def addClient(self, updateUri, label): |
| 796 | 289 [self.clients.remove(c) |
| 290 for c in self.clients if c.updateUri == updateUri] | |
| 291 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
292 log.info("new client %s at %s" % (label, updateUri)) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
293 self.clients.append(Client(updateUri, label, self)) |
| 796 | 294 self.sendClientsToAllLivePages() |
| 295 | |
| 296 def sendClientsToAllLivePages(self): | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
297 sendToLiveClients({"clients":[ |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
298 dict(updateUri=c.updateUri, label=c.label) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
299 for c in self.clients]}) |
| 796 | 300 |
| 301 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler): | |
| 302 def get(self): | |
| 303 pass | |
| 304 | |
| 305 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler): | |
| 306 def __init__(self, *args, **kw): | |
| 307 cyclone.web.RequestHandler.__init__(self, *args, **kw) | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
308 p = syncedgraph.makePatchEndpointPutMethod(self.settings.db.patch) |
| 796 | 309 self.put = lambda: p(self) |
| 310 | |
| 311 def get(self): | |
| 312 pass | |
| 313 | |
| 314 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler): | |
| 315 def get(self): | |
| 316 pass | |
| 317 | |
| 318 def post(self): | |
| 319 upd = self.get_argument("clientUpdate") | |
| 320 try: | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
321 self.settings.db.addClient(upd, self.get_argument("label")) |
| 796 | 322 except: |
| 323 import traceback | |
| 324 traceback.print_exc() | |
| 325 raise | |
| 326 | |
| 327 liveClients = set() | |
| 328 def sendToLiveClients(d=None, asJson=None): | |
| 329 j = asJson or json.dumps(d) | |
| 330 for c in liveClients: | |
| 331 c.sendMessage(j) | |
| 332 | |
| 333 class Live(cyclone.websocket.WebSocketHandler): | |
| 334 | |
| 335 def connectionMade(self, *args, **kwargs): | |
| 811 | 336 log.info("websocket opened") |
| 796 | 337 liveClients.add(self) |
| 338 self.settings.db.sendClientsToAllLivePages() | |
| 339 | |
| 340 def connectionLost(self, reason): | |
| 811 | 341 log.info("websocket closed") |
| 796 | 342 liveClients.remove(self) |
| 343 | |
| 344 def messageReceived(self, message): | |
| 345 log.info("got message %s" % message) | |
| 346 self.sendMessage(message) | |
| 347 | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
348 class NoExts(cyclone.web.StaticFileHandler): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
349 # .xhtml pages can be get() without .xhtml on them |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
350 def get(self, path, *args, **kw): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
351 if path and '.' not in path: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
352 path = path + ".xhtml" |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
353 cyclone.web.StaticFileHandler.get(self, path, *args, **kw) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
354 |
| 796 | 355 if __name__ == "__main__": |
| 356 logging.basicConfig() | |
| 357 log = logging.getLogger() | |
| 358 | |
| 359 parser = optparse.OptionParser() | |
| 360 parser.add_option('--show', | |
| 361 help='show URI, like http://light9.bigasterisk.com/show/dance2008', | |
| 362 default=showconfig.showUri()) | |
| 363 parser.add_option("-v", "--verbose", action="store_true", | |
| 364 help="logging.DEBUG") | |
| 365 (options, args) = parser.parse_args() | |
| 366 | |
| 367 log.setLevel(logging.DEBUG if options.verbose else logging.INFO) | |
| 368 | |
| 369 if not options.show: | |
| 370 raise ValueError("missing --show http://...") | |
| 371 | |
| 372 db = Db() | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
373 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
374 from twisted.python import log as twlog |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
375 twlog.startLogging(sys.stdout) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
376 |
| 796 | 377 port = 8051 |
| 378 reactor.listenTCP(port, cyclone.web.Application(handlers=[ | |
| 379 (r'/live', Live), | |
| 380 (r'/graph', GraphResource), | |
| 381 (r'/patches', Patches), | |
| 382 (r'/graphClients', GraphClients), | |
| 383 | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
384 (r'/(.*)', NoExts, |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
385 {"path" : "light9/rdfdb/web", |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
386 "default_filename" : "index.xhtml"}), |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
387 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
388 ], debug=True, db=db)) |
| 796 | 389 log.info("serving on %s" % port) |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
390 prof.run(reactor.run, profile=False) |