Mercurial > code > home > repos > light9
annotate bin/rdfdb @ 1648:ffa3b81c6d95
some missing askedFor auditing
Ignore-this: 581b102bbdaef0dc715e61430d639e01
| author | Drew Perttula <drewp@bigasterisk.com> |
|---|---|
| date | Sat, 10 Jun 2017 10:23:48 +0000 |
| parents | 8603ab0b9fa8 |
| children | 50267ea8c675 |
| rev | line source |
|---|---|
| 796 | 1 #!bin/python |
| 2 """ | |
| 3 other tools POST themselves to here as subscribers to the graph. They | |
| 811 | 4 are providing a URL we can PUT to with graph updates. |
| 796 | 5 |
| 6 we immediately PUT them back all the contents of the graph as a bunch | |
| 7 of adds. | |
| 8 | |
| 811 | 9 later we PUT them back with patches (del/add lists) when there are |
| 796 | 10 changes. |
| 11 | |
| 12 If we fail to reach a registered caller, we forget about it for future | |
| 811 | 13 calls. We could PUT empty diffs as a heartbeat to notice disappearing |
| 796 | 14 callers faster. |
| 15 | |
| 811 | 16 A caller can submit a patch which we'll persist and broadcast to every |
| 17 other client. | |
| 796 | 18 |
|
815
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
19 Global data undo should probably happen within this service. Some |
|
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
20 operations should not support undo, such as updating the default |
|
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
21 position of a window. How will we separate those? A blacklist of |
|
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
22 subj+pred pairs that don't save undo? Or just save the updates like |
|
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
23 everything else, but when you press undo, there's a way to tell which |
|
d7f1f868eb6c
toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents:
814
diff
changeset
|
24 updates *should* be part of your app's undo system? |
| 796 | 25 |
| 26 Maybe some subgraphs are for transient data (e.g. current timecode, | |
| 27 mouse position in curvecalc) that only some listeners want to hear about. | |
| 28 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
29 Deletes are graph-specific, so callers may be surprised to delete a |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
30 stmt from one graph but then find that statement is still true. |
| 796 | 31 |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
32 Alternate plan: would it help to insist that every patch is within |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
33 only one subgraph? I think it's ok for them to span multiple ones. |
| 796 | 34 |
| 35 Inserts can be made on any subgraphs, and each subgraph is saved in | |
| 36 its own file. The file might not be in a format that can express | |
| 37 graphs, so I'm just going to not store the subgraph URI in any file. | |
| 38 | |
| 39 I don't support wildcard deletes, and there are race conditions where a | |
| 40 s-p could end up with unexpected multiple objects. Every client needs | |
| 41 to be ready for this. | |
| 42 | |
| 43 We watch the files and push their own changes back to the clients. | |
| 44 | |
| 45 Persist our client list, to survive restarts. In another rdf file? A | |
| 46 random json one? memcache? Also hold the recent changes. We're not | |
| 47 logging everything forever, though, since the output files and a VCS | |
| 48 shall be used for that | |
| 49 | |
| 50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and | |
| 51 they make for more compact n3 files. I'm not sure if it's going to be | |
| 52 hard to keep the client bnodes in sync though. File rereads would be | |
| 811 | 53 hard, if ever a bnode was used across graphs, so that probably should |
| 796 | 54 not be allowed. |
| 55 | |
| 56 Our API: | |
| 57 | |
| 58 GET / ui | |
| 811 | 59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?) |
| 796 | 60 PUT /patches clients submit changes |
| 61 GET /patches (recent) patches from clients | |
| 62 POST /graphClients clientUpdate={uri} to subscribe | |
| 63 GET /graphClients current clients | |
| 64 | |
| 65 format: | |
| 66 json {"adds" : [[quads]...], | |
| 67 "deletes": [[quads]], | |
| 811 | 68 "senderUpdateUri" : tooluri, |
| 69 "created":tttt // maybe to help resolve some conflicts | |
| 796 | 70 } |
| 71 maybe use some http://json-ld.org/ in there. | |
| 72 | |
| 806 | 73 proposed rule feature: |
| 74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and | |
| 75 rerun the rules when either one changes. Should the sourceFile be able | |
| 76 to specify its own rules file? That would be easier | |
| 77 configuration. How do edits work? Not allowed? Patch the source only? | |
| 78 Also see the source graph loaded into a different ctx, and you can | |
| 79 edit that one and see the results in the output context? | |
| 80 | |
| 796 | 81 Our web ui: |
| 82 | |
| 811 | 83 sections |
| 84 | |
| 85 registered clients | |
| 796 | 86 |
| 811 | 87 recent patches, each one says what client it came from. You can reverse |
| 88 them here. We should be able to take patches that are close in time | |
| 89 and keep updating the same data (e.g. a stream of changes as the user | |
| 90 drags a slider) and collapse them into a single edit for clarity. | |
| 91 | |
| 92 Ways to display patches, using labels and creator/subj icons | |
| 93 where possible: | |
| 796 | 94 |
| 811 | 95 <creator> set <subj>'s <p> to <o> |
| 96 <creator> changed <subj>'s <pred> from <o1> to <o2> | |
| 97 <creator> added <o> to <s> <p> | |
| 98 | |
| 99 raw messages for debugging this client | |
| 806 | 100 |
| 811 | 101 ctx urls take you to-> |
| 102 files, who's dirty, have we seen external changes, notice big | |
| 103 files that are taking a long time to save | |
| 806 | 104 |
| 811 | 105 graph contents. plain rdf browser like an outliner or |
| 106 something. clicking any resource from the other displays takes you | |
| 107 to this, focused on that resource | |
| 803 | 108 |
| 796 | 109 """ |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
110 from twisted.internet import reactor, defer |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
111 import twisted.internet.error |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
112 from twisted.python.filepath import FilePath |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
113 from twisted.python.failure import Failure |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
114 from twisted.internet.inotify import humanReadableMask, IN_CREATE |
| 796 | 115 import sys, optparse, logging, json, os |
| 116 import cyclone.web, cyclone.httpclient, cyclone.websocket | |
| 117 sys.path.append(".") | |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
118 from light9 import networking, showconfig, prof |
| 796 | 119 from rdflib import ConjunctiveGraph, URIRef, Graph |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
120 from light9.rdfdb.graphfile import GraphFile |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
121 from light9.rdfdb.patch import Patch, ALLSTMTS |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
122 from light9.rdfdb.rdflibpatch import patchQuads |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
123 from light9.rdfdb.file_vs_uri import correctToTopdirPrefix, fileForUri, uriFromFile |
|
843
77b5dbcf688d
split syncedgraph into several layers
drewp@bigasterisk.com
parents:
833
diff
changeset
|
124 from light9.rdfdb.patchsender import sendPatch |
|
77b5dbcf688d
split syncedgraph into several layers
drewp@bigasterisk.com
parents:
833
diff
changeset
|
125 from light9.rdfdb.patchreceiver import makePatchEndpointPutMethod |
| 796 | 126 |
| 127 from twisted.internet.inotify import INotify | |
| 833 | 128 from run_local import log |
| 129 log.setLevel(logging.DEBUG) | |
| 796 | 130 |
|
1060
473db8bebb8f
install a copy of cycloneerr.py
Drew Perttula <drewp@bigasterisk.com>
parents:
1045
diff
changeset
|
131 from lib.cycloneerr import PrettyErrorHandler |
| 796 | 132 |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
133 class WebsocketDisconnect(ValueError): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
134 pass |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
135 |
|
1345
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
136 def sendGraphToClient(graph, client): |
|
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
137 """send the client the whole graph contents""" |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
138 log.info("sending all graphs to %r" % client) |
|
1345
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
139 client.sendPatch(Patch( |
|
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
140 addQuads=graph.quads(ALLSTMTS), |
|
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
141 delQuads=[])) |
|
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
142 |
|
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
143 |
| 796 | 144 class Client(object): |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
145 """ |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
146 one of our syncedgraph clients |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
147 """ |
|
1343
a214a9d6f2f0
refactor rdfdb: Client doesn't need Db at construction
Drew Perttula <drewp@bigasterisk.com>
parents:
1309
diff
changeset
|
148 def __init__(self, updateUri, label): |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
149 self.label = label |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
150 # todo: updateUri is used publicly to compare clients. Replace |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
151 # it with Client.__eq__ so WsClient doesn't have to fake an |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
152 # updateUri. |
| 796 | 153 self.updateUri = updateUri |
| 154 | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
155 def __repr__(self): |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
156 return "<%s client at %s>" % (self.label, self.updateUri) |
|
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
157 |
| 796 | 158 def sendPatch(self, p): |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
159 """ |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
160 returns deferred. error will be interpreted as the client being |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
161 broken. |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
162 """ |
|
843
77b5dbcf688d
split syncedgraph into several layers
drewp@bigasterisk.com
parents:
833
diff
changeset
|
163 return sendPatch(self.updateUri, p) |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
164 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
165 class WsClient(object): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
166 def __init__(self, connectionId, sendMessage): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
167 self.updateUri = connectionId |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
168 self.sendMessage = sendMessage |
| 796 | 169 |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
170 def __repr__(self): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
171 return "<WsClient %s>" % self.updateUri |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
172 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
173 def sendPatch(self, p): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
174 self.sendMessage(p.makeJsonRepr()) |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
175 return defer.succeed(None) |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
176 |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
177 class WatchedFiles(object): |
| 811 | 178 """ |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
179 find files, notice new files. |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
180 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
181 This object watches directories. Each GraphFile watches its own file. |
| 811 | 182 """ |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
183 def __init__(self, dirUriMap, patch, getSubgraph, addlPrefixes): |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
184 self.dirUriMap = dirUriMap # {abspath : uri prefix} |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
185 self.patch, self.getSubgraph = patch, getSubgraph |
|
1559
1ba1d46a70a2
put song: prefix into song files
Drew Perttula <drewp@bigasterisk.com>
parents:
1547
diff
changeset
|
186 self.addlPrefixes = addlPrefixes |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
187 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
188 self.graphFiles = {} # context uri : GraphFile |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
189 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
190 self.notifier = INotify() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
191 self.notifier.startReading() |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
192 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
193 self.findAndLoadFiles() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
194 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
195 def findAndLoadFiles(self): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
196 self.initialLoad = True |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
197 try: |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
198 for topdir in self.dirUriMap: |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
199 for dirpath, dirnames, filenames in os.walk(topdir): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
200 for base in filenames: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
201 self.watchFile(os.path.join(dirpath, base)) |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
202 self.notifier.watch(FilePath(dirpath), autoAdd=True, |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
203 callbacks=[self.dirChange]) |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
204 finally: |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
205 self.initialLoad = False |
| 821 | 206 |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
207 def dirChange(self, watch, path, mask): |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
208 if mask & IN_CREATE: |
| 1642 | 209 if path.path.endswith(('~', '.swp', 'swx', '.rdfdb-temp')): |
| 210 return | |
| 211 | |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
212 log.debug("%s created; consider adding a watch", path) |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
213 self.watchFile(path.path) |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
214 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
215 def watchFile(self, inFile): |
|
857
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
216 """ |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
217 consider adding a GraphFile to self.graphFiles |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
218 |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
219 inFile needs to be a relative path, not an absolute (e.g. in a |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
220 FilePath) because we use its exact relative form in the |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
221 context URI |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
222 """ |
|
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
223 if not os.path.isfile(inFile): |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
224 return |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
225 |
|
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
226 inFile = correctToTopdirPrefix(self.dirUriMap, inFile) |
|
857
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
227 if os.path.splitext(inFile)[1] not in ['.n3']: |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
228 return |
|
830
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
229 |
|
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
230 # an n3 file with rules makes it all the way past this reading |
|
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
231 # and the serialization. Then, on the receiving side, a |
|
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
232 # SyncedGraph calls graphFromNQuad on the incoming data and |
|
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
233 # has a parse error. I'm not sure where this should be fixed |
|
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
234 # yet. |
|
857
1bda494a8c3a
attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents:
843
diff
changeset
|
235 if '-rules' in inFile: |
|
830
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
236 return |
|
904
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
237 |
|
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
238 # for legacy versions, compile all the config stuff you want |
|
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
239 # read into one file called config.n3. New versions won't read |
|
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
240 # it. |
|
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
241 if inFile.endswith("config.n3"): |
|
86c6700d1d63
2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents:
899
diff
changeset
|
242 return |
|
830
9ba1c866bf4c
workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents:
829
diff
changeset
|
243 |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
244 ctx = uriFromFile(self.dirUriMap, inFile) |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
245 gf = self._addGraphFile(ctx, inFile) |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
246 log.info("%s do initial read", inFile) |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
247 gf.reread() |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
248 |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
249 def aboutToPatch(self, ctx): |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
250 """ |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
251 warn us that a patch is about to come to this context. it's more |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
252 straightforward to create the new file now |
| 931 | 253 |
| 254 this is meant to make the file before we add triples, so we | |
| 255 wouldn't see the blank file and lose those triples. But it | |
| 256 didn't work, so there are other measures that make us not lose | |
| 257 the triples from a new file. Calling this before patching the | |
| 258 graph is still a reasonable thing to do, though. | |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
259 """ |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
260 g = self.getSubgraph(ctx) |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
261 |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
262 if ctx not in self.graphFiles: |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
263 outFile = fileForUri(self.dirUriMap, ctx) |
|
1282
2c0c3aec0d63
critical fix- put slash in the uri map prefix so it gets removed from uris, so filenames don't have // in them and fail to compare right to other versions
drewp@bigasterisk.com
parents:
1229
diff
changeset
|
264 assert '//' not in outFile, (outFile, self.dirUriMap, ctx) |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
265 log.info("starting new file %r", outFile) |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
266 self._addGraphFile(ctx, outFile) |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
267 |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
268 def _addGraphFile(self, ctx, path): |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
269 self.addlPrefixes.setdefault(ctx, {}) |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
270 self.addlPrefixes.setdefault(None, {}) |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
271 gf = GraphFile(self.notifier, path, ctx, |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
272 self.patch, self.getSubgraph, |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
273 globalPrefixes=self.addlPrefixes[None], |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
274 ctxPrefixes=self.addlPrefixes[ctx]) |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
275 self.graphFiles[ctx] = gf |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
276 return gf |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
277 |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
278 |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
279 def dirtyFiles(self, ctxs): |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
280 """mark dirty the files that we watch in these contexts. |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
281 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
282 the ctx might not be a file that we already read; it might be |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
283 for a new file we have to create, or it might be for a |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
284 transient context that we're not going to save |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
285 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
286 if it's a ctx with no file, error |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
287 """ |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
288 for ctx in ctxs: |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
289 g = self.getSubgraph(ctx) |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
290 self.graphFiles[ctx].dirty(g) |
| 796 | 291 |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
292 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
293 class Db(object): |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
294 """ |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
295 the master graph, all the connected clients, all the files we're watching |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
296 """ |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
297 def __init__(self, dirUriMap, addlPrefixes): |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
298 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
299 self.clients = [] |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
300 self.graph = ConjunctiveGraph() |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
301 |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
302 self.watchedFiles = WatchedFiles(dirUriMap, |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
303 self.patch, self.getSubgraph, |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
304 addlPrefixes) |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
305 |
|
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
306 self.summarizeToLog() |
| 821 | 307 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
308 def patch(self, p, dueToFileChange=False): |
| 796 | 309 """ |
| 310 apply this patch to the master graph then notify everyone about it | |
| 811 | 311 |
| 312 dueToFileChange if this is a patch describing an edit we read | |
| 313 *from* the file (such that we shouldn't write it back to the file) | |
| 314 | |
| 315 if p has a senderUpdateUri attribute, we won't send this patch | |
| 316 back to the sender with that updateUri | |
| 796 | 317 """ |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
318 ctx = p.getContext() |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
319 log.info("patching graph %s -%d +%d" % ( |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
320 ctx, len(p.delQuads), len(p.addQuads))) |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
321 |
| 931 | 322 if hasattr(self, 'watchedFiles'): # not available during startup |
|
924
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
323 self.watchedFiles.aboutToPatch(ctx) |
|
dcab422615ca
working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents:
921
diff
changeset
|
324 |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
325 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True) |
|
1083
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
326 self._sendPatch(p) |
|
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
327 if not dueToFileChange: |
|
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
328 self.watchedFiles.dirtyFiles([ctx]) |
|
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
329 sendToLiveClients(asJson=p.jsonRepr) |
|
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
330 |
|
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
331 def _sendPatch(self, p): |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
332 senderUpdateUri = getattr(p, 'senderUpdateUri', None) |
|
829
e53e78db7b17
refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents:
826
diff
changeset
|
333 |
| 796 | 334 for c in self.clients: |
|
808
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
335 if c.updateUri == senderUpdateUri: |
|
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
336 # this client has self-applied the patch already |
|
a631e075a5bf
KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents:
806
diff
changeset
|
337 continue |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
338 d = c.sendPatch(p) |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
339 d.addErrback(self.clientErrored, c) |
|
1083
7aca3145c96e
refactor rdfdb patch()
Drew Perttula <drewp@bigasterisk.com>
parents:
1060
diff
changeset
|
340 |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
341 def clientErrored(self, err, c): |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
342 err.trap(twisted.internet.error.ConnectError, WebsocketDisconnect) |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
343 log.info("%r %r - dropping client", c, err.getErrorMessage()) |
|
1547
3ca103d97f98
rdfdb don't error on dropping gone clients (even though that does seem like a bug)
Drew Perttula <drewp@bigasterisk.com>
parents:
1351
diff
changeset
|
344 if c in self.clients: |
|
3ca103d97f98
rdfdb don't error on dropping gone clients (even though that does seem like a bug)
Drew Perttula <drewp@bigasterisk.com>
parents:
1351
diff
changeset
|
345 self.clients.remove(c) |
| 821 | 346 self.sendClientsToAllLivePages() |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
347 |
| 796 | 348 def summarizeToLog(self): |
|
798
5c158d37f1ce
autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents:
797
diff
changeset
|
349 log.info("contexts in graph (%s total stmts):" % len(self.graph)) |
| 796 | 350 for c in self.graph.contexts(): |
| 351 log.info(" %s: %s statements" % | |
| 352 (c.identifier, len(self.getSubgraph(c.identifier)))) | |
| 353 | |
| 354 def getSubgraph(self, uri): | |
| 811 | 355 """ |
| 356 this is meant to return a live view of the given subgraph, but | |
| 357 if i'm still working around an rdflib bug, it might return a | |
| 358 copy | |
| 359 | |
| 360 and it's returning triples, but I think quads would be better | |
| 361 """ | |
| 796 | 362 # this is returning an empty Graph :( |
| 363 #return self.graph.get_context(uri) | |
| 364 | |
| 365 g = Graph() | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
366 for s in self.graph.triples(ALLSTMTS, uri): |
| 796 | 367 g.add(s) |
| 368 return g | |
| 821 | 369 |
|
1344
01f7ece0cb13
refactor rdfdb: Db doesn't make Client objects
Drew Perttula <drewp@bigasterisk.com>
parents:
1343
diff
changeset
|
370 def addClient(self, newClient): |
| 796 | 371 [self.clients.remove(c) |
|
1344
01f7ece0cb13
refactor rdfdb: Db doesn't make Client objects
Drew Perttula <drewp@bigasterisk.com>
parents:
1343
diff
changeset
|
372 for c in self.clients if c.updateUri == newClient.updateUri] |
| 796 | 373 |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
374 log.info("new client %r" % newClient) |
|
1345
13f758eda3b4
refactor rdfdb: sendGraphToClient
Drew Perttula <drewp@bigasterisk.com>
parents:
1344
diff
changeset
|
375 sendGraphToClient(self.graph, newClient) |
|
1344
01f7ece0cb13
refactor rdfdb: Db doesn't make Client objects
Drew Perttula <drewp@bigasterisk.com>
parents:
1343
diff
changeset
|
376 self.clients.append(newClient) |
| 796 | 377 self.sendClientsToAllLivePages() |
| 378 | |
| 379 def sendClientsToAllLivePages(self): | |
|
797
904913de4599
deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents:
796
diff
changeset
|
380 sendToLiveClients({"clients":[ |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
381 dict(updateUri=c.updateUri, label=repr(c)) |
| 821 | 382 for c in self.clients]}) |
| 796 | 383 |
| 384 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler): | |
| 385 def get(self): | |
|
905
741aa0f9be48
very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents:
904
diff
changeset
|
386 accept = self.request.headers.get('accept', '') |
|
741aa0f9be48
very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents:
904
diff
changeset
|
387 format = 'n3' |
|
741aa0f9be48
very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents:
904
diff
changeset
|
388 if accept == 'text/plain': |
|
741aa0f9be48
very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents:
904
diff
changeset
|
389 format = 'nt' |
|
1032
54027815c6cc
rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents:
1017
diff
changeset
|
390 elif accept == 'application/n-quads': |
|
54027815c6cc
rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents:
1017
diff
changeset
|
391 format = 'nquads' |
|
1629
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
392 elif accept == 'pickle': |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
393 # don't use this; it's just for speed comparison |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
394 import cPickle as pickle |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
395 pickle.dump(self.settings.db.graph, self, protocol=2) |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
396 return |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
397 elif accept == 'msgpack': |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
398 self.write(repr(self.settings.db.graph.__getstate__)) |
|
2fc0e726a3c3
rdfdb alternate graph formats for timing tests
drewp@bigasterisk.com
parents:
1581
diff
changeset
|
399 return |
|
905
741aa0f9be48
very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents:
904
diff
changeset
|
400 self.write(self.settings.db.graph.serialize(format=format)) |
| 821 | 401 |
| 796 | 402 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler): |
| 403 def __init__(self, *args, **kw): | |
| 404 cyclone.web.RequestHandler.__init__(self, *args, **kw) | |
|
843
77b5dbcf688d
split syncedgraph into several layers
drewp@bigasterisk.com
parents:
833
diff
changeset
|
405 p = makePatchEndpointPutMethod(self.settings.db.patch) |
| 796 | 406 self.put = lambda: p(self) |
| 407 | |
| 408 def get(self): | |
| 409 pass | |
| 410 | |
| 411 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler): | |
| 412 def get(self): | |
| 413 pass | |
| 821 | 414 |
| 796 | 415 def post(self): |
| 416 upd = self.get_argument("clientUpdate") | |
| 417 try: | |
|
1344
01f7ece0cb13
refactor rdfdb: Db doesn't make Client objects
Drew Perttula <drewp@bigasterisk.com>
parents:
1343
diff
changeset
|
418 self.settings.db.addClient(Client(upd, self.get_argument("label"))) |
| 796 | 419 except: |
| 420 import traceback | |
| 421 traceback.print_exc() | |
| 422 raise | |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
423 |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
424 class Prefixes(PrettyErrorHandler, cyclone.web.RequestHandler): |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
425 def post(self): |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
426 suggestion = json.loads(self.request.body) |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
427 addlPrefixes = self.settings.db.watchedFiles.addlPrefixes |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
428 addlPrefixes.setdefault(URIRef(suggestion['ctx']), {}).update(suggestion['prefixes']) |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
429 |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
430 _wsClientSerial = 0 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
431 class WebsocketClient(cyclone.websocket.WebSocketHandler): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
432 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
433 def connectionMade(self, *args, **kwargs): |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
434 global _wsClientSerial |
| 1348 | 435 connectionId = 'connection-%s' % _wsClientSerial |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
436 _wsClientSerial += 1 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
437 |
| 1348 | 438 self.wsClient = WsClient(connectionId, self.sendMessage) |
| 439 log.info("new ws client %r", self.wsClient) | |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
440 self.settings.db.addClient(self.wsClient) |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
441 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
442 def connectionLost(self, reason): |
| 1348 | 443 log.info("bye ws client %r", self.wsClient) |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
444 self.settings.db.clientErrored( |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
445 Failure(WebsocketDisconnect(reason)), self.wsClient) |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
446 |
|
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
447 def messageReceived(self, message): |
|
1351
6044e527d795
ping loop (and some lines from the next patch). My setup seemed to be dropping WS after like 30 sec.
Drew Perttula <drewp@bigasterisk.com>
parents:
1350
diff
changeset
|
448 if message == 'PING': |
|
6044e527d795
ping loop (and some lines from the next patch). My setup seemed to be dropping WS after like 30 sec.
Drew Perttula <drewp@bigasterisk.com>
parents:
1350
diff
changeset
|
449 self.sendMessage('PONG') |
|
6044e527d795
ping loop (and some lines from the next patch). My setup seemed to be dropping WS after like 30 sec.
Drew Perttula <drewp@bigasterisk.com>
parents:
1350
diff
changeset
|
450 return |
|
1350
36f58b2aa8ef
browser syncedgraph sends patches back to server
Drew Perttula <drewp@bigasterisk.com>
parents:
1348
diff
changeset
|
451 log.info("got message from %r: %s", self.wsClient, message) |
|
36f58b2aa8ef
browser syncedgraph sends patches back to server
Drew Perttula <drewp@bigasterisk.com>
parents:
1348
diff
changeset
|
452 p = Patch(jsonRepr=message) |
|
36f58b2aa8ef
browser syncedgraph sends patches back to server
Drew Perttula <drewp@bigasterisk.com>
parents:
1348
diff
changeset
|
453 p.senderUpdateUri = self.wsClient.updateUri |
|
36f58b2aa8ef
browser syncedgraph sends patches back to server
Drew Perttula <drewp@bigasterisk.com>
parents:
1348
diff
changeset
|
454 self.settings.db.patch(p) |
| 796 | 455 |
| 456 liveClients = set() | |
| 457 def sendToLiveClients(d=None, asJson=None): | |
| 458 j = asJson or json.dumps(d) | |
| 459 for c in liveClients: | |
| 460 c.sendMessage(j) | |
| 461 | |
| 462 class Live(cyclone.websocket.WebSocketHandler): | |
| 821 | 463 |
| 796 | 464 def connectionMade(self, *args, **kwargs): |
| 811 | 465 log.info("websocket opened") |
| 796 | 466 liveClients.add(self) |
| 467 self.settings.db.sendClientsToAllLivePages() | |
| 468 | |
| 469 def connectionLost(self, reason): | |
| 811 | 470 log.info("websocket closed") |
| 796 | 471 liveClients.remove(self) |
| 472 | |
| 473 def messageReceived(self, message): | |
| 474 log.info("got message %s" % message) | |
| 475 self.sendMessage(message) | |
| 476 | |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
477 class NoExts(cyclone.web.StaticFileHandler): |
| 1309 | 478 # .html pages can be get() without .html on them |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
479 def get(self, path, *args, **kw): |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
480 if path and '.' not in path: |
| 1309 | 481 path = path + ".html" |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
482 cyclone.web.StaticFileHandler.get(self, path, *args, **kw) |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
483 |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
484 |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
485 |
| 796 | 486 if __name__ == "__main__": |
| 487 logging.basicConfig() | |
| 488 log = logging.getLogger() | |
| 489 | |
| 490 parser = optparse.OptionParser() | |
| 491 parser.add_option("-v", "--verbose", action="store_true", | |
| 492 help="logging.DEBUG") | |
| 493 (options, args) = parser.parse_args() | |
| 494 | |
| 495 log.setLevel(logging.DEBUG if options.verbose else logging.INFO) | |
| 496 | |
|
1045
c1face79c0e1
fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents:
1041
diff
changeset
|
497 db = Db(dirUriMap={os.environ['LIGHT9_SHOW'].rstrip('/') + '/': |
|
1559
1ba1d46a70a2
put song: prefix into song files
Drew Perttula <drewp@bigasterisk.com>
parents:
1547
diff
changeset
|
498 showconfig.showUri() + '/'}, |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
499 addlPrefixes={None: { |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
500 'show': showconfig.showUri() + '/', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
501 '': 'http://light9.bigasterisk.com/', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
502 'rdf': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
503 'rdfs': 'http://www.w3.org/2000/01/rdf-schema#', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
504 'xsd': 'http://www.w3.org/2001/XMLSchema#', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
505 'effect': 'http://light9.bigasterisk.com/effect/', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
506 'dev': 'http://light9.bigasterisk.com/device/', |
|
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
507 }}) |
| 821 | 508 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
509 from twisted.python import log as twlog |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
510 twlog.startLogging(sys.stdout) |
| 821 | 511 |
|
1225
70f42f9d6e04
simplify homepageConfig. get homepage port from rdf. get rdfdb port from rdf
Drew Perttula <drewp@bigasterisk.com>
parents:
1212
diff
changeset
|
512 reactor.listenTCP(networking.rdfdb.port, cyclone.web.Application(handlers=[ |
| 796 | 513 (r'/live', Live), |
| 514 (r'/graph', GraphResource), | |
| 515 (r'/patches', Patches), | |
| 516 (r'/graphClients', GraphClients), | |
|
1347
5c54a1f94050
browser SyncedGraph client connects and receives patches
Drew Perttula <drewp@bigasterisk.com>
parents:
1345
diff
changeset
|
517 (r'/syncedGraph', WebsocketClient), |
|
1581
30c79f1dc4f8
fix suggestPrefixes to send suggestions to rdfdb
Drew Perttula <drewp@bigasterisk.com>
parents:
1559
diff
changeset
|
518 (r'/prefixes', Prefixes), |
| 796 | 519 |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
520 (r'/(.*)', NoExts, |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
521 {"path" : "light9/rdfdb/web", |
| 1309 | 522 "default_filename" : "index.html"}), |
|
814
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
523 |
|
1ae8e6b287e3
improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents:
811
diff
changeset
|
524 ], debug=True, db=db)) |
|
1225
70f42f9d6e04
simplify homepageConfig. get homepage port from rdf. get rdfdb port from rdf
Drew Perttula <drewp@bigasterisk.com>
parents:
1212
diff
changeset
|
525 log.info("serving on %s" % networking.rdfdb.port) |
|
1178
a296d842d0da
better cmdline option error
Drew Perttula <drewp@bigasterisk.com>
parents:
1083
diff
changeset
|
526 prof.run(reactor.run, profile=None) |