annotate bin/rdfdb @ 811:b19cd005a491

just comments Ignore-this: 850b0da90240ab6f764430505809fd6d
author drewp@bigasterisk.com
date Sat, 29 Sep 2012 20:59:13 +0000
parents a631e075a5bf
children 1ae8e6b287e3
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
19 Global data undo should probably happen within this service.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
20
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
21 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
22 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
23
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
24 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
25 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
27 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
28 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
29
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
30 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
32 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
33
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
48 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
53 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
54 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
59
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
63 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
64 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
68 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
69 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
70 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
71 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
72 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
76 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
77
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
78 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
79
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
80 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
82 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
86
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
89
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
94 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
95
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
99
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
100 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
103
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
104 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
105 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
106 import twisted.internet.error
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
107 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
108 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
110 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
111 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
112 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
113 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
114 from light9.rdfdb.rdflibpatch import patchQuads
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
115 from light9.rdfdb import syncedgraph
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
117 from twisted.internet.inotify import INotify
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 logging.basicConfig(level=logging.DEBUG)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
119 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
120
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
121 try:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
122 import sys
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
123 sys.path.append("../homeauto/lib")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
124 from cycloneerr import PrettyErrorHandler
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125 except ImportError:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 class PrettyErrorHandler(object):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
127 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
128
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
130 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
131 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
132 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
133 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
134 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
135 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
136 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
137 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
138
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
139 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
140 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
141
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
142 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
143 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
144 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
145 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
146 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
147 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148 delQuads=[]))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
149
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
150 def sendPatch(self, p):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
151 return syncedgraph.sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
152
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
153 class Db(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
154 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
155 the master graph, all the connected clients, all the files we're watching
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
156 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
157 def __init__(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
158 self.clients = []
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
159 self.graph = ConjunctiveGraph()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
160
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
161 notifier = INotify()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
162 notifier.startReading()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
163
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
164 for inFile in [#"show/dance2012/config.n3",
801
caeaa88430b8 KC sends real levels
drewp@bigasterisk.com
parents: 799
diff changeset
165 "show/dance2012/patch.n3",
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
166 "show/dance2012/subs/bcools",
799
fcf95ff23cc5 PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents: 798
diff changeset
167 "show/dance2012/subs/bwarm",
fcf95ff23cc5 PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents: 798
diff changeset
168 "show/dance2012/subs/house",
fcf95ff23cc5 PersistentSubmaster split. keyboardcomposer now notices submaster changes
drewp@bigasterisk.com
parents: 798
diff changeset
169 "demo.n3",
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
170 ]:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
171 self.g = GraphFile(notifier,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
172 inFile,
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
173 URIRef("http://example.com/file/%s" %
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
174 os.path.basename(inFile)),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
175 self.patch,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
176 self.getSubgraph)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
177
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
178 def patch(self, p):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
179 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
180 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
181
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
182 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
183 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
184
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
185 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
186 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
187 """
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
188 log.info("patching graph -%d +%d" % (len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
189
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
190 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
191 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
192 self.summarizeToLog()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
193 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
194 print "send to %s? %s %s" % (c, c.updateUri, senderUpdateUri)
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
195 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
196 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
197 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
198 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
199 d.addErrback(self.clientErrored, c)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
200 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
201
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
202 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
203 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
204 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
205 self.clients.remove(c)
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
206 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
207
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
208 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
209 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
210 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
211 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
212 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
213
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
214 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
215 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
216 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
217 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
218 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
219
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
220 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
221 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
222 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
223 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
224
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
225 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
226 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
227 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
228 return g
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
229
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
230 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
231 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
232 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
233
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
234 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
235 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
236 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
237
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
238 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
239 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
240 dict(updateUri=c.updateUri, label=c.label)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
241 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
242
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
243 class Index(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
244 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
245 self.set_header("Content-Type", "application/xhtml+xml")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
246 self.write(open("light9/rdfdb.xhtml").read())
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
247
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
248 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
249 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
250 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
251
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
252 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
253 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
254 cyclone.web.RequestHandler.__init__(self, *args, **kw)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
255 p = syncedgraph.makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
256 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
257
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
258 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
259 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
260
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
261
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
262 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
263 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
264 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
265
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
266 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
267 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
268 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
269 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
270 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
271 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
272 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
273 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
274
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
275 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
276 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
277 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
278 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
279 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
280
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
281 class Live(cyclone.websocket.WebSocketHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
282
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
283 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
284 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
285 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
286 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
287
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
288 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
289 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
290 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
291
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
292 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
293 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
294 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
295
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
296 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
297 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
298 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
299
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
300 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
301 parser.add_option('--show',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
302 help='show URI, like http://light9.bigasterisk.com/show/dance2008',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
303 default=showconfig.showUri())
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
304 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
305 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
306 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
307
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
308 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
309
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
310 if not options.show:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
311 raise ValueError("missing --show http://...")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
312
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
313 db = Db()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
314
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
315 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
316 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
317 (r'/', Index),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
318 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
319 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
320 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
321 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
322
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
323 (r"/(jquery-1\.7\.2\.min\.js)", cyclone.web.StaticFileHandler,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
324 dict(path='lib')),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
325
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
326 ], db=db))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
327 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
328 prof.run(reactor.run, profile=False)