annotate bin/rdfdb @ 856:a59d0f4563cc

starting curvecalc editchoice Ignore-this: 3885f61f6addba30ea997cc471fd39a9
author drewp@bigasterisk.com
date Wed, 05 Jun 2013 23:39:44 +0000
parents 77b5dbcf688d
children 1bda494a8c3a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
815
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
19 Global data undo should probably happen within this service. Some
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
20 operations should not support undo, such as updating the default
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
21 position of a window. How will we separate those? A blacklist of
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
22 subj+pred pairs that don't save undo? Or just save the updates like
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
23 everything else, but when you press undo, there's a way to tell which
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
24 updates *should* be part of your app's undo system?
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
25
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
27 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
28
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
29 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
30 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
32 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
33 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
48 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
53 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
54 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
63 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
64
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
68 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
69 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
70 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
71 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
72
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
76 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
77 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
78 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
79 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
80
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
82
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
86
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
89 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
94
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
95 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
99 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
100
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
103 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
104
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
105 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
106 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
107 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
108
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
110 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
111 import twisted.internet.error
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
112 from twisted.python.filepath import FilePath
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
113 from twisted.internet.inotify import humanReadableMask, IN_CREATE
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
114 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
115 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
117 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
119 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
120 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
121 from light9.rdfdb.rdflibpatch import patchQuads
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
122
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
123 from light9.rdfdb.patchsender import sendPatch
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
124 from light9.rdfdb.patchreceiver import makePatchEndpointPutMethod
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 from twisted.internet.inotify import INotify
833
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
127 from run_local import log
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
128 log.setLevel(logging.DEBUG)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
130 try:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
131 import sys
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
132 sys.path.append("../homeauto/lib")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
133 from cycloneerr import PrettyErrorHandler
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
134 except ImportError:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
135 class PrettyErrorHandler(object):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
136 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
137
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
138 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
139 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
140 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
141 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
142 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
143 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
144 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
145 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
146 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
147
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
149 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
150
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
151 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
152 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
153 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
154 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
155 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
156 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
157 delQuads=[]))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
158
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
159 def sendPatch(self, p):
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
160 return sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
161
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
162 class WatchedFiles(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
163 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
164 find files, notice new files.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
165
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
166 This object watches directories. Each GraphFile watches its own file.
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
167 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
168 def __init__(self, topDirsToWatch, patch, getSubgraph):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
169 self.topDirsToWatch = topDirsToWatch
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
170 self.patch, self.getSubgraph = patch, getSubgraph
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
171
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
172 # files from cwd become uris starting with this. *should* be
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
173 # building uris from the show uri in $LIGHT9_SHOW/URI
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
174 # instead. Who wants to keep their data in the same dir tree
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
175 # as the source code?!
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
176 self.topUri = URIRef("http://light9.bigasterisk.com/")
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
177
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
178 self.graphFiles = {} # context uri : GraphFile
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
179
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
180 self.notifier = INotify()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
181 self.notifier.startReading()
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
182
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
183 self.findAndLoadFiles()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
184
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
185 def findAndLoadFiles(self):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
186 self.initialLoad = True
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
187 try:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
188 for topdir in self.topDirsToWatch:
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
189 for dirpath, dirnames, filenames in os.walk(topdir):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
190 for base in filenames:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
191 self.watchFile(os.path.join(dirpath, base))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
192 self.notifier.watch(FilePath(dirpath), autoAdd=True,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
193 callbacks=[self.dirChange])
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
194 finally:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
195 self.initialLoad = False
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
196
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
197 def dirChange(self, watch, path, mask):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
198 if mask & IN_CREATE:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
199 self.watchFile(path.path)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
200
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
201 def watchFile(self, inFile):
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
202 """consider adding a GraphFile to self.graphFiles"""
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
203 if not isinstance(inFile, FilePath):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
204 inFile = FilePath(inFile)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
205 if not inFile.isfile():
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
206 return
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
207 if inFile.splitext()[1] not in ['.n3']:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
208 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
209
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
210 # an n3 file with rules makes it all the way past this reading
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
211 # and the serialization. Then, on the receiving side, a
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
212 # SyncedGraph calls graphFromNQuad on the incoming data and
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
213 # has a parse error. I'm not sure where this should be fixed
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
214 # yet.
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
215 if '-rules' in inFile.path:
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
216 return
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
217
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
218 ctx = self.uriFromFile(inFile)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
219 gf = GraphFile(self.notifier, inFile.path, ctx,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
220 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
221 self.graphFiles[ctx] = gf
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
222 gf.reread()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
223
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
224 def dirtyFiles(self, ctxs):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
225 """mark dirty the files that we watch in these contexts.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
226
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
227 the ctx might not be a file that we already read; it might be
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
228 for a new file we have to create, or it might be for a
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
229 transient context that we're not going to save
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
230
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
231 if it's a ctx with no file, error
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
232 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
233 for ctx in ctxs:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
234 g = self.getSubgraph(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
235
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
236 if ctx not in self.graphFiles:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
237 outFile = self.fileForUri(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
238 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
239 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
240
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
241 self.graphFiles[ctx].dirty(g)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
242
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
243 def uriFromFile(self, filename):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
244 if isinstance(filename, FilePath):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
245 filename = filename.path
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
246 assert filename.endswith('.n3'), filename
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
247 return URIRef(self.topUri + filename[:-len('.n3')])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
248
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
249 def fileForUri(self, ctx):
826
05aabe3d7b02 fix some minor issues with graph contexts
Drew Perttula <drewp@bigasterisk.com>
parents: 821
diff changeset
250 assert isinstance(ctx, URIRef), ctx
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
251 if not ctx.startswith(self.topUri):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
252 raise ValueError("don't know what filename to use for %s" % ctx)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
253 return ctx[len(self.topUri):] + ".n3"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
254
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
255
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
256 class Db(object):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
257 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
258 the master graph, all the connected clients, all the files we're watching
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
259 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
260 def __init__(self, topDirsToWatch):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
261
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
262 self.clients = []
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
263 self.graph = ConjunctiveGraph()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
264
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
265 self.watchedFiles = WatchedFiles(topDirsToWatch,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
266 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
267
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
268 self.summarizeToLog()
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
269
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
270 def patch(self, p, dueToFileChange=False):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
271 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
272 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
273
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
274 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
275 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
276
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
277 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
278 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
279 """
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
280 ctx = p.getContext()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
281 log.info("patching graph %s -%d +%d" % (
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
282 ctx, len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
283
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
284 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
285 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
286
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
287 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
288 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
289 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
290 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
291 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
292 d.addErrback(self.clientErrored, c)
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
293 if not dueToFileChange:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
294 self.watchedFiles.dirtyFiles([ctx])
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
295 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
296
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
297 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
298 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
299 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
300 self.clients.remove(c)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
301 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
302
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
303 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
304 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
305 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
306 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
307 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
308
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
309 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
310 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
311 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
312 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
313 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
314
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
315 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
316 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
317 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
318 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
319
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
320 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
321 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
322 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
323 return g
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
324
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
325 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
326 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
327 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
328
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
329 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
330 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
331 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
332
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
333 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
334 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
335 dict(updateUri=c.updateUri, label=c.label)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
336 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
337
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
338 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
339 def get(self):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
340 self.write(self.settings.db.graph.serialize(format='n3'))
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
341
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
342 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
343 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
344 cyclone.web.RequestHandler.__init__(self, *args, **kw)
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
345 p = makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
346 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
347
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
348 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
349 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
350
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
351 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
352 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
353 pass
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
354
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
355 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
356 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
357 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
358 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
359 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
360 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
361 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
362 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
363
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
364 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
365 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
366 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
367 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
368 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
369
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
370 class Live(cyclone.websocket.WebSocketHandler):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
371
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
372 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
373 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
374 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
375 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
376
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
377 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
378 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
379 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
380
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
381 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
382 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
383 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
384
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
385 class NoExts(cyclone.web.StaticFileHandler):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
386 # .xhtml pages can be get() without .xhtml on them
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
387 def get(self, path, *args, **kw):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
388 if path and '.' not in path:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
389 path = path + ".xhtml"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
390 cyclone.web.StaticFileHandler.get(self, path, *args, **kw)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
391
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
392 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
393 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
394 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
395
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
396 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
397 parser.add_option('--show',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
398 help='show URI, like http://light9.bigasterisk.com/show/dance2008',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
399 default=showconfig.showUri())
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
400 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
401 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
402 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
403
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
404 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
405
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
406 if not options.show:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
407 raise ValueError("missing --show http://...")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
408
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
409 db = Db(topDirsToWatch=['show/dance2013'])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
410
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
411 from twisted.python import log as twlog
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
412 twlog.startLogging(sys.stdout)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
413
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
414 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
415 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
416 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
417 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
418 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
419 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
420
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
421 (r'/(.*)', NoExts,
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
422 {"path" : "light9/rdfdb/web",
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
423 "default_filename" : "index.xhtml"}),
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
424
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
425 ], debug=True, db=db))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
426 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
427 prof.run(reactor.run, profile=False)