Mercurial > code > home > repos > light9

annotate bin/rdfdb @ 929:c20c2eea6fce

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
another hack to stop graphfile from thinking its own new empty file means it should delete all the statements in that context Ignore-this: 94639afbbe8d3a0abd8a06e16922394f
author Drew Perttula <drewp@bigasterisk.com>
date Tue, 11 Jun 2013 20:25:02 +0000
parents dcab422615ca
children 3d8ad77176ec
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
815
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
19 Global data undo should probably happen within this service. Some
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
20 operations should not support undo, such as updating the default
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
21 position of a window. How will we separate those? A blacklist of
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
22 subj+pred pairs that don't save undo? Or just save the updates like
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
23 everything else, but when you press undo, there's a way to tell which
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
24 updates *should* be part of your app's undo system?
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
25
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
27 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
28
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
29 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
30 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
32 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
33 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
48 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
53 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
54 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
63 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
64
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
68 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
69 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
70 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
71 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
72
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
76 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
77 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
78 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
79 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
80
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
82
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
86
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
89 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
94
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
95 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
99 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
100
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
103 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
104
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
105 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
106 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
107 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
108
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
110 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
111 import twisted.internet.error
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
112 from twisted.python.filepath import FilePath
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
113 from twisted.internet.inotify import humanReadableMask, IN_CREATE
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
114 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
115 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
117 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
119 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
120 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
121 from light9.rdfdb.rdflibpatch import patchQuads
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
122
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
123 from light9.rdfdb.patchsender import sendPatch
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
124 from light9.rdfdb.patchreceiver import makePatchEndpointPutMethod
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 from twisted.internet.inotify import INotify
833
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
127 from run_local import log
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
128 log.setLevel(logging.DEBUG)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
130 try:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
131 import sys
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
132 sys.path.append("../homeauto/lib")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
133 from cycloneerr import PrettyErrorHandler
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
134 except ImportError:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
135 class PrettyErrorHandler(object):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
136 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
137
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
138 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
139 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
140 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
141 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
142 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
143 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
144 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
145 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
146 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
147
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
149 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
150
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
151 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
152 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
153 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
154 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
155 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
156 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
157 delQuads=[]))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
158
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
159 def sendPatch(self, p):
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
160 return sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
161
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
162 class WatchedFiles(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
163 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
164 find files, notice new files.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
165
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
166 This object watches directories. Each GraphFile watches its own file.
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
167 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
168 def __init__(self, topDirsToWatch, patch, getSubgraph):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
169 self.topDirsToWatch = topDirsToWatch
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
170 self.patch, self.getSubgraph = patch, getSubgraph
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
171
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
172 # files from cwd become uris starting with this. *should* be
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
173 # building uris from the show uri in $LIGHT9_SHOW/URI
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
174 # instead. Who wants to keep their data in the same dir tree
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
175 # as the source code?!
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
176 self.topUri = URIRef("http://light9.bigasterisk.com/")
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
177
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
178 self.graphFiles = {} # context uri : GraphFile
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
179
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
180 self.notifier = INotify()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
181 self.notifier.startReading()
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
182
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
183 self.findAndLoadFiles()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
184
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
185 def findAndLoadFiles(self):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
186 self.initialLoad = True
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
187 try:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
188 for topdir in self.topDirsToWatch:
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
189 for dirpath, dirnames, filenames in os.walk(topdir):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
190 for base in filenames:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
191 self.watchFile(os.path.join(dirpath, base))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
192 self.notifier.watch(FilePath(dirpath), autoAdd=True,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
193 callbacks=[self.dirChange])
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
194 finally:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
195 self.initialLoad = False
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
196
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
197 def dirChange(self, watch, path, mask):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
198 if mask & IN_CREATE:
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
199 log.debug("%s created; consider adding a watch", path)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
200 self.watchFile(path.path)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
201
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
202 def watchFile(self, inFile):
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
203 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
204 consider adding a GraphFile to self.graphFiles
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
205
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
206 inFile needs to be a relative path, not an absolute (e.g. in a
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
207 FilePath) because we use its exact relative form in the
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
208 context URI
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
209 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
210 if not os.path.isfile(inFile):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
211 return
921
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
212 if not any(inFile.startswith(prefix) for prefix in self.topDirsToWatch):
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
213 for prefix in self.topDirsToWatch:
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
214 prefixAbs = os.path.abspath(prefix)
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
215 if inFile.startswith(prefixAbs):
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
216 inFile = prefix + inFile[len(prefixAbs):]
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
217 break
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
218 else:
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
219 raise ValueError("can't correct %s to start with one of %s" %
e673bdbe0be9 rdfdb: when a new n3 file shows up, fix its path to be relative again so the uri ends up right
drewp@bigasterisk.com
parents: 905
diff changeset
220 (inFile, self.topDirsToWatch))
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
221 if os.path.splitext(inFile)[1] not in ['.n3']:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
222 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
223
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
224 # an n3 file with rules makes it all the way past this reading
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
225 # and the serialization. Then, on the receiving side, a
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
226 # SyncedGraph calls graphFromNQuad on the incoming data and
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
227 # has a parse error. I'm not sure where this should be fixed
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
228 # yet.
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
229 if '-rules' in inFile:
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
230 return
904
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
231
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
232 # for legacy versions, compile all the config stuff you want
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
233 # read into one file called config.n3. New versions won't read
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
234 # it.
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
235 if inFile.endswith("config.n3"):
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
236 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
237
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
238 ctx = self.uriFromFile(inFile)
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
239 gf = GraphFile(self.notifier, inFile, ctx,
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
240 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
241 self.graphFiles[ctx] = gf
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
242 log.info("%s do initial read", inFile)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
243 gf.reread()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
244
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
245 def aboutToPatch(self, ctx):
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
246 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
247 warn us that a patch is about to come to this context. it's more
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
248 straightforward to create the new file now
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
249 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
250
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
251 g = self.getSubgraph(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
252
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
253 if ctx not in self.graphFiles:
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
254 outFile = self.fileForUri(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
255 log.info("starting new file %r", outFile)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
256 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx,
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
257 self.patch, self.getSubgraph)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
258
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
259 def dirtyFiles(self, ctxs):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
260 """mark dirty the files that we watch in these contexts.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
261
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
262 the ctx might not be a file that we already read; it might be
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
263 for a new file we have to create, or it might be for a
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
264 transient context that we're not going to save
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
265
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
266 if it's a ctx with no file, error
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
267 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
268 for ctx in ctxs:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
269 g = self.getSubgraph(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
270 self.graphFiles[ctx].dirty(g)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
271
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
272 def uriFromFile(self, filename):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
273 assert filename.endswith('.n3'), filename
899
21974e0ef0c4 guard against an abspath getting into a uri
Drew Perttula <drewp@bigasterisk.com>
parents: 857
diff changeset
274 if not any(filename.startswith(t) for t in self.topDirsToWatch):
21974e0ef0c4 guard against an abspath getting into a uri
Drew Perttula <drewp@bigasterisk.com>
parents: 857
diff changeset
275 raise ValueError("filename %s doesn't start with any of %s" %
21974e0ef0c4 guard against an abspath getting into a uri
Drew Perttula <drewp@bigasterisk.com>
parents: 857
diff changeset
276 (filename, self.topDirsToWatch))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
277 return URIRef(self.topUri + filename[:-len('.n3')])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
278
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
279 def fileForUri(self, ctx):
826
05aabe3d7b02 fix some minor issues with graph contexts
Drew Perttula <drewp@bigasterisk.com>
parents: 821
diff changeset
280 assert isinstance(ctx, URIRef), ctx
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
281 if not ctx.startswith(self.topUri):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
282 raise ValueError("don't know what filename to use for %s" % ctx)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
283 return ctx[len(self.topUri):] + ".n3"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
284
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
285
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
286 class Db(object):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
287 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
288 the master graph, all the connected clients, all the files we're watching
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
289 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
290 def __init__(self, topDirsToWatch):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
291
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
292 self.clients = []
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
293 self.graph = ConjunctiveGraph()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
294
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
295 self.watchedFiles = WatchedFiles(topDirsToWatch,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
296 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
297
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
298 self.summarizeToLog()
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
299
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
300 def patch(self, p, dueToFileChange=False):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
301 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
302 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
303
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
304 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
305 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
306
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
307 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
308 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
309 """
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
310 ctx = p.getContext()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
311 log.info("patching graph %s -%d +%d" % (
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
312 ctx, len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
313
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
314 if hasattr(self, 'watchedFiles'): # not during startup
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
315 self.watchedFiles.aboutToPatch(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
316
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
317 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
318 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
319
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
320 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
321 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
322 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
323 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
324 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
325 d.addErrback(self.clientErrored, c)
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
326 if not dueToFileChange:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
327 self.watchedFiles.dirtyFiles([ctx])
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
328 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
329
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
330 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
331 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
332 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
333 self.clients.remove(c)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
334 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
335
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
336 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
337 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
338 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
339 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
340 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
341
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
342 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
343 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
344 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
345 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
346 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
347
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
348 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
349 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
350 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
351 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
352
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
353 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
354 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
355 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
356 return g
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
357
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
358 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
359 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
360 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
361
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
362 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
363 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
364 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
365
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
366 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
367 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
368 dict(updateUri=c.updateUri, label=c.label)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
369 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
370
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
371 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
372 def get(self):
905
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
373 accept = self.request.headers.get('accept', '')
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
374 format = 'n3'
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
375 if accept == 'text/plain':
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
376 format = 'nt'
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
377 self.write(self.settings.db.graph.serialize(format=format))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
378
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
379 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
380 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
381 cyclone.web.RequestHandler.__init__(self, *args, **kw)
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
382 p = makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
383 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
384
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
385 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
386 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
387
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
388 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
389 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
390 pass
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
391
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
392 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
393 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
394 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
395 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
396 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
397 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
398 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
399 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
400
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
401 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
402 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
403 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
404 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
405 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
406
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
407 class Live(cyclone.websocket.WebSocketHandler):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
408
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
409 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
410 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
411 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
412 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
413
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
414 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
415 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
416 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
417
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
418 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
419 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
420 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
421
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
422 class NoExts(cyclone.web.StaticFileHandler):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
423 # .xhtml pages can be get() without .xhtml on them
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
424 def get(self, path, *args, **kw):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
425 if path and '.' not in path:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
426 path = path + ".xhtml"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
427 cyclone.web.StaticFileHandler.get(self, path, *args, **kw)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
428
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
429 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
430 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
431 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
432
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
433 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
434 parser.add_option('--show',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
435 help='show URI, like http://light9.bigasterisk.com/show/dance2008',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
436 default=showconfig.showUri())
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
437 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
438 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
439 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
440
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
441 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
442
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
443 if not options.show:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
444 raise ValueError("missing --show http://...")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
445
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
446 db = Db(topDirsToWatch=['show/dance2013'])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
447
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
448 from twisted.python import log as twlog
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
449 twlog.startLogging(sys.stdout)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
450
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
451 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
452 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
453 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
454 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
455 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
456 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
457
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
458 (r'/(.*)', NoExts,
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
459 {"path" : "light9/rdfdb/web",
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
460 "default_filename" : "index.xhtml"}),
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
461
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
462 ], debug=True, db=db))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
463 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
464 prof.run(reactor.run, profile=False)