annotate bin/rdfdb @ 1045:c1face79c0e1

fix rdfdb's filename<->uri mapping system Ignore-this: 2236b295035273370138943857894699
author Drew Perttula <drewp@bigasterisk.com>
date Thu, 29 May 2014 06:46:18 +0000
parents a4632a7b2e17
children 473db8bebb8f
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
815
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
19 Global data undo should probably happen within this service. Some
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
20 operations should not support undo, such as updating the default
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
21 position of a window. How will we separate those? A blacklist of
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
22 subj+pred pairs that don't save undo? Or just save the updates like
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
23 everything else, but when you press undo, there's a way to tell which
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
24 updates *should* be part of your app's undo system?
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
25
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
27 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
28
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
29 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
30 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
32 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
33 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
48 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
53 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
54 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
63 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
64
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
68 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
69 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
70 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
71 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
72
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
76 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
77 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
78 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
79 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
80
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
82
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
86
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
89 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
94
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
95 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
99 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
100
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
103 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
104
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
105 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
106 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
107 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
108
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
110 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
111 import twisted.internet.error
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
112 from twisted.python.filepath import FilePath
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
113 from twisted.internet.inotify import humanReadableMask, IN_CREATE
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
114 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
115 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
117 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
119 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
120 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
121 from light9.rdfdb.rdflibpatch import patchQuads
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
122 from light9.rdfdb.file_vs_uri import correctToTopdirPrefix, fileForUri, uriFromFile
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
123 from light9.rdfdb.patchsender import sendPatch
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
124 from light9.rdfdb.patchreceiver import makePatchEndpointPutMethod
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 from twisted.internet.inotify import INotify
833
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
127 from run_local import log
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
128 log.setLevel(logging.DEBUG)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
130 try:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
131 import sys
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
132 sys.path.append("../homeauto/lib")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
133 from cycloneerr import PrettyErrorHandler
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
134 except ImportError:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
135 class PrettyErrorHandler(object):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
136 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
137
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
138 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
139 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
140 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
141 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
142 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
143 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
144 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
145 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
146 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
147
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
149 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
150
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
151 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
152 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
153 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
154 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
155 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
156 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
157 delQuads=[]))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
158
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
159 def sendPatch(self, p):
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
160 return sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
161
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
162 class WatchedFiles(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
163 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
164 find files, notice new files.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
165
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
166 This object watches directories. Each GraphFile watches its own file.
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
167 """
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
168 def __init__(self, dirUriMap, patch, getSubgraph):
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
169 self.dirUriMap = dirUriMap # {abspath : uri prefix}
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
170 self.patch, self.getSubgraph = patch, getSubgraph
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
171
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
172 self.graphFiles = {} # context uri : GraphFile
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
173
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
174 self.notifier = INotify()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
175 self.notifier.startReading()
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
176
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
177 self.findAndLoadFiles()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
178
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
179 def findAndLoadFiles(self):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
180 self.initialLoad = True
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
181 try:
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
182 for topdir in self.dirUriMap:
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
183 for dirpath, dirnames, filenames in os.walk(topdir):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
184 for base in filenames:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
185 self.watchFile(os.path.join(dirpath, base))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
186 self.notifier.watch(FilePath(dirpath), autoAdd=True,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
187 callbacks=[self.dirChange])
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
188 finally:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
189 self.initialLoad = False
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
190
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
191 def dirChange(self, watch, path, mask):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
192 if mask & IN_CREATE:
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
193 log.debug("%s created; consider adding a watch", path)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
194 self.watchFile(path.path)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
195
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
196 def watchFile(self, inFile):
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
197 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
198 consider adding a GraphFile to self.graphFiles
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
199
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
200 inFile needs to be a relative path, not an absolute (e.g. in a
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
201 FilePath) because we use its exact relative form in the
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
202 context URI
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
203 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
204 if not os.path.isfile(inFile):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
205 return
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
206
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
207 inFile = correctToTopdirPrefix(self.dirUriMap, inFile)
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
208 if os.path.splitext(inFile)[1] not in ['.n3']:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
209 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
210
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
211 # an n3 file with rules makes it all the way past this reading
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
212 # and the serialization. Then, on the receiving side, a
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
213 # SyncedGraph calls graphFromNQuad on the incoming data and
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
214 # has a parse error. I'm not sure where this should be fixed
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
215 # yet.
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
216 if '-rules' in inFile:
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
217 return
904
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
218
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
219 # for legacy versions, compile all the config stuff you want
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
220 # read into one file called config.n3. New versions won't read
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
221 # it.
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
222 if inFile.endswith("config.n3"):
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
223 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
224
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
225 ctx = uriFromFile(self.dirUriMap, inFile)
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
226 gf = GraphFile(self.notifier, inFile, ctx,
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
227 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
228 self.graphFiles[ctx] = gf
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
229 log.info("%s do initial read", inFile)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
230 gf.reread()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
231
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
232 def aboutToPatch(self, ctx):
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
233 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
234 warn us that a patch is about to come to this context. it's more
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
235 straightforward to create the new file now
931
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
236
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
237 this is meant to make the file before we add triples, so we
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
238 wouldn't see the blank file and lose those triples. But it
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
239 didn't work, so there are other measures that make us not lose
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
240 the triples from a new file. Calling this before patching the
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
241 graph is still a reasonable thing to do, though.
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
242 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
243 g = self.getSubgraph(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
244
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
245 if ctx not in self.graphFiles:
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
246 outFile = fileForUri(self.dirUriMap, ctx)
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
247 log.info("starting new file %r", outFile)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
248 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx,
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
249 self.patch, self.getSubgraph)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
250
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
251 def dirtyFiles(self, ctxs):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
252 """mark dirty the files that we watch in these contexts.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
253
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
254 the ctx might not be a file that we already read; it might be
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
255 for a new file we have to create, or it might be for a
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
256 transient context that we're not going to save
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
257
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
258 if it's a ctx with no file, error
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
259 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
260 for ctx in ctxs:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
261 g = self.getSubgraph(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
262 self.graphFiles[ctx].dirty(g)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
263
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
264
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
265 class Db(object):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
266 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
267 the master graph, all the connected clients, all the files we're watching
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
268 """
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
269 def __init__(self, dirUriMap):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
270
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
271 self.clients = []
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
272 self.graph = ConjunctiveGraph()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
273
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
274 self.watchedFiles = WatchedFiles(dirUriMap,
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
275 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
276
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
277 self.summarizeToLog()
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
278
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
279 def patch(self, p, dueToFileChange=False):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
280 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
281 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
282
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
283 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
284 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
285
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
286 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
287 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
288 """
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
289 ctx = p.getContext()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
290 log.info("patching graph %s -%d +%d" % (
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
291 ctx, len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
292
931
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
293 if hasattr(self, 'watchedFiles'): # not available during startup
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
294 self.watchedFiles.aboutToPatch(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
295
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
296 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
297 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
298
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
299 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
300 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
301 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
302 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
303 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
304 d.addErrback(self.clientErrored, c)
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
305 if not dueToFileChange:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
306 self.watchedFiles.dirtyFiles([ctx])
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
307 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
308
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
309 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
310 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
311 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
312 self.clients.remove(c)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
313 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
314
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
315 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
316 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
317 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
318 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
319 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
320
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
321 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
322 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
323 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
324 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
325 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
326
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
327 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
328 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
329 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
330 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
331
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
332 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
333 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
334 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
335 return g
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
336
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
337 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
338 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
339 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
340
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
341 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
342 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
343 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
344
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
345 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
346 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
347 dict(updateUri=c.updateUri, label=c.label)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
348 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
349
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
350 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
351 def get(self):
905
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
352 accept = self.request.headers.get('accept', '')
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
353 format = 'n3'
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
354 if accept == 'text/plain':
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
355 format = 'nt'
1032
54027815c6cc rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents: 1017
diff changeset
356 elif accept == 'application/n-quads':
54027815c6cc rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents: 1017
diff changeset
357 format = 'nquads'
905
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
358 self.write(self.settings.db.graph.serialize(format=format))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
359
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
360 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
361 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
362 cyclone.web.RequestHandler.__init__(self, *args, **kw)
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
363 p = makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
364 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
365
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
366 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
367 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
368
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
369 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
370 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
371 pass
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
372
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
373 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
374 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
375 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
376 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
377 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
378 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
379 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
380 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
381
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
382 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
383 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
384 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
385 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
386 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
387
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
388 class Live(cyclone.websocket.WebSocketHandler):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
389
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
390 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
391 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
392 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
393 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
394
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
395 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
396 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
397 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
398
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
399 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
400 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
401 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
402
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
403 class NoExts(cyclone.web.StaticFileHandler):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
404 # .xhtml pages can be get() without .xhtml on them
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
405 def get(self, path, *args, **kw):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
406 if path and '.' not in path:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
407 path = path + ".xhtml"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
408 cyclone.web.StaticFileHandler.get(self, path, *args, **kw)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
409
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
410 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
411 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
412 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
413
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
414 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
415 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
416 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
417 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
418
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
419 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
420
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
421 db = Db(dirUriMap={os.environ['LIGHT9_SHOW'].rstrip('/') + '/':
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
422 URIRef('http://light9.bigasterisk.com/show/dance2014/')})
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
423
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
424 from twisted.python import log as twlog
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
425 twlog.startLogging(sys.stdout)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
426
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
427 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
428 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
429 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
430 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
431 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
432 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
433
1041
a4632a7b2e17 upgrade knockout and jquery, simplify the static/ dirs for all web services
Drew Perttula <drewp@bigasterisk.com>
parents: 1032
diff changeset
434 (r'/static/(.*)', cyclone.web.StaticFileHandler, {'path': 'static/'}),
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
435 (r'/(.*)', NoExts,
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
436 {"path" : "light9/rdfdb/web",
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
437 "default_filename" : "index.xhtml"}),
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
438
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
439 ], debug=True, db=db))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
440 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
441 prof.run(reactor.run, profile=False)