annotate bin/rdfdb @ 1073:52f049d85f96

less CC crashing. curve points are stored in their own n3 files. fix curve row height edge cases. Ignore-this: 7bd8e0c82d4106d53c24d67b981baf60
author Drew Perttula <drewp@bigasterisk.com>
date Tue, 03 Jun 2014 05:09:41 +0000
parents 473db8bebb8f
children 7aca3145c96e
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
815
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
19 Global data undo should probably happen within this service. Some
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
20 operations should not support undo, such as updating the default
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
21 position of a window. How will we separate those? A blacklist of
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
22 subj+pred pairs that don't save undo? Or just save the updates like
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
23 everything else, but when you press undo, there's a way to tell which
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
24 updates *should* be part of your app's undo system?
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
25
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
27 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
28
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
29 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
30 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
32 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
33 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
48 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
53 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
54 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
63 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
64
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
68 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
69 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
70 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
71 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
72
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
76 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
77 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
78 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
79 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
80
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
82
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
86
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
89 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
94
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
95 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
99 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
100
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
103 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
104
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
105 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
106 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
107 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
108
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
110 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
111 import twisted.internet.error
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
112 from twisted.python.filepath import FilePath
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
113 from twisted.internet.inotify import humanReadableMask, IN_CREATE
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
114 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
115 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
117 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
119 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
120 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
121 from light9.rdfdb.rdflibpatch import patchQuads
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
122 from light9.rdfdb.file_vs_uri import correctToTopdirPrefix, fileForUri, uriFromFile
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
123 from light9.rdfdb.patchsender import sendPatch
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
124 from light9.rdfdb.patchreceiver import makePatchEndpointPutMethod
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 from twisted.internet.inotify import INotify
833
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
127 from run_local import log
bdfdfea84510 colored logs
Drew Perttula <drewp@bigasterisk.com>
parents: 830
diff changeset
128 log.setLevel(logging.DEBUG)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129
1060
473db8bebb8f install a copy of cycloneerr.py
Drew Perttula <drewp@bigasterisk.com>
parents: 1045
diff changeset
130 from lib.cycloneerr import PrettyErrorHandler
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
131
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
132 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
133 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
134 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
135 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
136 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
137 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
138 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
139 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
140 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
141
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
142 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
143 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
144
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
145 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
146 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
147 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
149 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
150 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
151 delQuads=[]))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
152
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
153 def sendPatch(self, p):
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
154 return sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
155
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
156 class WatchedFiles(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
157 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
158 find files, notice new files.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
159
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
160 This object watches directories. Each GraphFile watches its own file.
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
161 """
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
162 def __init__(self, dirUriMap, patch, getSubgraph):
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
163 self.dirUriMap = dirUriMap # {abspath : uri prefix}
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
164 self.patch, self.getSubgraph = patch, getSubgraph
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
165
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
166 self.graphFiles = {} # context uri : GraphFile
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
167
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
168 self.notifier = INotify()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
169 self.notifier.startReading()
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
170
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
171 self.findAndLoadFiles()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
172
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
173 def findAndLoadFiles(self):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
174 self.initialLoad = True
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
175 try:
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
176 for topdir in self.dirUriMap:
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
177 for dirpath, dirnames, filenames in os.walk(topdir):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
178 for base in filenames:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
179 self.watchFile(os.path.join(dirpath, base))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
180 self.notifier.watch(FilePath(dirpath), autoAdd=True,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
181 callbacks=[self.dirChange])
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
182 finally:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
183 self.initialLoad = False
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
184
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
185 def dirChange(self, watch, path, mask):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
186 if mask & IN_CREATE:
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
187 log.debug("%s created; consider adding a watch", path)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
188 self.watchFile(path.path)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
189
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
190 def watchFile(self, inFile):
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
191 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
192 consider adding a GraphFile to self.graphFiles
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
193
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
194 inFile needs to be a relative path, not an absolute (e.g. in a
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
195 FilePath) because we use its exact relative form in the
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
196 context URI
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
197 """
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
198 if not os.path.isfile(inFile):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
199 return
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
200
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
201 inFile = correctToTopdirPrefix(self.dirUriMap, inFile)
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
202 if os.path.splitext(inFile)[1] not in ['.n3']:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
203 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
204
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
205 # an n3 file with rules makes it all the way past this reading
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
206 # and the serialization. Then, on the receiving side, a
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
207 # SyncedGraph calls graphFromNQuad on the incoming data and
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
208 # has a parse error. I'm not sure where this should be fixed
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
209 # yet.
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
210 if '-rules' in inFile:
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
211 return
904
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
212
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
213 # for legacy versions, compile all the config stuff you want
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
214 # read into one file called config.n3. New versions won't read
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
215 # it.
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
216 if inFile.endswith("config.n3"):
86c6700d1d63 2013 code ignores 'config.n3' so 2012 code can use that instead
Drew Perttula <drewp@bigasterisk.com>
parents: 899
diff changeset
217 return
830
9ba1c866bf4c workaround for n3 file with rules in it
Drew Perttula <drewp@bigasterisk.com>
parents: 829
diff changeset
218
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
219 ctx = uriFromFile(self.dirUriMap, inFile)
857
1bda494a8c3a attempted fix for FilePath making paths absolute (when they need to stay relative since they're used in uris)
drewp@bigasterisk.com
parents: 843
diff changeset
220 gf = GraphFile(self.notifier, inFile, ctx,
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
221 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
222 self.graphFiles[ctx] = gf
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
223 log.info("%s do initial read", inFile)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
224 gf.reread()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
225
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
226 def aboutToPatch(self, ctx):
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
227 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
228 warn us that a patch is about to come to this context. it's more
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
229 straightforward to create the new file now
931
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
230
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
231 this is meant to make the file before we add triples, so we
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
232 wouldn't see the blank file and lose those triples. But it
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
233 didn't work, so there are other measures that make us not lose
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
234 the triples from a new file. Calling this before patching the
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
235 graph is still a reasonable thing to do, though.
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
236 """
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
237 g = self.getSubgraph(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
238
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
239 if ctx not in self.graphFiles:
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
240 outFile = fileForUri(self.dirUriMap, ctx)
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
241 log.info("starting new file %r", outFile)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
242 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx,
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
243 self.patch, self.getSubgraph)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
244
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
245 def dirtyFiles(self, ctxs):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
246 """mark dirty the files that we watch in these contexts.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
247
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
248 the ctx might not be a file that we already read; it might be
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
249 for a new file we have to create, or it might be for a
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
250 transient context that we're not going to save
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
251
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
252 if it's a ctx with no file, error
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
253 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
254 for ctx in ctxs:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
255 g = self.getSubgraph(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
256 self.graphFiles[ctx].dirty(g)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
257
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
258
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
259 class Db(object):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
260 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
261 the master graph, all the connected clients, all the files we're watching
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
262 """
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
263 def __init__(self, dirUriMap):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
264
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
265 self.clients = []
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
266 self.graph = ConjunctiveGraph()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
267
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
268 self.watchedFiles = WatchedFiles(dirUriMap,
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
269 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
270
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
271 self.summarizeToLog()
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
272
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
273 def patch(self, p, dueToFileChange=False):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
274 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
275 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
276
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
277 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
278 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
279
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
280 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
281 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
282 """
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
283 ctx = p.getContext()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
284 log.info("patching graph %s -%d +%d" % (
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
285 ctx, len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
286
931
3d8ad77176ec rdfdb handles file removals
drewp@bigasterisk.com
parents: 924
diff changeset
287 if hasattr(self, 'watchedFiles'): # not available during startup
924
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
288 self.watchedFiles.aboutToPatch(ctx)
dcab422615ca working on local->global sub action. creation of new file in rdfdb is still a problem
drewp@bigasterisk.com
parents: 921
diff changeset
289
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
290 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
291 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
292
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
293 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
294 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
295 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
296 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
297 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
298 d.addErrback(self.clientErrored, c)
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
299 if not dueToFileChange:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
300 self.watchedFiles.dirtyFiles([ctx])
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
301 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
302
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
303 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
304 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
305 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
306 self.clients.remove(c)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
307 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
308
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
309 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
310 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
311 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
312 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
313 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
314
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
315 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
316 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
317 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
318 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
319 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
320
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
321 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
322 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
323 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
324 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
325
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
326 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
327 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
328 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
329 return g
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
330
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
331 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
332 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
333 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
334
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
335 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
336 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
337 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
338
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
339 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
340 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
341 dict(updateUri=c.updateUri, label=c.label)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
342 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
343
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
344 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
345 def get(self):
905
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
346 accept = self.request.headers.get('accept', '')
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
347 format = 'n3'
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
348 if accept == 'text/plain':
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
349 format = 'nt'
1032
54027815c6cc rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents: 1017
diff changeset
350 elif accept == 'application/n-quads':
54027815c6cc rdfdb can return 'application/n-quads' response
Drew Perttula <drewp@bigasterisk.com>
parents: 1017
diff changeset
351 format = 'nquads'
905
741aa0f9be48 very simple con-neg on rdfdb /graph to support text/plain (ntriples)
Drew Perttula <drewp@bigasterisk.com>
parents: 904
diff changeset
352 self.write(self.settings.db.graph.serialize(format=format))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
353
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
354 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
355 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
356 cyclone.web.RequestHandler.__init__(self, *args, **kw)
843
77b5dbcf688d split syncedgraph into several layers
drewp@bigasterisk.com
parents: 833
diff changeset
357 p = makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
358 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
359
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
360 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
361 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
362
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
363 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
364 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
365 pass
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
366
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
367 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
368 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
369 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
370 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
371 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
372 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
373 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
374 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
375
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
376 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
377 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
378 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
379 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
380 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
381
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
382 class Live(cyclone.websocket.WebSocketHandler):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
383
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
384 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
385 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
386 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
387 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
388
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
389 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
390 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
391 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
392
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
393 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
394 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
395 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
396
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
397 class NoExts(cyclone.web.StaticFileHandler):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
398 # .xhtml pages can be get() without .xhtml on them
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
399 def get(self, path, *args, **kw):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
400 if path and '.' not in path:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
401 path = path + ".xhtml"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
402 cyclone.web.StaticFileHandler.get(self, path, *args, **kw)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
403
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
404 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
405 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
406 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
407
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
408 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
409 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
410 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
411 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
412
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
413 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
414
1045
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
415 db = Db(dirUriMap={os.environ['LIGHT9_SHOW'].rstrip('/') + '/':
c1face79c0e1 fix rdfdb's filename<->uri mapping system
Drew Perttula <drewp@bigasterisk.com>
parents: 1041
diff changeset
416 URIRef('http://light9.bigasterisk.com/show/dance2014/')})
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
417
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
418 from twisted.python import log as twlog
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
419 twlog.startLogging(sys.stdout)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
420
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
421 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
422 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
423 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
424 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
425 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
426 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
427
1041
a4632a7b2e17 upgrade knockout and jquery, simplify the static/ dirs for all web services
Drew Perttula <drewp@bigasterisk.com>
parents: 1032
diff changeset
428 (r'/static/(.*)', cyclone.web.StaticFileHandler, {'path': 'static/'}),
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
429 (r'/(.*)', NoExts,
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
430 {"path" : "light9/rdfdb/web",
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
431 "default_filename" : "index.xhtml"}),
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
432
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
433 ], debug=True, db=db))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
434 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
435 prof.run(reactor.run, profile=False)