annotate bin/rdfdb @ 829:e53e78db7b17

refactor file watching. notice new files and dirs. Ignore-this: 7f571fd9b309b54cff22b2fee5e68322
author Drew Perttula <drewp@bigasterisk.com>
date Tue, 04 Jun 2013 22:46:34 +0000
parents 05aabe3d7b02
children 9ba1c866bf4c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
1 #!bin/python
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
2 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
3 other tools POST themselves to here as subscribers to the graph. They
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
4 are providing a URL we can PUT to with graph updates.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
5
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
6 we immediately PUT them back all the contents of the graph as a bunch
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
7 of adds.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
8
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
9 later we PUT them back with patches (del/add lists) when there are
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
10 changes.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
11
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
12 If we fail to reach a registered caller, we forget about it for future
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
13 calls. We could PUT empty diffs as a heartbeat to notice disappearing
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
14 callers faster.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
15
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
16 A caller can submit a patch which we'll persist and broadcast to every
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
17 other client.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
18
815
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
19 Global data undo should probably happen within this service. Some
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
20 operations should not support undo, such as updating the default
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
21 position of a window. How will we separate those? A blacklist of
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
22 subj+pred pairs that don't save undo? Or just save the updates like
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
23 everything else, but when you press undo, there's a way to tell which
d7f1f868eb6c toplevel window pos is saved in the graph. Patch conflicts no longer break as hard, but they don't exactly reset themselves right yet eiher
drewp@bigasterisk.com
parents: 814
diff changeset
24 updates *should* be part of your app's undo system?
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
25
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
26 Maybe some subgraphs are for transient data (e.g. current timecode,
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
27 mouse position in curvecalc) that only some listeners want to hear about.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
28
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
29 Deletes are graph-specific, so callers may be surprised to delete a
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
30 stmt from one graph but then find that statement is still true.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
31
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
32 Alternate plan: would it help to insist that every patch is within
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
33 only one subgraph? I think it's ok for them to span multiple ones.
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
34
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
35 Inserts can be made on any subgraphs, and each subgraph is saved in
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
36 its own file. The file might not be in a format that can express
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
37 graphs, so I'm just going to not store the subgraph URI in any file.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
38
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
39 I don't support wildcard deletes, and there are race conditions where a
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
40 s-p could end up with unexpected multiple objects. Every client needs
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
41 to be ready for this.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
42
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
43 We watch the files and push their own changes back to the clients.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
44
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
45 Persist our client list, to survive restarts. In another rdf file? A
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
46 random json one? memcache? Also hold the recent changes. We're not
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
47 logging everything forever, though, since the output files and a VCS
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
48 shall be used for that
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
49
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
50 Bnodes: this rdfdb graph might be able to track bnodes correctly, and
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
51 they make for more compact n3 files. I'm not sure if it's going to be
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
52 hard to keep the client bnodes in sync though. File rereads would be
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
53 hard, if ever a bnode was used across graphs, so that probably should
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
54 not be allowed.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
55
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
56 Our API:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
57
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
58 GET / ui
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
59 GET /graph the whole graph, or a query from it (needed? just for ui browsing?)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
60 PUT /patches clients submit changes
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
61 GET /patches (recent) patches from clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
62 POST /graphClients clientUpdate={uri} to subscribe
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
63 GET /graphClients current clients
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
64
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
65 format:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
66 json {"adds" : [[quads]...],
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
67 "deletes": [[quads]],
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
68 "senderUpdateUri" : tooluri,
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
69 "created":tttt // maybe to help resolve some conflicts
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
70 }
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
71 maybe use some http://json-ld.org/ in there.
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
72
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
73 proposed rule feature:
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
74 rdfdb should be able to watch a pair of (sourceFile, rulesFile) and
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
75 rerun the rules when either one changes. Should the sourceFile be able
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
76 to specify its own rules file? That would be easier
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
77 configuration. How do edits work? Not allowed? Patch the source only?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
78 Also see the source graph loaded into a different ctx, and you can
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
79 edit that one and see the results in the output context?
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
80
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
81 Our web ui:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
82
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
83 sections
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
84
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
85 registered clients
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
86
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
87 recent patches, each one says what client it came from. You can reverse
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
88 them here. We should be able to take patches that are close in time
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
89 and keep updating the same data (e.g. a stream of changes as the user
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
90 drags a slider) and collapse them into a single edit for clarity.
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
91
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
92 Ways to display patches, using labels and creator/subj icons
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
93 where possible:
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
94
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
95 <creator> set <subj>'s <p> to <o>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
96 <creator> changed <subj>'s <pred> from <o1> to <o2>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
97 <creator> added <o> to <s> <p>
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
98
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
99 raw messages for debugging this client
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
100
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
101 ctx urls take you to->
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
102 files, who's dirty, have we seen external changes, notice big
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
103 files that are taking a long time to save
806
6d8f0c088a26 logging and notes
drewp@bigasterisk.com
parents: 803
diff changeset
104
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
105 graph contents. plain rdf browser like an outliner or
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
106 something. clicking any resource from the other displays takes you
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
107 to this, focused on that resource
803
ce4fffe8e413 update SC to read rdf graph
drewp@bigasterisk.com
parents: 801
diff changeset
108
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
109 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
110 from twisted.internet import reactor
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
111 import twisted.internet.error
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
112 from twisted.python.filepath import FilePath
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
113 from twisted.internet.inotify import humanReadableMask, IN_CREATE
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
114 import sys, optparse, logging, json, os
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
115 import cyclone.web, cyclone.httpclient, cyclone.websocket
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
116 sys.path.append(".")
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
117 from light9 import networking, showconfig, prof
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
118 from rdflib import ConjunctiveGraph, URIRef, Graph
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
119 from light9.rdfdb.graphfile import GraphFile
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
120 from light9.rdfdb.patch import Patch, ALLSTMTS
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
121 from light9.rdfdb.rdflibpatch import patchQuads
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
122 from light9.rdfdb import syncedgraph
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
123
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
124 from twisted.internet.inotify import INotify
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
125 logging.basicConfig(level=logging.DEBUG)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
126 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
127
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
128 try:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
129 import sys
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
130 sys.path.append("../homeauto/lib")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
131 from cycloneerr import PrettyErrorHandler
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
132 except ImportError:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
133 class PrettyErrorHandler(object):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
134 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
135
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
136 class Client(object):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
137 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
138 one of our syncedgraph clients
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
139 """
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
140 def __init__(self, updateUri, label, db):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
141 self.db = db
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
142 self.label = label
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
143 self.updateUri = updateUri
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
144 self.sendAll()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
145
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
146 def __repr__(self):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
147 return "<%s client at %s>" % (self.label, self.updateUri)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
148
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
149 def sendAll(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
150 """send the client the whole graph contents"""
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
151 log.info("sending all graphs to %s at %s" %
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
152 (self.label, self.updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
153 self.sendPatch(Patch(
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
154 addQuads=self.db.graph.quads(ALLSTMTS),
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
155 delQuads=[]))
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
156
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
157 def sendPatch(self, p):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
158 return syncedgraph.sendPatch(self.updateUri, p)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
159
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
160 class WatchedFiles(object):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
161 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
162 find files, notice new files.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
163
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
164 This object watches directories. Each GraphFile watches its own file.
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
165 """
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
166 def __init__(self, topDirsToWatch, patch, getSubgraph):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
167 self.topDirsToWatch = topDirsToWatch
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
168 self.patch, self.getSubgraph = patch, getSubgraph
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
169
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
170 # files from cwd become uris starting with this. *should* be
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
171 # building uris from the show uri in $LIGHT9_SHOW/URI
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
172 # instead. Who wants to keep their data in the same dir tree
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
173 # as the source code?!
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
174 self.topUri = URIRef("http://light9.bigasterisk.com/")
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
175
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
176 self.graphFiles = {} # context uri : GraphFile
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
177
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
178 self.notifier = INotify()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
179 self.notifier.startReading()
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
180
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
181 self.findAndLoadFiles()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
182
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
183 def findAndLoadFiles(self):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
184 self.initialLoad = True
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
185 try:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
186 for topdir in self.topDirsToWatch:
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
187 for dirpath, dirnames, filenames in os.walk(topdir):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
188 for base in filenames:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
189 self.watchFile(os.path.join(dirpath, base))
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
190 self.notifier.watch(FilePath(dirpath), autoAdd=True,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
191 callbacks=[self.dirChange])
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
192 finally:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
193 self.initialLoad = False
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
194
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
195 def dirChange(self, watch, path, mask):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
196 if mask & IN_CREATE:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
197 self.watchFile(path.path)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
198
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
199 def watchFile(self, inFile):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
200 if not isinstance(inFile, FilePath):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
201 inFile = FilePath(inFile)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
202 if not inFile.isfile():
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
203 return
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
204 if inFile.splitext()[1] not in ['.n3']:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
205 return
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
206 ctx = self.uriFromFile(inFile)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
207 gf = GraphFile(self.notifier, inFile.path, ctx,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
208 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
209 self.graphFiles[ctx] = gf
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
210 gf.reread()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
211
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
212 def dirtyFiles(self, ctxs):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
213 """mark dirty the files that we watch in these contexts.
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
214
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
215 the ctx might not be a file that we already read; it might be
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
216 for a new file we have to create, or it might be for a
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
217 transient context that we're not going to save
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
218
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
219 if it's a ctx with no file, error
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
220 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
221 for ctx in ctxs:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
222 g = self.getSubgraph(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
223
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
224 if ctx not in self.graphFiles:
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
225 outFile = self.fileForUri(ctx)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
226 self.graphFiles[ctx] = GraphFile(self.notifier, outFile, ctx,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
227 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
228
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
229 self.graphFiles[ctx].dirty(g)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
230
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
231 def uriFromFile(self, filename):
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
232 if isinstance(filename, FilePath):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
233 filename = filename.path
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
234 assert filename.endswith('.n3'), filename
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
235 return URIRef(self.topUri + filename[:-len('.n3')])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
236
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
237 def fileForUri(self, ctx):
826
05aabe3d7b02 fix some minor issues with graph contexts
Drew Perttula <drewp@bigasterisk.com>
parents: 821
diff changeset
238 assert isinstance(ctx, URIRef), ctx
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
239 if not ctx.startswith(self.topUri):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
240 raise ValueError("don't know what filename to use for %s" % ctx)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
241 return ctx[len(self.topUri):] + ".n3"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
242
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
243
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
244 class Db(object):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
245 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
246 the master graph, all the connected clients, all the files we're watching
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
247 """
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
248 def __init__(self, topDirsToWatch):
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
249
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
250 self.clients = []
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
251 self.graph = ConjunctiveGraph()
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
252
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
253 self.watchedFiles = WatchedFiles(topDirsToWatch,
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
254 self.patch, self.getSubgraph)
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
255
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
256 self.summarizeToLog()
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
257
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
258 def patch(self, p, dueToFileChange=False):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
259 """
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
260 apply this patch to the master graph then notify everyone about it
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
261
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
262 dueToFileChange if this is a patch describing an edit we read
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
263 *from* the file (such that we shouldn't write it back to the file)
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
264
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
265 if p has a senderUpdateUri attribute, we won't send this patch
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
266 back to the sender with that updateUri
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
267 """
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
268 ctx = p.getContext()
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
269 log.info("patching graph %s -%d +%d" % (
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
270 ctx, len(p.delQuads), len(p.addQuads)))
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
271
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
272 patchQuads(self.graph, p.delQuads, p.addQuads, perfect=True)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
273 senderUpdateUri = getattr(p, 'senderUpdateUri', None)
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
274
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
275 for c in self.clients:
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
276 if c.updateUri == senderUpdateUri:
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
277 # this client has self-applied the patch already
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
278 continue
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
279 d = c.sendPatch(p)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
280 d.addErrback(self.clientErrored, c)
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
281 if not dueToFileChange:
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
282 self.watchedFiles.dirtyFiles([ctx])
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
283 sendToLiveClients(asJson=p.jsonRepr)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
284
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
285 def clientErrored(self, err, c):
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
286 err.trap(twisted.internet.error.ConnectError)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
287 log.info("connection error- dropping client %r" % c)
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
288 self.clients.remove(c)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
289 self.sendClientsToAllLivePages()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
290
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
291 def summarizeToLog(self):
798
5c158d37f1ce autoretry websocket. fix rdflib quad patching. only rerun handlers that asked for the affected subj-preds.
drewp@bigasterisk.com
parents: 797
diff changeset
292 log.info("contexts in graph (%s total stmts):" % len(self.graph))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
293 for c in self.graph.contexts():
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
294 log.info(" %s: %s statements" %
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
295 (c.identifier, len(self.getSubgraph(c.identifier))))
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
296
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
297 def getSubgraph(self, uri):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
298 """
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
299 this is meant to return a live view of the given subgraph, but
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
300 if i'm still working around an rdflib bug, it might return a
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
301 copy
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
302
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
303 and it's returning triples, but I think quads would be better
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
304 """
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
305 # this is returning an empty Graph :(
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
306 #return self.graph.get_context(uri)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
307
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
308 g = Graph()
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
309 for s in self.graph.triples(ALLSTMTS, uri):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
310 g.add(s)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
311 return g
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
312
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
313 def addClient(self, updateUri, label):
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
314 [self.clients.remove(c)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
315 for c in self.clients if c.updateUri == updateUri]
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
316
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
317 log.info("new client %s at %s" % (label, updateUri))
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
318 self.clients.append(Client(updateUri, label, self))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
319 self.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
320
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
321 def sendClientsToAllLivePages(self):
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
322 sendToLiveClients({"clients":[
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
323 dict(updateUri=c.updateUri, label=c.label)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
324 for c in self.clients]})
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
325
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
326 class GraphResource(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
327 def get(self):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
328 self.write(self.settings.db.graph.serialize(format='n3'))
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
329
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
330 class Patches(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
331 def __init__(self, *args, **kw):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
332 cyclone.web.RequestHandler.__init__(self, *args, **kw)
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
333 p = syncedgraph.makePatchEndpointPutMethod(self.settings.db.patch)
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
334 self.put = lambda: p(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
335
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
336 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
337 pass
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
338
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
339 class GraphClients(PrettyErrorHandler, cyclone.web.RequestHandler):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
340 def get(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
341 pass
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
342
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
343 def post(self):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
344 upd = self.get_argument("clientUpdate")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
345 try:
797
904913de4599 deletes are now quads. refactor files. named clients. auto client port
drewp@bigasterisk.com
parents: 796
diff changeset
346 self.settings.db.addClient(upd, self.get_argument("label"))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
347 except:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
348 import traceback
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
349 traceback.print_exc()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
350 raise
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
351
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
352 liveClients = set()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
353 def sendToLiveClients(d=None, asJson=None):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
354 j = asJson or json.dumps(d)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
355 for c in liveClients:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
356 c.sendMessage(j)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
357
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
358 class Live(cyclone.websocket.WebSocketHandler):
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
359
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
360 def connectionMade(self, *args, **kwargs):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
361 log.info("websocket opened")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
362 liveClients.add(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
363 self.settings.db.sendClientsToAllLivePages()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
364
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
365 def connectionLost(self, reason):
811
b19cd005a491 just comments
drewp@bigasterisk.com
parents: 808
diff changeset
366 log.info("websocket closed")
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
367 liveClients.remove(self)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
368
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
369 def messageReceived(self, message):
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
370 log.info("got message %s" % message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
371 self.sendMessage(message)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
372
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
373 class NoExts(cyclone.web.StaticFileHandler):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
374 # .xhtml pages can be get() without .xhtml on them
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
375 def get(self, path, *args, **kw):
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
376 if path and '.' not in path:
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
377 path = path + ".xhtml"
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
378 cyclone.web.StaticFileHandler.get(self, path, *args, **kw)
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
379
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
380 if __name__ == "__main__":
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
381 logging.basicConfig()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
382 log = logging.getLogger()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
383
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
384 parser = optparse.OptionParser()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
385 parser.add_option('--show',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
386 help='show URI, like http://light9.bigasterisk.com/show/dance2008',
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
387 default=showconfig.showUri())
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
388 parser.add_option("-v", "--verbose", action="store_true",
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
389 help="logging.DEBUG")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
390 (options, args) = parser.parse_args()
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
391
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
392 log.setLevel(logging.DEBUG if options.verbose else logging.INFO)
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
393
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
394 if not options.show:
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
395 raise ValueError("missing --show http://...")
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
396
829
e53e78db7b17 refactor file watching. notice new files and dirs.
Drew Perttula <drewp@bigasterisk.com>
parents: 826
diff changeset
397 db = Db(topDirsToWatch=['show/dance2013'])
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
398
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
399 from twisted.python import log as twlog
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
400 twlog.startLogging(sys.stdout)
821
295b867fd810 just whitespace (hopefully)
drewp@bigasterisk.com
parents: 815
diff changeset
401
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
402 port = 8051
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
403 reactor.listenTCP(port, cyclone.web.Application(handlers=[
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
404 (r'/live', Live),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
405 (r'/graph', GraphResource),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
406 (r'/patches', Patches),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
407 (r'/graphClients', GraphClients),
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
408
814
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
409 (r'/(.*)', NoExts,
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
410 {"path" : "light9/rdfdb/web",
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
411 "default_filename" : "index.xhtml"}),
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
412
1ae8e6b287e3 improvements to file watching. outline of how resync will work
drewp@bigasterisk.com
parents: 811
diff changeset
413 ], debug=True, db=db))
796
37d05bd17b10 rdfdb first pass
drewp@bigasterisk.com
parents:
diff changeset
414 log.info("serving on %s" % port)
808
a631e075a5bf KC big rewrites, now multiple KC instances can sync with rdfdb
drewp@bigasterisk.com
parents: 806
diff changeset
415 prof.run(reactor.run, profile=False)