Mercurial > code > home > repos > homeauto

annotate service/reasoning/rdflibtrig.py @ 722:a93fbf0d0daa

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
dep updates; graph url renames; and other build updates Ignore-this: 4603ef3d8db650a13e543dad8580ade8
author drewp@bigasterisk.com
date Wed, 05 Feb 2020 00:23:06 -0800
parents e5c27d2f11ab
children 3c18b4b3b72c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
1 import time, logging
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 129
diff changeset
2 from rdflib import ConjunctiveGraph
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
3 from rdflib.parser import StringInputSource
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
4 import treq
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
5 from twisted.internet.defer import inlineCallbacks, returnValue
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
6 log = logging.getLogger('fetch')
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
7
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
8 from private_ipv6_addresses import ipv6Addresses
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
9
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
10 @inlineCallbacks
129
745eff67ad40 reasoning actions: generalize them a bit but then add a bunch of special cases for mpd for now
drewp@bigasterisk.com
parents: 118
diff changeset
11 def addTrig(graph, url, timeout=2):
114
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents:
diff changeset
12 t1 = time.time()
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
13 # workaround for some reason my ipv6 names don't resolve
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
14 for name, addr in ipv6Addresses.iteritems():
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
15 url = url.replace('/' + name + ':', '/[' + addr + ']:')
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
16 log.debug(' fetching %r', url)
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
17 response = yield treq.get(url, headers={'accept': ['application/trig']}, timeout=timeout)
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
18 if response.code != 200:
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
19 raise ValueError("status %s from %s" % (response.code, url))
240
0c306e76d8c5 ipv6 fetch support. refactor Actions to new class and file
drewp@bigasterisk.com
parents: 235
diff changeset
20 g = ConjunctiveGraph()
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
21 g.parse(StringInputSource((yield response.content())), format='trig')
114
4cd065b97fa1 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp@bigasterisk.com
parents:
diff changeset
22 fetchTime = time.time() - t1
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
23 log.debug(' %r done in %.04f sec', url, fetchTime)
235
5ad229334a88 rdflib can parse trig now
drewp@bigasterisk.com
parents: 129
diff changeset
24 graph.addN(g.quads())
249
e5c27d2f11ab fetch all source graphs in parallel
drewp@bigasterisk.com
parents: 240
diff changeset
25 returnValue(fetchTime)