Mercurial > code > home > repos > homeauto

annotate service/reasoning/rdflibtrig.py @ 1409:e78b8806ad05

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
actually set put payload this time. add treq dep Ignore-this: b6a05c294d9a5ee6052972cad4ecb5c5 darcs-hash:d41a16eb44db885ff391ddbe64352245b885a838
author drewp <drewp@bigasterisk.com>
date Tue, 23 Jul 2019 17:37:24 -0700
parents bbaf0576f653
children 3c18b4b3b72c
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
1 import time, logging
1040
2fa7daf06f34 rdflib can parse trig now
drewp <drewp@bigasterisk.com>
parents: 934
diff changeset
2 from rdflib import ConjunctiveGraph
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
3 from rdflib.parser import StringInputSource
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
4 import treq
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
5 from twisted.internet.defer import inlineCallbacks, returnValue
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
6 log = logging.getLogger('fetch')
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
7
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
8 from private_ipv6_addresses import ipv6Addresses
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
9
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
10 @inlineCallbacks
934
3bb18b7d21df reasoning actions: generalize them a bit but then add a bunch of special cases for mpd for now
drewp <drewp@bigasterisk.com>
parents: 923
diff changeset
11 def addTrig(graph, url, timeout=2):
919
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents:
diff changeset
12 t1 = time.time()
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
13 # workaround for some reason my ipv6 names don't resolve
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
14 for name, addr in ipv6Addresses.iteritems():
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
15 url = url.replace('/' + name + ':', '/[' + addr + ']:')
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
16 log.debug(' fetching %r', url)
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
17 response = yield treq.get(url, headers={'accept': ['application/trig']}, timeout=timeout)
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
18 if response.code != 200:
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
19 raise ValueError("status %s from %s" % (response.code, url))
1045
a328cc370b22 ipv6 fetch support. refactor Actions to new class and file
drewp <drewp@bigasterisk.com>
parents: 1040
diff changeset
20 g = ConjunctiveGraph()
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
21 g.parse(StringInputSource((yield response.content())), format='trig')
919
6ee2a90fc816 bugs in async http client. move trig helpers to rdflibtrig, which can work with rdflib 4
drewp <drewp@bigasterisk.com>
parents:
diff changeset
22 fetchTime = time.time() - t1
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
23 log.debug(' %r done in %.04f sec', url, fetchTime)
1040
2fa7daf06f34 rdflib can parse trig now
drewp <drewp@bigasterisk.com>
parents: 934
diff changeset
24 graph.addN(g.quads())
1054
bbaf0576f653 fetch all source graphs in parallel
drewp <drewp@bigasterisk.com>
parents: 1045
diff changeset
25 returnValue(fetchTime)