Mercurial > code > home > repos > video

annotate video_ingest.py @ 28:1e058bea3ac2

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
crash better when mongo is unreachable
author drewp@bigasterisk.com
date Sat, 23 Mar 2024 16:11:10 -0700
parents 9e56e86a6814
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
17
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
1 import asyncio
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
2 from dataclasses import dataclass
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
3 from typing import Any, Coroutine
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
4
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
5 import dl_queue
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
6 from video_file_store import VideoFileStore
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
7
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
8
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
9 @dataclass
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
10 class VideoIngest:
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
11 store: VideoFileStore
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
12
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
13 async def addContent(self, name: str, body: Coroutine[Any, Any, bytes]):
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
14 await self.store.save(name, iter([await body]))
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
15
23
9e56e86a6814 server supports downloading into a given folder
drewp@bigasterisk.com
parents: 17
diff changeset
16 async def ingestUrl(self, url: str, folder:str):
9e56e86a6814 server supports downloading into a given folder
drewp@bigasterisk.com
parents: 17
diff changeset
17 outDir = self.store.top / folder.lstrip('/') #danger user input
9e56e86a6814 server supports downloading into a given folder
drewp@bigasterisk.com
parents: 17
diff changeset
18 dl_queue.queue.put({'url': url, 'outDir': str(outDir)})
17
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
19
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
20 async def events(self):
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
21 prev = None
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
22 while True:
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
23 p = dl_queue.pending()
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
24 if p != prev:
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
25 prev = p
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
26 yield p
071943adf000 dnd a file or a url which we'll queue and fetch
drewp@bigasterisk.com
parents:
diff changeset
27 await asyncio.sleep(1)