video: video.py annotate

annotate video.py @ 49:1bd17c2e5517 default tip

video.py must sign video urls for serve-files.js to serve them

author	drewp@bigasterisk.com
date	Fri, 06 Dec 2024 17:13:51 -0800
parents	882d0bb0f801
children

rev	line source
17 071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	1 import asyncio
071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	2 import json
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	3 import logging
46 882d0bb0f801 clean up; quiet logs; factor out PATH_PREFIX drewp@bigasterisk.com parents: 44 diff changeset	4 from functools import partial
882d0bb0f801 clean up; quiet logs; factor out PATH_PREFIX drewp@bigasterisk.com parents: 44 diff changeset	5 from urllib.parse import unquote
5 75b54be050bc show one subdir of archive at once, with folders and parents-breadcrumbs drewp@bigasterisk.com parents: 4 diff changeset	6
46 882d0bb0f801 clean up; quiet logs; factor out PATH_PREFIX drewp@bigasterisk.com parents: 44 diff changeset	7 import pymongo.database
28 1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	8 import uvicorn
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	9 from prometheus_client import Gauge
28 1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	10 from sse_starlette.sse import EventSourceResponse
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	11 from starlette.applications import Starlette
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	12 from starlette.requests import Request
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	13 from starlette.responses import HTMLResponse, JSONResponse, Response
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	14 from starlette.routing import Route
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	15 from starlette_exporter import PrometheusMiddleware, handle_metrics
5 75b54be050bc show one subdir of archive at once, with folders and parents-breadcrumbs drewp@bigasterisk.com parents: 4 diff changeset	16
28 1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	17 import dl_queue
49 1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	18 from signature_gen import makePlaybackSig
46 882d0bb0f801 clean up; quiet logs; factor out PATH_PREFIX drewp@bigasterisk.com parents: 44 diff changeset	19 import thumbnail
882d0bb0f801 clean up; quiet logs; factor out PATH_PREFIX drewp@bigasterisk.com parents: 44 diff changeset	20 from mongo_required import open_mongo_or_die
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	21 from video_file_store import VideoFileStore
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	22 from video_ingest import VideoIngest
18 1b388ee5dd09 reformat drewp@bigasterisk.com parents: 17 diff changeset	23
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	24 logging.basicConfig(level=logging.DEBUG)
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	25 log = logging.getLogger()
17 071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	26 logging.getLogger('sse_starlette').setLevel(logging.WARNING)
32 d83dbda8bccf quieter log drewp@bigasterisk.com parents: 28 diff changeset	27 logging.getLogger('pymongo').setLevel(logging.INFO)
0 3f2da406c788 start drewp@bigasterisk.com parents: diff changeset	28
3f2da406c788 start drewp@bigasterisk.com parents: diff changeset	29
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	30 def root(req):
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	31 return HTMLResponse("api")
0 3f2da406c788 start drewp@bigasterisk.com parents: diff changeset	32
3f2da406c788 start drewp@bigasterisk.com parents: diff changeset	33
6 ccfea3625cf6 render thumbs and display them (no video player at all atm) drewp@bigasterisk.com parents: 5 diff changeset	34 async def videos(req: Request) -> JSONResponse:
44 239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	35 # either like /dir1/dir2/ or /dir1/dir2/vid1
38 0aea9e55899b hack in path router - dirs kind of work; putting a video in the path doesn't drewp@bigasterisk.com parents: 37 diff changeset	36 subdir = req.query_params.get('subdir', '/')
49 1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	37 user = req.headers.get('x-pomerium-email', '')
1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	38 if not user:
1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	39 raise ValueError('must be logged in')
1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	40 log.info(f'videos for {user!r}')
44 239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	41
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	42 subdir = unquote(subdir)
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	43 webDirRelPath = subdir.rsplit('/', 1)[0] + '/'
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	44 autoplayRelPath = subdir if not subdir.endswith('/') else None
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	45
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	46 log.info(f'loading {webDirRelPath=} {autoplayRelPath=}')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	47
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	48 assert webDirRelPath.startswith('/')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	49 assert webDirRelPath.endswith('/')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	50
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	51 resp: dict[str, object] = {
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	52 'webDirRelPath': webDirRelPath,
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	53 'dirLabel': webDirRelPath.strip('/').split('/')[-1], #todo
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	54 }
39 b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	55
44 239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	56 d = webDirRelPath[1:-1] or '.'
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	57 log.info(f'loading {d=}')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	58 vfInDir = list(store.findInDir(d))
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	59 resp["videos"] = [{
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	60 'webRelPath': '/' + vf.webRelPath,
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	61 'webDataPath': '/' + vf.webDataPath,
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	62 'label': vf.label,
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	63 } for vf in vfInDir]
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	64
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	65 if autoplayRelPath:
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	66 for vf in vfInDir:
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	67 if '/' + vf.webRelPath == autoplayRelPath:
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	68 resp['autoplay'] = {
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	69 'webRelPath': '/' + vf.webRelPath,
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	70 'webDataPath': '/' + vf.webDataPath,
49 1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	71 'label': vf.label,
1bd17c2e5517 video.py must sign video urls for serve-files.js to serve them drewp@bigasterisk.com parents: 46 diff changeset	72 'sig': makePlaybackSig(user, '/' + vf.webDataPath),
44 239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	73 }
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	74 break
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	75 else:
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	76 raise ValueError(f'{autoplayRelPath=} not in dir')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	77
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	78 log.info(f'{subdir=}')
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	79 resp['subdirs'] = []
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	80 for s in store.findSubdirs(subdir.strip('/') or '/'):
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	81 resp['subdirs'].append({
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	82 'label': s['label'],
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	83 'path': '/' + s['path'] + '/',
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	84 })
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	85
239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	86 return JSONResponse(resp)
4 c8a41359505c server provides video listing from disk drewp@bigasterisk.com parents: 3 diff changeset	87
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	88
20 cf6842952e13 serve api/folderTree drewp@bigasterisk.com parents: 18 diff changeset	89 def folderTree(req: Request) -> JSONResponse:
cf6842952e13 serve api/folderTree drewp@bigasterisk.com parents: 18 diff changeset	90 return JSONResponse(store.folderTree())
cf6842952e13 serve api/folderTree drewp@bigasterisk.com parents: 18 diff changeset	91
cf6842952e13 serve api/folderTree drewp@bigasterisk.com parents: 18 diff changeset	92
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	93 async def ingestVideoUrl(req: Request) -> Response:
23 9e56e86a6814 server supports downloading into a given folder drewp@bigasterisk.com parents: 20 diff changeset	94 folder = req.query_params['folder']
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	95 url = await req.body()
23 9e56e86a6814 server supports downloading into a given folder drewp@bigasterisk.com parents: 20 diff changeset	96 await svc.ingestUrl(url.decode('utf8'), folder)
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	97 return Response(status_code=202)
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	98
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	99
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	100 async def ingestVideoUpload(req: Request) -> Response:
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	101 name = req.query_params['name']
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	102 await svc.addContent(name, req.body())
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	103 return Response(status_code=200)
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	104
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	105
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	106 async def ingestQueue(req: Request) -> EventSourceResponse:
18 1b388ee5dd09 reformat drewp@bigasterisk.com parents: 17 diff changeset	107
17 071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	108 async def g():
071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	109 async for ev in svc.events():
071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	110 yield json.dumps(ev)
18 1b388ee5dd09 reformat drewp@bigasterisk.com parents: 17 diff changeset	111
17 071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	112 return EventSourceResponse(g())
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	113
39 b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	114
37 7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	115 def getDiskPath(fs, webRelPath):
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	116 doc = fs.find_one({'webRelPath': webRelPath})
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	117 if doc is None:
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	118 raise ValueError
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	119 return doc['diskPath']
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	120
39 b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	121
b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	122 async def getThumbnail(db: pymongo.database.Database,
b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	123 req: Request) -> Response:
44 239a83d46d48 make the server return urls with the (new) correct slashes drewp@bigasterisk.com parents: 39 diff changeset	124 webRelPath = req.query_params['webRelPath'].lstrip('/')
37 7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	125 fs = db.get_collection('fs')
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	126 diskPath = getDiskPath(fs, webRelPath)
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	127 th = db.get_collection('thumb')
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	128 async with asyncio.timeout(10):
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	129 data = await thumbnail.getThumbnailData(th, diskPath)
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	130 return Response(content=data, media_type='image/jpeg')
7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	131
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	132
36 ed16fdbb3996 rewrite WIP. scan fs separately; store in db. thumbs are broken for now drewp@bigasterisk.com parents: 32 diff changeset	133 db = open_mongo_or_die().get_database('video')
ed16fdbb3996 rewrite WIP. scan fs separately; store in db. thumbs are broken for now drewp@bigasterisk.com parents: 32 diff changeset	134 store = VideoFileStore(db.get_collection('fs'))
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	135 svc = VideoIngest(store)
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	136
18 1b388ee5dd09 reformat drewp@bigasterisk.com parents: 17 diff changeset	137
4 c8a41359505c server provides video listing from disk drewp@bigasterisk.com parents: 3 diff changeset	138 def main():
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	139
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	140 app = Starlette(
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	141 debug=True,
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	142 routes=[
4 c8a41359505c server provides video listing from disk drewp@bigasterisk.com parents: 3 diff changeset	143 Route('/video/api/', root),
c8a41359505c server provides video listing from disk drewp@bigasterisk.com parents: 3 diff changeset	144 Route('/video/api/videos', videos),
20 cf6842952e13 serve api/folderTree drewp@bigasterisk.com parents: 18 diff changeset	145 Route('/video/api/folderTree', folderTree),
15 53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	146 Route('/video/api/ingest/videoUpload',
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	147 ingestVideoUpload,
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	148 methods=['POST']),
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	149 Route('/video/api/ingest/videoUrl',
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	150 ingestVideoUrl,
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	151 methods=['POST']),
53d99454f394 support dropping url or file drewp@bigasterisk.com parents: 6 diff changeset	152 Route('/video/api/ingest/queue', ingestQueue),
37 7cacfae58430 thumbnails rewrite - store in db; don't use YT-provided pics for now drewp@bigasterisk.com parents: 36 diff changeset	153 Route('/video/api/thumbnail', partial(getThumbnail, db)),
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	154 ],
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	155 )
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	156
ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	157 app.add_middleware(PrometheusMiddleware, app_name='video_api')
25 bf860a23d0c1 deployment and metrics drewp@bigasterisk.com parents: 23 diff changeset	158 app.add_route("/video/api/metrics", handle_metrics)
27 27a754f8e8f8 deployment fixes drewp@bigasterisk.com parents: 25 diff changeset	159 app.add_route("/metrics", handle_metrics)
18 1b388ee5dd09 reformat drewp@bigasterisk.com parents: 17 diff changeset	160
17 071943adf000 dnd a file or a url which we'll queue and fetch drewp@bigasterisk.com parents: 15 diff changeset	161 app.state.processTask = asyncio.create_task(dl_queue.process())
4 c8a41359505c server provides video listing from disk drewp@bigasterisk.com parents: 3 diff changeset	162 return app
3 ee55ed10faec start py service deployment drewp@bigasterisk.com parents: 0 diff changeset	163
39 b5b29f6ef5cb cleanup and refactor drewp@bigasterisk.com parents: 38 diff changeset	164
28 1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	165 uvicorn.run(main,
1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	166 host="0.0.0.0",
1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	167 port=8004,
1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	168 log_level=logging.INFO,
1e058bea3ac2 crash better when mongo is unreachable drewp@bigasterisk.com parents: 27 diff changeset	169 factory=True)

Mercurial > code > home > repos > video

annotate video.py @ 49:1bd17c2e5517 default tip