Generative prior-driven discovery and editing of video content

Video content discovery and editing are two core tasks in computer vision. Discovery localises an object and its key points in each frame and tracks the object across frames, whereas editing modifies the discovered content according to the user’s intent. Deep learning has advanced both tasks and enabled their application in visual surveillance, content creation, and film production. However, current methods typically rely on large-scale task-specific annotations, such as key point labels, bounding boxes, and paired source and edited videos, which are expensive to collect and limit generalisation to unseen categories and edits. Meanwhile, large-scale pre-trained text-to-image diffusion models have acquired rich knowledge of visual semantics and structure from web-scale data, referred to as the generative prior. Exploiting this prior raises two challenges. The prior is implicitly encoded in a model trained for generation, and is therefore hard to elicit for discovery without task-specific supervision. In addition, the sampling process of a diffusion model does not keep the edited video consistent with the source video. This thesis addresses both challenges by developing methods that use prompting as the interface to the generative prior. First, we exploit the cross-attention of text-to-image diffusion models as a semantic bridge and optimise pseudo prompts to localize key points of unseen categories in each frame. Second, we show that a text-to-image diffusion model can serve as an unsupervised tracker, in which an initial prompt learner and an online prompt updater learn a prompt that activates the target across frames without any annotation. Third, we recast one-shot controllable video editing as visual prompting on an inpainting diffusion model, which eliminates DDIM inversion. To keep the edited frames consistent with the source frames, we further introduce content consistency sampling and temporal-content consistency sampling, which build on a projected-kernel Stein update and motion-adaptive affinity augmentation. Experiments on multiple benchmarks demonstrate that the proposed methods enable label-free content discovery and consistent, controllable editing of videos.

Speaker’s profile

Zhengbo Zhang is a PhD candidate at Singapore University of Technology and Design (SUTD). He received his BS and MS degrees from Wuhan University, China. His PhD research focuses on deep learning and computer vision.