Evaluating Design Video Generation: Metrics for Compositional Fidelity
Adrienne Deganutti, Dingning Cao, Jaejung Seol, Elad Hirsch, Purvanshi Mehta
Automated evaluation of generative video models for design animation — think motion graphics, UI animations, and marketing assets — across four dimensions: layout fidelity (does structure hold?), motion correctness (right type, direction, speed, timing?), temporal quality (stability in non-animated regions?), and content fidelity (does content survive generation?). The claim is that this fully automated suite eliminates the need for human raters and gives the field a shared benchmark baseline. No quantitative comparisons against prior metrics are presented; this is a framework paper, not a performance breakthrough.
No production traction yet. Zero citations and all GitHub references are automated arXiv digest tools, not implementers. The framework appears to have originated inside a design or product org (likely Canva or similar, given the author affiliations implied by the domain focus), but there's no public code or dataset released alongside it.
Generative video models are increasingly used in design animation tasks, yet no standardized evaluation framework exists for this domain. Unlike natural video generation, design animation imposes structured constraints: specific components shall animate with prescribed motion types, directions, speed and timing, while non-animated regions must remain stable and layout structure must be preserved. This paper provides a fully automated evaluation framework organized across four dimensions: layout fidelity, motion correctness, temporal quality, and content fidelity. This eliminates the reliance on subjective human evaluation and establishes a common basis for benchmarking progress in the field.