Generative UI Design System

An Evaluation Framework for Next-Generation Generative User Interface

Problem Statement

Traditional UI design systems rely heavily on manual, static workflows, making it difficult to adapt to diverse and dynamic user needs. Recent advances in generative AI have enabled personalized and dynamic UI generation, but evaluating the quality of generated interfaces remains an open challenge. Existing evaluation methods are largely ad hoc, subjective, or limited to isolated usability metrics, and they fail to capture whether generative UI systems are aligned with human intent, context, and interaction goals. As generative UI systems become more autonomous, there is a growing need for a systematic, scalable evaluation framework that integrates usability, system performance, and human alignment.

Methods

We design an evaluation framework for generative UI systems that integrates vision-language models and user-centered evaluation. The framework incorporates vision-based localization techniques to ground UI elements spatially and structurally, enabling automated assessment of layout fidelity and interaction affordances.

In parallel, we develop LLM-based multi-dimensional evaluation metrics that assess generated UIs across usability, semantic alignment, and system-level consistency. Human-in-the-loop user studies are used to calibrate and validate automated metrics, forming a closed-loop evaluation process that supports iterative improvement of generative UI systems.

My Role

• I advanced the generative UI pipeline by introducing vision-based techniques, such as localizers, to improve spatial grounding and structural understanding of generated interfaces.
• I designed and conducted user studies based on a generative UI demo system, and iteratively refined the metrics and evaluation criteria of a Generative UI Evaluator using user study results, with a focus on improving human alignment.

Outputs

This is an ongoing research project with Samsung Design Office.