Using LLMs to generate UX Wireframes
Experiments turning Text Requirements into User Interfaces
We have seen Large Language Models (LLMs) being used to transform ‘napkin sketches’ into wireframes and convert wireframes into working applications. However, we were unable to find any explorations of using LLMs to turn a list of UX requirements directly into a wireframe.
This post goes over some of our experiments and results with the hope that this inspires others to share their work, as well.
We approached the problem from two different directions – HTML/CSS and free-floating axis-aligned bounding boxes (AABB):
Beyond basic prompt engineering, we explored several representations per each approach (links to images of results below)
HTML/CSS
AABB
Requirements input to LLM:
- Player hands: Three distinct areas showing the cards held by each player (face-up for the active player, face-down for others)
- Discard pile: A central area showing the top card of the discard pile
- Draw pile: A face-down stack of remaining cards
- Turn indicator: A visual cue showing which player’s turn it is
- Card count: A display of how many cards each player has left
- Action area: A space where special actions (like drawing cards or skipping turns) are animated
- Message area: A space for game messages (e.g., “Player 2 must skip their turn”)
- Score display: If playing multiple rounds, an area showing each player’s current score
- Settings/menu button: For accessing game options or returning to the main menu
- Player avatars or names: To easily identify each player
Results:
CSS Flow Layout
CSS Flexbox Layout
CSS Grid Layout
Bootstrap
Bulma
Tailwind
JSON
YAML
XML
We also ran our experiments with several LLMs:
- Claude 3.5 Sonnet
- GPT 4o
- OpenAI o1-preview
- Llama 3.1 405b
- Mistral large
We observed that Claude 3.5 Sonnet was the winner in several key areas: accurately translating requirements, addressing the maximum number of requirements, and generating high-quality code with appropriate utility classes and UI components. The wireframes generated had better visual hierarchy, semantic consistency, and well-proportioned elements compared to outputs from other models.
Requirements input to LLM:
- Order Display: A section of the screen that shows the current order to be fulfilled, consisting of the required rice, fish type, and side dish.
- Timer: A progress bar timer (starting from 15 seconds) to complete the current order.
- Ingredient Cards: A collection of 16 cards representing the available ingredients (rice, fish types, and side dishes) that the player can select from to fulfill the order.
- Selection Area: A designated area where the player can drag and drop or click to select the cards representing their chosen ingredients for the current order.
- Score/Money Display: A display showing the player’s current score or money earned, based on correct orders and penalties.
- Level Indicator: An indicator showing the player’s current level, which affects the reward and penalty values.
- Streak Counter: A display tracking the player’s current streak of consecutive correct orders.
- Power-up Icons: Icons or buttons representing the available power-ups (Extra Time, Double Reward, Reroll Order) and their remaining uses.
- Power-up Activation Buttons: Buttons or controls to activate the respective power-ups when needed.
- Game Status Display: A display area showing the game’s current status, such as “Order Correct,” “Order Incorrect,” “Game Won,” or “Game Over.”
- Pause/Resume Button: A button to pause or resume the game, if applicable.
- Restart/Quit Buttons: Buttons to restart the game or quit and return to the main menu.
- Customer Avatar: Avatar for customer placing the order.
Results:
Claude 3.5 Sonnet
GPT 4o
OpenAI o1-preview
Meta Llama 3.1 405B
Mistral Large
Prompt Engineering
- Chain of Thought (CoT) prompting
- Using Auto-CoT: We included phrases like ‘think step by step’ in our prompt. This approach proposed by Zhang et al. (2022) eliminates the need for manual effort in generating reasoning chains.
- Few-Shot CoT: While Auto-CoT is powerful, we found that combining it with few-shot examples led to even better results. The expanded requirements provide a detailed spatial layout and hierarchical structure, serving as a mental map for the LLM. This approach guides the LLM to think in terms of specific visual organization and spatial relationships, leading to more coherent and practical UI designs.
For instance, we included an example like this:
```
<example>
<example_requirements>
- 3 cards open in the middle
- 5 cards for each player
- 2 players (A player can only see their cards)
...
</example_requirements>
<expanded_requirements>
[Center of the table]
- 3 open cards (horizontally arranged):
Card 1: [Word 1]
Card 2: [Word 2]
Card 3: [Word 3]
...
</expanded_requirements>
</example>
```
- Structured prompting: Claude is familiar with processing structured inputs. Using XML-like tags such as <example>, <styling_rules>, and <output> helps compartmentalize information, providing clear context and structure.
Conclusion
Based on our experiments, as illustrated in the figure (link), we found that HTML/CSS typically produced clean and balanced wireframes. These resulted in consistent card sizes and alignment. Among the frameworks tested, Tailwind CSS yielded the best results. We observed that the wireframes exhibited several desirable qualities: balance, well-defined action areas, appropriately sized elements, and consistent spacing and provided a clearer representation of the game state while appearing visually appealing and user-friendly. In contrast, the AABB approach struggled with clutter and inconsistent card sizes.
Final prompt used
You are an expert UI UX designer. Your task is to design high fidelity wireframes for card games given certain requirements of the gameplay screen elements. Think step-by-step. The requirements will be given as bullet points in <given_requirements> tag and you must expand them (you can refer to the examples below in <example> tag). Then you must think about placing the elements according to the design principles given in <design_principles> tag . Once you have the design, you must convert the design a single HTML file by adhering to the rules in <styling_rules> and design principles.
<example>
<example_requirements>
3 cards open in the middle
5 cards for each player
2 players (A player can only see their cards)
Deck of cards next to 3 open cards
A draw button (the deck of cards can also act as a draw button)
Score for each player
Card backs have no information
Card front has a designated word
Names of each player
Winning streak of each player (ex. 3rd win)
Quit button (End game)
</example_requirements>
<expanded_requirements> [Center of the table]
3 open cards (horizontally arranged): Card 1: [Word 1] Card 2: [Word 2] Card 3: [Word 3]
Deck of cards (face down, acts as draw button) Located to the right of the 3 open cards
[Bottom of the screen - Player A's area]
Name: [Player A's name]
Score: [Player A's score]
Winning streak: [X]th win
5 cards (horizontally arranged, face up): Card 1: [Word] Card 2: [Word] Card 3: [Word] Card 4: [Word] Card 5: [Word]
[Top of the screen - Player B's area]
Name: [Player B's name]
Score: [Player B's score]
Winning streak: [Y]th win
5 cards (horizontally arranged, face down): Card 1: [Hidden] Card 2: [Hidden] Card 3: [Hidden] Card 4: [Hidden] Card 5: [Hidden]
[Top right corner]
Quit button (to end the game)
</expanded_requirements>
<example_requirements>
Up to 5 cards for up to 4 players
One of those sets is ‘face up’ (i can see the value of the cards)
The other sets (the other players) are ‘face down’
Card backs have no information
Card front have numeric value and a color
In addition to the player hands, we have a ‘spread’ of up to 5 face up cards
Scores for all players
Names of each player
Avatar for each player (just face)
</example_requirements>
<expanded_requirements>
[Center of the table]
'Spread' of up to 5 face-up cards (horizontally arranged): Card 1: [Value, Color] Card 2: [Value, Color] Card 3: [Value, Color] Card 4: [Value, Color] Card 5: [Value, Color]
[Bottom of the screen - Current Player's area]
Name: [Player 1's name]
Avatar: [Player 1's face]
Score: [Player 1's score]
5 cards (horizontally arranged, face up): Card 1: [Value, Color] Card 2: [Value, Color] Card 3: [Value, Color] Card 4: [Value, Color] Card 5: [Value, Color]
[Other Players' areas]
For each of the 3 other players:
Name: [Player's name]
Avatar: [Player's face]
Score: [Player's score]
5 cards (horizontally or vertically arranged, face down): Card 1: [Hidden] Card 2: [Hidden] Card 3: [Hidden] Card 4: [Hidden] Card 5: [Hidden]
</expanded_requirements>
<styling_rules>
Use Tailwind CSS Styling
Create an outer container that follows an aspect ratio of 16:9.
Make the outer container centered but fixed.
All elements must be represented as rectangular AABB boxes inside this container (layout).
All elements must be labeled properly indicating what they represent
Ensure that the container does not allow its content to overflow by using overflow-hidden.
No elements should overlap each other
Ensure that all buttons are placed with adequate spacing from other elements to prevent overlap.
</styling_rules>
<design_principles>
Clarity and Readability:
Use consistent and clear labeling for all elements
Implement a clear visual hierarchy to distinguish between primary and secondary information
Choose appropriate font sizes and styles for different types of information
Group related information together for easier comprehension
Layout and Distribution:
Apply the rule of thirds or golden ratio for a balanced composition
Create a focal point to guide the user's attention
Use alignment and grid systems to organize elements coherently
Ensure consistent spacing between elements
Consider the natural reading patterns (e.g., F-pattern, Z-pattern) when arranging elements
Make the design visually appealing
Effective Use of Whitespace:
Use whitespace to create visual separation between distinct areas
Employ whitespace to improve readability and reduce cognitive load
Utilize whitespace to guide the user's eye through the interface
Balance whitespace with content density to avoid a sparse or cluttered look
Use whitespace consistently to maintain rhythm and flow in the design
Contextual Placement:
Arrange elements based on their frequency of use and importance
Use spatial relationships to imply connections between elements
Ensure the layout reflects the natural flow of the game or task
Proportions and Sizing:
Use a consistent scale for similar elements throughout the design
Size elements based on their importance and frequency of use
Maintain appropriate aspect ratios for images and cards
Ensure touch targets are appropriately sized for easy interaction
Consistency and Patterns:
Maintain consistent styling for similar elements throughout the design
Use established design patterns that users are familiar with
Create a cohesive visual language across all components
Ensure consistency in the treatment of icons, buttons, and other UI elements
Scalability and Flexibility:
Design components that can easily adapt to different content lengths
Create layouts that can accommodate varying numbers of elements (e.g., different numbers of players or cards)
</design_principles>
<output>
Generate single HTML file with complete code (all repeatitive divs should be included in the code). Do not add comments to repeat anything. Do not include any Javascript code. After designing the layout, perform a final check to ensure no elements are overlapping, especially buttons with player information or cards. If the standard layout causes overlap issues, provide alternative arrangement that maintain all required elements without overlap. Also make sure that the outer layout follows 16:9 ratio.
</output>
Haven Studios is a game development studio with a goal to create systemic and evolving worlds focused on freedom, thrill and playfulness that will keep players entertained and engaged for years. Haven joined Sony Interactive Entertainment, PlayStation Studios in 2022 as the first Sony game development team based in Canada. Rucha Shende is an ML Researcher at Haven Studios.