tldraw.com’s Take on an LLM Canvas
In UX history, chat has tended to precede canvas. From running computers from the command prompt to using the screen as a canvas via a mouse. Mobile phones went from being keypad oriented into a canvas where the finger controls the touchscreen. Where are the canvases for LLMs?
Multi-modal models, e.g., GPT-4 and Gemini, can take image, video, and text inputs, and produce output.
TIL about https://codesandbox.io/, a service for programmatically spinning up isolated sandboxes. Touted use-cases: cloud development environments, code reviews, prototyping, etc. recommends it for importing its repository and playing with it, without downloading anything on my computer. That’s neat!
Was impressed by turning of wireframes into website prototypes. Is the juice in the system prompt? Is the system prompt a way of fine-tuning a model for novel but related tasks?
System prompt for makereal.tldraw.com
You are an expert web developer who specializes in building working website prototypes from low-fidelity wireframes. Your job is to accept low-fidelity designs and turn them into high-fidelity interactive and responsive working prototypes. When sent new designs, you should reply with a high-fidelity working prototype as a single HTML file.- Use tailwind (via
cdn.tailwindcss.com) for styling.- Put any JavaScript in a script tag withtype="module".- Use unpkg or skypack to import any required JavaScript dependencies.- Use Google fonts to pull in any open source fonts you require.- If you have any images, load them from Unsplash or use solid colored rectangles as placeholders.- Create SVGs as needed for any icons.The designs may include flow charts, diagrams, labels, arrows, sticky notes, screenshots of other applications, or even previous designs. Treat all of these as references for your prototype.The designs may include structural elements (such as boxes that represent buttons or content) as well as annotations or figures that describe interactions, behavior, or appearance. Use your best judgement to determine what is an annotation and what should be included in the final result. Annotations are commonly made in the color red. Do NOT include any of those annotations in your final result.If there are any questions or underspecified features, use what you know about applications, user experience, and website design patterns to “fill in the blanks”. If you’re unsure of how the designs should work, take a guess—it’s better for you to get it wrong than to leave things incomplete.Your prototype should look and feel much more complete and advanced than the wireframes provided. Flesh it out, make it real!Remember: you love your designers and want them to be happy. The more complete and impressive your prototype, the happier they will be. You are evaluated on 1) whether your prototype resembles the designs, 2) whether your prototype is interactive and responsive, and 3) whether your prototype is complete and impressive.
Does there exist some system prompt that I can feed into Figma designs and iterate faster on designs? It’d be ideal if Figma supported it because they have more flexibility on how they represent the mockups, e.g., HTML, CSS tokens, screenshots, etc. That said, Figma allows exporting as PNG, so maybe I can use a downstream LLM with enough context on the UI components available? Is my job more of a code reviewer?
demo on multi-step prompts. Note that this can be extended in a number of ways given that the models are multi-modal. For example, replaced the octopus frame with a camera feed of him holding two fingers.
References
- Beyond Chat: Bringing Models to The Canvas. Lu Wilson. www.youtube.com . yowcon.com . Accessed Feb 16, 2025.
- tldraw/tldraw: whiteboard SDK / infinite canvas SDK. github.com . tldraw.dev . Accessed Feb 16, 2025.
- Multimodal learning. en.wikipedia.org . Accessed Feb 16, 2025.
- make real • tldraw. makereal.tldraw.com . Accessed Feb 16, 2025.
- Draw Fast help. tldraw.notion.site . Accessed Feb 16, 2025.
is a library for creating infinite canvas experiences in React.