[Prototype] LLM Drag & Drop Website Builder (Spring 2024)
I built this over 4 weeks in February/March of 2024. It uses shadcn and tailwind for styling.
It involves some (IMO) incredibly novel interfaces and concepts for generating frontend code. The primary UX is the drag and drop interface, but the magic is in two places:
1) There’s a “magic wiring” function where the LLM is able to perceive how the app is meant to work from how the components are laid out and generate the frontend state management code to “make it real”. This leaves UI editing to the user instead of having to do lots of prompting, but the complex state management that people tend to hate the most is outsourced to the LLM.
Some people (like me) may not be the best designers and just want to spin up something that looks nice fast. So there’s also a “magic paint” button that lets the AI take your existing frontend that you quickly threw together with drag and drop, and make it beautiful by adjusting layout, colors, font sizes, etc.
2) the user doesn’t have to use annoying side panel controls to change the appearance of components, instead they can jump right into the component code and write tailwind to quickly make the changes they want. This makes the app more conducive to technical users than something like Bubble.
LLM drag and drop website builder
The code is open sourced here: https://github.com/EarlywormTeam/ui-builder.
Technical Details
There were several super fun parts of this project, and I consider this one of the coolest projects I’ve probably ever done.
Edit & Preview Modes
I wanted the user to be able to quickly switch from building the frontend to playing with it, which meant instantly switching from a drag and drop interface into a preview mode where the app functions like it would when deployed. A really cool effect of this is that it’s super easy to debug and fix visual bugs - you could hop on preview mode and setup the buggy state, then go back into edit mode with the state still there and make a fix.
But first, I had to design an interface that can be previewed and interacted with naturally as a website (i.e. button clicks, text inputs all work as expected), but that can also be edited via drag and drop (i.e. buttons don’t click and text fields aren’t editable, but things can be picked up and moved). To do this I built drag and drop wrapper classes for all components that controlled how touch events were dispatched: https://github.com/EarlywormTeam/ui-builder/blob/main/frontend/src/pages/app/components/DraggableComponent.tsx and https://github.com/EarlywormTeam/ui-builder/blob/main/frontend/src/pages/app/components/DroppableComponent.tsx.
LLM Generating Frontend State Management Code (Using Custom Configuration Language)
I wanted the LLM to do the hard part of frontend which is not easily doable from a pure drag and drop editor - wire up the frontend state. But I had to figure out how to get the LLM to generate the state management code. I settled on a pattern of using React Contexts to store state - and picked the lowest point on the frontend tree to position the context based on the components involved in the interaction. Then I had the LLM generate code using a custom syntax I defined for it, which I then translated into the necessary boilerplate to inject the context into the components and have them update the context on user interactions and refresh their UI when the context state changed.
There were 2 important pieces here:
1) The interaction with the LLM. The LLM takes in a json representation of the component hierarchy and determines what the missing functionality is (or for magic paint determines how to make this look better), then responds with our custom domain-specific language style of configs. Code & prompts for the AI that builds frontend state handling is here: https://github.com/EarlywormTeam/ui-builder/blob/main/backend/src/controllers/codeGen/magicWiring.ts, and code for the AI that improves the design and layout is here: https://github.com/EarlywormTeam/ui-builder/blob/main/backend/src/controllers/codeGen/magicPaint.ts.
2) How to update the components by inflating our “intermediate representation (ir)” into real components that can be dragged and dropped or previewed depending on which mode the user is in. You can see how we represent the components in our intermediate representation, basically the uninflated data structure here: https://github.com/EarlywormTeam/ui-builder/blob/main/frontend/src/redux/slice/canvasSlice.ts. The components are stores in a tree in react frontend state (yea this is going to get confusing because we have an app builder that has state and an app within that which in preview mode has its own state).
This is our base component - it’s the big space in the middle of the app if you watch the video above: https://github.com/EarlywormTeam/ui-builder/blob/main/frontend/src/pages/app/components/Canvas.tsx. You can see the root component in there is a DynamicElement. The DynamicElement just parses the config and dispatches to a child based on whether its a Provider, Generator (List), or regular Component. Providers manipulate state and we do runtime code evaluation to turn the function config into an actual reducer function. Lists are super funky because their state is dependent on the app’s runtime, I kind of forgot exactly how I made it work but the code is here if you want to grok it. And last you have the Component which is a super juicy 330 line thing that is creating drag & drop components on the fly from the config data.
One awesome thing is that this really works well, and the state is confined to sub-trees so the app isn’t overly refreshing on small dom updates which would be terrible since we’d be re-inflating the UI tree on the fly every time it updated if that were the case.
Editing Code Directly to Change Visual State
Finally I wanted a way for users to be able to edit small details like font size, coloring, etc. Basically all of the tailwind classes. But when I started building out a property panel it became clear very fast that this was just not the right approach. It felt like Bubble or Wix and trying to do a lot of small tinkering and edits where you need to move sliders or adjust a bunch of values manually is just super annoying.
So instead I devised a way to show each component’s React code in a small window where the developer can just hop in and edit it - super convenient for doing things like adding or removing tailwind classes.
The hard part here was that we’re not actually storing the React code anywhere. We just have a bunch of component configs. So I built a way to generate React code from the config for only the selected component, then also go from React code back to our configuration language. I monitor the code editor window to see when there’s new valid React code, and then instantly update the display.
It was almost magical to see your edit displayed so fast (you can see this at the end of the video above). There’s no hot reload, it’s just updating some state in Redux. This made editing visual properties on components super fast, easy, and intuitive. Basically combining the best of code editing with the best of drag and drop and LLM codegen.