Generating forms with large Language Models.
I’ve been experimenting with using multimodal large language models (LLMs) to generate forms. Farms generate a ton of data, but much of it still goes uncaptured. Often, this is because the sensors and software required either aren’t available or don’t align smoothly with farm operations.
A while back, I came across an experiment by Tim Paul where he used an LLM to extract form data from PDFs, integrating it into the UK Government Digital Service (GDS) forms design system. That got me thinking: could I create something similar to make farm data capture more flexible, consistently rendered, and—crucially—possible without needing to write code?
The result so far is a form-generation prototype that gives me what I want about 80% of the time.
There are quite a few moving parts to make this work: models, JSON schemas, CSS, and, of course, the mechanisms to generate and store the data from the resulting forms. It’s been a fun project to build, and I think the prototype demonstrates a lot of potential.
While the system isn’t 100% accurate yet, I’m happy with the results so far. There’s still plenty to improve—I’m particularly excited to experiment with different vision models (looking at you, Pixtral). Other improvements include refining prompts, tweaking grammars, and enhancing how results are evaluated.
LLMs are getting a lot of hype, but a couple of things aren’t mentioned as often. One big thing: multimodal, vision-enabled LLMs have become really good at OCR. A few years ago, I had a project that required digitising a lot of photos with badly handwritten forms. Traditional OCR tools like Tesseract just didn’t cut it. Now, based on my experience with this prototype, I think it’s possible—with the right extraction tuning. This could unlock huge value, especially for compliance records, from animal movement logs to agrochemical records and grain quality documentation. And that’s just scratching the surface.
One thing that became glaringly obvious during this project is that the ag industry has really missed the boat when it comes to data standardisation. I’ve had to create my own schemas for most of the forms I’ve tested, which is both an opportunity and a frustration. This is work that should already be underway. But as the saying goes, changing wetware (mindsets) is often harder than changing software.
Thanks again to Tim Paul and the GDS team for the inspiration behind this experiment. I particularly enjoyed the double-diamond design approach they’ve shared.