AI projects are expensive, time-consuming, and risky. And let’s be honest—most of them fail. A lot. If you’re going to take on the challenge, you’d better be sure you’re building something valuable. The problem? You can’t be sure until you experiment.
AI experimentation is a balancing act. On one side, you have the need to test, learn, and refine. On the other, AI and machine learning (ML) projects are notoriously complex and don’t fit neatly into agile frameworks and two week increments. So how do you figure out what works before you bet big?
Why We Need to Experiment with AI
AI is a big, risky bet, especially in tight budget times. It often takes longer than expected to figure out whether you’re on the right track because there are so many variables:
Is this valuable for customers?
Do we have the right inputs?
Does the output make sense?
How will people use this output?
What kind of guidance or context do users need?
One common hurdle is that business leaders often struggle to describe what they’re looking for in an AI/ML solution. They’re not experts in AI terminology, and the words they use—classifiers, predictors, summarizers—don’t always match the actual need. This mismatch can lead to misaligned expectations, wasted time, and frustration on all sides.
Prototyping to Align and Refine
Here’s how prototyping can guide your AI efforts:
Define the input and output. What data goes in, and what comes out? If you show the output to a user, does it help them take the next step? Is it ambiguous? Too complex?
Test assumptions. Does this use case even make sense in the customer’s workflow? Would they find it valuable? The faster you can test and learn, the less time you’ll waste heading in the wrong direction.
Ways to Experiment
There are many ways to experiment with AI, but here are some ideas to get you started:
Visual Prototypes
Use tools like Figma or even simple AI-generated mockups to show the flow and potential results. These prototypes are great for communicating intent to both stakeholders and AI experts. So everyone understands what we're trying to do.
Wizard of Oz Testing
This involves having humans perform the tasks that the AI is supposed to do, giving the illusion of a fully functioning AI system. This method helps teams gather early insights into user needs, identify gaps in workflows, and observe unexpected behaviors without needing to build the AI first.
Coded Prototypes
With small datasets and open-source models, you can create quick-and-dirty prototypes to test feasibility. While say a 70% accuracy rate wouldn’t cut it in production, it’s enough to decide if an idea is worth pursuing further. Tools like LangChain and Flowise can provide prebuilt modules to speed things up.
Fewer Failures, More Focus
Dr. Rebecca Bilbro from Rotational Labs says 87% of data science projects never make it to their intended end user. Why? Often, it’s because the interface and experience don’t align with reality. Experimentation isn’t just about validating ideas—it’s about designing AI solutions that actually fit into the workflows of the people they’re meant to help.
Experimentation isn’t just a nice-to-have in AI. It’s a necessity. In a world where the stakes are high, budgets are tight, and time is precious, the ability to test, learn, and adapt will separate the winners from the rest.
Image: An old watercolor experiment