Experiment Format: Desktop or Field?
This article is the fourth in an exploration of the experiment-led design work I’ve been conducting for the last 18 months. You’ll find links to the whole series at the bottom of this article.
Once you’ve determined whether you need discovery or validation, there’s another decision to make: where will you conduct your experiment? The format you choose determines not just logistics, but the kind of insight you’ll gain, the risk you’ll carry, and how quickly you can move forward.
There are two formats to choose from: desktop experiments and field experiments. Each comes with distinct characteristics, benefits, and appropriate use cases.
Desktop Experiments: Thinking on Steroids
Desktop experiments are what they sound like: experiments you conduct at your desk rather than in the real world of work. They’re lower cost, lower risk, and generally quicker to execute, but they offer less real-world insight than field experiments.
Think of desktop experiments as thinking on steroids. You’re testing the practicalities of an idea without exposing it to the messy complexity of actual work environments. One client captured this perfectly: it’s like trying to create the world’s best aeroplane by sitting at your desk making paper aeroplanes and seeing which one flies furthest across your office. You won’t discover aerodynamics this way, but you’ll certainly avoid building something with the wings on backwards.
Desktop experiments become substantially more powerful when you have access to data. If your organisation has a robust data model, you can use it to predict the potential impact of a change before testing it in the field. This transforms your desktop work from pure speculation into informed hypothesis testing; you’re still at your desk, but you’re working with real constraints and real numbers.
The primary value of desktop experiments is that they stop you from making the biggest mistakes before you transition to the field. They’re faster, cheaper, and safer, which makes them ideal for eliminating obviously flawed approaches without wasting anyone’s time or putting anyone at risk.
Field Experiments: Testing in the Real World
Field experiments involve conducting tests in the environment where the work happens. They potentially take longer (though not always), cost more, and carry higher risk, but they give you the insight that matters: does this work when real people are trying to get real work done?
The trade-off is significant, and so is the responsibility we carry when we move into the field.
The Ethics of Field Experimentation
When you experiment in the field, you’re interfering with people’s jobs, which means you’re interfering with their ability to pay rent and buy food. This isn’t abstract; it’s someone’s livelihood.
We cannot conduct an experiment if the risk includes permanent damage to anyone around us. This isn’t flexible. This is not a “what if” situation. This is a “no, we just don’t do it” situation.
Everyone involved must understand where the lines are: senior leaders, involved line managers, the employees themselves, and us. Not only is this the right thing to do, it’s also how we build confidence in experiment-led design. We must be seen to be taking responsibility and believing in this way of working if we’re going to ask others to allow us to do it and to get involved in these experiments.
Stop Conditions: Your Safety Net
Field experiments require clear stop conditions. In their simplest form, these are: “If this happens, whatever ‘this’ may be, we immediately stop the experiment and do whatever is necessary to remediate the situation.”
Stopping may be enough. It may be that we need to stop and take corrective action. It may be that we need to stop and roll back a previous decision. Whatever it might be, the conditions must be clear before you begin.
These stop conditions might include negative business impacts, employee discomfort, or anything else your organisation’s risk appetite dictates. If there is a very low tolerance for risk, you will probably have more stop conditions. If there is a higher tolerance for risk, you might have a shorter list. The important thing is that you and everyone else involved understand these conditions before the experiment starts.
The Progression: Desktop to Field
As you move from discovery to validation experiments, you’ll also tend to move from desktop to field. This progression mirrors your increasing certainty and decreasing risk; you start by testing ideas cheaply and safely at your desk, then graduate to the real world once you’ve eliminated the worst possibilities.
In the next article, we’ll explore the methodologies available for conducting experiments in both formats.