SHAMROCK experimentation
A lightweight framework for encouraging a more structured approach to experimentation with emerging technology in a business context.
Let me start off by disclaiming that I do not consider myself a ‘proper’ scientist.
I mean, I studied a number of the sciences in high school (like most people, I presume), and I studied computer science at university. But my knowledge and experience (and credentials!) pale in comparison to some of the people that I been lucky enough to work alongside and are what I would call proper scientists - as in the dictionary definition of the term: a person who has expert knowledge in one or more of the physical or natural sciences.
Given that distinction, I find it interesting how frequently we throw around the term ‘experiment’ when referring to testing a new technology in a business context, and yet the activities that follow bear little resemblance to what a proper scientist would perform when using the same term.
This recently came up at a conference I was speaking at where we talked about the need to take an experimental approach to exploring emerging and disruptive technologies. The one in focus in that discussion (and in so many other forums at the moment) was generative AI, but the topic applies to plenty of other examples as well.
On the panel, I shared a framework/mnemonic that I have used for years when both performing and leading others in undertaking experiments with technology. It is a simplification of the Scientific Method and has served as a reminder to me to not just aimlessly play with new and shiny technology under the auspices of testing it - which, if we’re being honest, is often how the word ‘experiment’ can be used in the business world.
A number of people reached out to me after that conference asking me to remind them of the parts of that framework. So, I thought I would share it here in the hope that it is of use to others.
The SHAMROCK framework
SHAMROCK is an acronym formed of the following steps: Scope, Hypothesis(es), Ask, Method, Results, Observations, Conclusions, and Key measurements. I’ll provide a quick summary of each below.
Scope
The starting point for me has always been the question: what is the scope of this experiment? How broad, or narrow, is the focus of this piece of work. Other questions that I would ask as part of this sections:
What is the observation that has triggered this piece of experimentation?
Who is the audience who care about this?
Is this going to involve customers, or staff?
Will it be secret, or public?
Is it a standalone exercise, or part of a wider research piece of work?
Is this part of an RFI or RFP or procurement process?
Basically, this first part is focused on summarising the case for this experiment and why it should be taking place now.
Hypothesis(es)
At the core of any good experiment is one or more hypotheses that are able to be tested. I’m always in favour of these being phrased as propositions that are going to be attempted to be proven as true or false. The key word to begin each hypothesis with would be ‘That’ (as in “That it is possible to…” or “That this technology provides…”) The ability to test these hypotheses really is key to their usefulness. Stating them clearly and getting agreement can be the hardest part, so I tend not to rush this step.
Ask
The term I remember from high school was ‘apparatus’, but ‘Ask’ does just as well. This is the resources and equipment that is needed/being asked for to enable the experiment to test the Hypotheses as stated. In this space, I would list any technology and teammates that would need to dedicate some time to the experiment.
Method
Here, I would outline the proposed process(es) for testing the Hypotheses. A simple list of steps that I am going to take, and alternative methods in case one isn’t going to be enough to bring in the data we need to prove the hypothesis one way or the other. I like to write these steps down as though someone else is going to follow them to perform the experiment - i.e. with enough detail that they can be followed without some other knowledge that I have.
These first four steps - the SHAM - is, in effect, an outline of the approach we are planning to take. When I have previously codified SHAMROCK into document templates and business processes, there is usually a natural break at this point to seek approval or gain endorsement of the approach, and availability of the resources that have been asked for.
Then we move on to the ROCK…
Results
Following the Method(s) from the previous section, this is where we start to get the valuable data. What happened when we took those steps? Did we get the outcome we expected? Did it work? Did it fail? Nothing in this section is about casting judgement or weighing up good results vs. bad results: there are only results. I always try to be as quantitative as possible here and encourage my teams to do the same.
Observations
In contrast to Results, I bring any qualitative data into the Observations section. Where the Result might be that the technology was up and running as per the Method, the Observation might be that it took a lot longer than expected to get there. Or that it was harder than we had hoped. Or that the Result was only achieved for a moment. This is the section to add some colour and context to the data from the previous section and helps round out the experiment to ensure all bases are covered.
Conclusions
Here, we conclude by summarising the Hypotheses and showing where they have been proven true, false or inconclusive via the Methods used, and the Results and Observations gathered. There can be a tendency to start to make recommendations here, but I try to hold those out of this section and put them in a separate document that might refer to multiple SHAMROCKs before making a recommendation on a particular technology.
Key (measurements/data/metrics)
Inevitably, there will be some data that will be of more interest to business stakeholders - whether it be indicators of cost, or performance, or some other key metric that is relevant to business operations and decision processes. So, I use this final section to summarise some of those measurements that might be otherwise lost in the detail of the research activity.
That’s it. I have found that it can be quite useful to build lightweight processes around this framework to guide experimentation in teams. Like all frameworks, it is far from perfect, but it does help things be a little more structured and consistent, and that can be valuable in itself.
I think that keeping curiosity and open-mindedness foremost in exploration is always a good idea. SHAMROCK helps me with that - I hope it helps you too.