How We Run Betas and Gammas
A behind the scenes look at what we do to ensure high quality and reduce stress as we approach new product launches.
Launching new software is terrifying!
Call it Agile, Lean, or what-have-you, we all like to work iteratively these days: deliver small, incremental chunks, listen to feedback, and repeat. It has low risk, and generally low stress.
Problem is, sometimes you have to do something BIG: launch a whole new product, or a new business model, or a big new set of features, or — like us — a fully rewritten version of our existing software.
It's a big deal, and try as you might, there's no real way to split it into smaller chunks.
So you work on it for months and months, and the new thing is finally good enough for proper testing. We usually call this alpha quality. You give the build to some internal stakeholders to play with, they come back with a ton of feedback, but generally it's nothing you didn't expect. It's all downhill from here.
After a short while, you're confident enough to announce that you'd be able to commit to a specific launch date. It's exciting, but everyone's stress level immediately goes up a notch: it's getting real!
So... how can we manage risk at this time in the development process? How can we ensure good quality, maintain speed, and keep the stress level reasonably low?
Private betas, to the rescue!
Private beta programs are a well-known solution to this problem. You give your software to some of your customers, and ask them to send you feedback and report any bugs they might find.
Since it's all private, you don't have to worry about your reputation being tarnished by a low quality build... expectations are clear on all sides.
How we run betas has evolved over time. There are tools that help you manage the whole process, but you don't need a fancy dedicated tool. We just invite people to a private Slack channel in our Slack community, and use it both to announce new builds and to reply to their questions and bug reports. It's informal, threaded (so it's not too noisy), and free for everyone involved.
The first thing to do is to build up a list of people who are interested in participating in a beta program — this is a job for the support team. While the software is in alpha (or even before), whenever a customer has an issue that will not be solved in the current version but will be solved in the new version, they ask them if they'd be interested in participating in a private beta, when it's available. What you want is a list of at least 100 names and emails.
I always describe the process of inviting people to a private beta as opening a sink's water faucet. The idea is to regulate the flow of water to optimize the amount of feedback the team can handle.
Let me explain: when we "go beta" at Balsamiq, it means that we send our build to TWO (yes, just 2) of our closest, friendliest, most active super-fan customers.
They will flood us with reports for a few days, which we fix as fast as we can. Usually at this point it's the developers themselves who reply to issues, but over time the support team takes over.
We go back and forth sending customers new builds a few times, and then our two friends' feedback generally slows down. Good, at least we know we've made 2 people happy. 😊
Time to open up the faucet a bit more! So now we invite another FIVE people to the beta. Again, picking from our friendliest customers, to keep the stress level low and our confidence high.
And so it repeats: the new people will inevitably report new bugs, because each of them uses our tool a bit differently. We fix those bugs, we make them happy, and when their feedback dies down, we invite more and more people.
As time goes by, we add 10 people, then 20, then another 50, then 100, and when we're really close, we invite everyone from the waiting list, even if they only get a couple of weeks with the beta before launch.
The important thing is to keep a volume that's high enough to keep your whole dev team busy, but still low enough for the team to be able to release updates and fixes quickly (like once a week).
If you can't keep up with their feedback, people will slow down in giving it to you, and gradually become less engaged.
If things go well, they'll stay engaged the whole time, and your confidence will go up and up. Once you've made all of your 100+ beta participants happy enough that they don't report any new issues, you're ready to launch!
Wait... that still sounds pretty stressful. What about testing the licensing or subscription code, or the website copy, or the docs, or training the support team on the new stuff? When do you do that? How do you know that people will actually pay for your product? What about scalability? Will your servers be able to handle the load after launch?
How do you get answers to all these things?
Do a gamma!
In the Greek alphabet, after alpha and beta, comes gamma!
A software gamma is what we call a period of time in which your application is effectively launched, live on production, but you're not telling the general public about it yet. The sign-up or download links are there, but not linked from anywhere on your site.
Your beta users automatically get included in the gamma, but now they have to start paying in order to keep using the software, just like public customers will.
Some people call this an "early access program", but we like the term gamma better... help us spread it across the software industry! 😊
Running a gamma gives you the opportunity to test all of your systems as if you were live: when a customer has an issue with the gamma, they no longer go through Slack to report it, but through your regular support channels. People have to pay or subscribe to continue using it. If none of your beta users decide to buy... that's a big warning flag. If a lot of them do, that's another huge vote of confidence.
You'll probably have to fix some bugs in your free trial and licensing code, since it's only getting tested now for the first time.
During this time you can also start taking the first measurements of scalability on your servers. The first performance issues usually start showing up now, and you can no longer afford having downtime for updates, so you'll work on setting up and testing your zero-downtime-update infrastructure.
We suggest running a gamma for at least 6 months. Sure you haven't announced, but money is already rolling in! 😊
Here's the revenue chart for when we ran the gamma and launched Balsamiq Cloud in 2017:
When everyone is confident that what's out there not only is good, but that your team knows how to maintain it, support it, and push out updates to it regularly... NOW you can launch!
Time to launch!
Launch, at this point, just becomes linking the new marketing pages from the rest of the site, publishing a blog post announcement, and that's it! You don't need to release anything because it's already been live for months!
Also, now in your announcement you can include a sentence like "we've been running privately for 6 months and already have 250 happy paying customers" which gives prospects a HUGE amount of confidence in your new product.
We've run betas and gammas for all of our major new versions or products. We can't imagine launching something new without them.
So, do you run betas? What about gammas? Do you do anything differently? Let us know!
Peldi for the Balsamiq Team
