When it comes to debate, “to TDD or not to TDD” is right up there with “rent vs. buy”, “Batman vs. Superman” and “Chocolate vs. Vanilla”. Like the other debates, there is only one right answer to this question:
It depends on the situation.
The amount of Test Driven Development (TDD) to practice works best when viewed on a spectrum. There are situations where TDD saves a lot of time and effort, but there are situations where it does the opposite. This mostly depends on which stage the business is at and how important it is to get things right.
Let’s start at the beginning.
Startup
In the beginning, a startup is created from nothing. The founders make an educated guess about what the marketplace needs and set out to create a solution. Often times, the guess will be incorrect or incomplete. The company will need to pivot to a different area or drastically change it’s business model based on early customer feedback.
At this stage, TDD makes little sense. Business requirements will change which means the code will change. What you wrote last week might be thrown out or completely re-written. It will be completely unrecognizable from its original self.
If you had spent time writing tests at this stage, too bad. They too would need to be thrown out or completely modified. It’s a lot of extra work for very little benefit.
If you do choose to do TDD at this stage, understand that it’s very risky because you are putting faith in the idea that the business will not change. The experience of many startups shows that this is a rare case.
Your company may be an exception but it’s doubtful. If you are going to have a big impact right from the start you may want to go ahead and implement TDD in your development process. For example, if you are developing an stock trading algorithm that will be processing hundreds of millions of dollars out of the gate, let’s hope for the sake of humanity you put tests in place lest you end up like Long Term Capital Management.
If you do decide to go ahead with TDD at this stage, limit it to the very important things that probably will not change that much like payment processing, for instance.
Growth or Product/Market Fit
If a startup gets to the point where they reach product/market fit, this is the stage where the company should invest more resources in testing. Hopefully cash flow is increasing, the sales processes are being defined and more customers are coming through the doors and staying. This should result in the company having more resources available for testing. Your software should start to become more stable. You’ll probably notice that there are parts of the codebase that don’t need to be touched that much anymore.
At this stage, the first kind of testing you will want to get in place is regression testing. Put processes in place in your software development lifecycle whereby when a bug is found, the developer in charge of fixing it first needs to write tests that will catch the bug if it occurs in the future. The bug came up once. Don’t let it come up again.
It can be tempting to ignore testing at this stage but it’s a major risk if you do. It’s tempting to ignore because the developers haven’t experienced any testing at this company until now. They may think “Why start now?”.
It can be risky to ignore testing at this stage because depending on how poorly or well thought out your codebase is, you could be in danger of becoming one of those software companies that plays whack-a-mole. You fix one bug but several others pop up. If you’re offering a stellar product that has no competitors, this is probably a risk you can handle. If you’re in a crowded marketplace, the more bugs users encounter, the more likely they may start to look elsewhere for a product that doesn’t irritate them as much.
BDT – Bug Driven Testing – is a way to ease your development process into a more test-like approach. Developers like efficiency and there’s nothing efficient about having to solve the same bug more than once.
By putting in place BDT you’ll slowly start to catch more situations where old bugs would have been re-introduced and you can start to become more confident in your release process.
At this stage, don’t worry about writing tests for new features. It’s more important to get new things out the door fast and into customers hands.
Mature
As a company evolves along from product/market fit to mature, you should gradually be putting more testing processes in place. It’s natural evolution that as people and companies grow older, they get more conservative. The focus becomes more on keeping what you have rather than taking risks and trying something new. This is where TDD can have the most impact.
As a business matures, the risk of making mistakes becomes higher. If a startup with a handful of customers releases something that introduces a security vulnerability, no one really notices. If you have millions of customers that rely on your service, that becomes a nightmare.
In order to keep some of the youthful energy and risk taking alive, a company at this stage may choose to forgo TDD in their new product launches. If you’re trying to create a startup-like environment for separate products, don’t burden the teams with the processes that are in place on more established product lines.
For the established software products, TDD should be put in place by the time a software business has reached this stage if it’s not already. Does this slow things down? Yes, it will. But the objective at this stage is risk management and not speed. In general, speed and low-risk do not go together.
Practical Examples
If you’ve been around on the Internet for a while, you’ll probably remember Twitter’s Fail Whale.
In its early days, Twitter used to go down all the time. Eventually Twitter got better but for a long time, up-time didn’t appear to be a priority. Sure, people complained but what’s the worst thing that happened? A few people missed reading their twitter feed. So what? Looking at where Twitter is now you can say that their failures early on had very little impact on their business, if any.
Now contrast that to NASA’s thorough development process as outlined in this still relevant Fast Company article from 20 years ago.
Every time it fires up the shuttle, their software is controlling a $4 billion piece of equipment, the lives of a half-dozen astronauts, and the dreams of the nation. Even the smallest error in space can have enormous consequences: the orbiting space shuttle travels at 17,500 miles per hour; a bug that causes a timing problem of just two-thirds of a second puts the space shuttle three miles off course. Bill Pate, who’s worked on the space flight software over the last 22 years, says the group understands the stakes: “If the software isn’t perfect, some of the people we go to meetings with might die.” While their development process reduces the risk, all of that up-front planning and testing does have a cost:
This software never crashes. It never needs to be re-booted. This software is bug-free. It is perfect, as perfect as human beings have achieved. Consider these stats : the last three versions of the program — each 420,000 lines long-had just one error each. The last 11 versions of this software had a total of 17 errors. Commercial programs of equivalent complexity would have 5,000 errors.
…on a dollars-per-line basis, it makes the group among the nation’s most expensive software organizations. Conclusion
TDD lowers your risk by increasing your up-front costs. If your organization wants to pursue TDD, you have to ask yourself “Is this tradeoff worth it?”. In the cases I outlined above, generally the smaller or earlier on in it’s life the company is, the answer tends to be “no”. Once the company is having more of an impact and potential failures become a bigger risk then the answer moves to “yes”.