Software Testing is context-dependent. Very much. 

But there are fundamental testing principles that can guide you on your journey to releasing high-quality software, regardless of your position in the team.

In the last couple of months, I’ve talked to several software teams that haven’t started testing yet. So I decided to share the topics and the questions raised during these meetings, as they are helpful for anyone involved in software development. 

Some of these principles come from theory and have proven to work in practice, others are purely empirical, but all of them will help you smudge the job description boundaries and take proper care of the quality of your features as a team.

Software testing is being conscious about doing the right thing all the time. 

Software testing is not achieved with clear-cut handovers and silos. Testing starts as soon as someone comes up with an idea about a new feature. This is when it’s time to start asking questions.

• Why is this a good idea?
• What is the part of that idea that must always work?
• What could go wrong when this idea gets implemented?
• What if? What if the user or the system we talk to doesn’t behave as we expected them? What if the circumstances under which the feature operates are different from what we’ve assumed?

Early testing

The most important testing principle is Early testing (shift-left testing) 

The later a defect of any kind is discovered, the more expensive it is to get fixed, and the cost grows exponentially from idea, through requirements, implementation to production. This cost is extremely high, regardless of the size of the story and the release cadence. 

A fix in the production environment doesn’t cost less because you have continuous delivery in practice, and you can quickly fix it. A part of the cost is the time to fix it, but the greater cost comes from the waste of implementing it wrong in the first place, especially if it should’ve been a requirement.

It is unbelievably more expensive to find missing requirements at code review. To wait for a tester to design, write and execute an input length validation test case, report the bug, and prioritise and plan when this bug should be fixed. Or to wait for your customer to complain that their company name is not fully visible in your invoice. Length validation should be a requirement, just like error conditions, performance criteria, and many more.

Most bugs are missing requirements. 

So to save the cost of your unhappy customer, start testing early and include everyone interested in the feature during the definition of the requirements.

If you care about writing high-quality user stories that inspire engineering and prevent production bugs, join me on December 1, 5 PM UTC for a User Story Analysis Workshop, where I’d show you how to turn a one-sentence user story into a low-risk, high-confidence requirement. 

You can’t test everything, and if you don’t see any more bugs, it doesn’t mean there aren’t any

You might have heard about these two principles as “Exhaustive testing is not possible” and “The absence of errors is a fallacy.”

They are only here to encourage us to be more creative. Of course, we can’t test our systems under all possible operational conditions. What we can do is use test design and risk assessment techniques to develop the best test cases. 

Keep asking what could go wrong all the time! This is where the most effective test scenarios come from. But don’t waste time, efforts and resources on highly improbable tests only to prove what a great tester you are. There’s no value in this. 

If you feel you’re done testing, look again. I’ve found my most interesting bugs this way when I feel we’ve more or less covered what we needed. Then just for the fun of it, I go off the script, and here they are. Testing never stops. We might pause it for a while, but as long as a feature is live, we must keep testing it every day. 

Test against your actual users. 

Your users are real people using home-speed internet on not a cutting-edge laptop, and they don’t have a degree in computer science. Or, on the contrary, they are your colleagues from another company dealing with the same problems you have daily with someone else’s SDK. Always think about the real person who will use your feature tomorrow. 

 Try to imagine how they feel and put yourself in their shoes. This is how you start if you want to deliver value to your users. 

There is always only one condition that makes the difference between Pass and Fail. 

This principle comes from pairwise testing and is the most effective strategy for two essential testing activities:

Effective Troubleshooting

You should always keep this in mind while you troubleshoot.

When we have a severe production incident to troubleshoot, under tons of pressure, and in a hurry, we naturally try to find the root cause very fast, hence trying random, wild guesses and different conditions at once. Well, I understand how you feel, but this would take you more time if you took a step back. Analyse the incident, brainstorm the potential causes, write down the most probable conditions that might be the reason and try them slowly and consciously one by one. 

You will discover it’s a combination of several elements that happened, but you would never guess them all at once. On the contrary, you will get lost and might end up testing the same thing again without getting any more clarity about the problem. Remember, always try to change only one condition in your test to be able to isolate the faulty one quickly. 

Smart Integration Test Strategy

This principle is especially powerful when you plan your integration testing. Companies that don’t have any automation very often make the same mistake of starting what they would call E2E UI automation tests because of the diversity of their inhomogeneous architecture and technology stack. 

Why starting with E2E UI is the worst approach to start with test automation? 

1. UI automation tests are slow. You can’t run them at Pull requests or at every change submitted to your source control.
2. UI automation is still quite unstable regardless of the framework in use. The company becomes very quickly disappointed with the results. 
3. Troubleshooting a failed UI test is slow, and the errors you’d get from a failed UI test rarely direct to the actual reason.
4. You can’t test your entire business/user E2E flow in one UI test. It will never pass. 

Instead, if you don’t have any automation yet, and no unit tests, I suggest starting with integration tests. 

Architecture diagrams must be your reference book 

Actually, start with drawing your architectural diagram to identify the hot spots in your system. It might be your first day. That’s the best question you could ask. What we need is a map (or a web) – a diagram of boxes and arrows connecting them. Nothing more than that. You don’t need to know what the Registration and Reporting services look like. You need to draw a bubble, type in ‘registration service’, then draw another bubble, type in Reporting and ask what the Registration service has to do with the Reporting service. 

I bet there is an arrow between these two, even though the User management is another service, and the Reporting could fetch the user rights from the User management directly. However, because we’ve forgotten, we still call Registration first, and because of that, the flow is outdated and times slower, with increased maintainability cost and regression risk. 

Draw your architecture diagram first

Your system might be based on a monolithic architecture, and some new parts are deployed as microservices. You inevitably use external services and integrate with other vendors. Most systems use both relational and NoSQL databases, and all your software lives somewhere in the Cloud. 

Once you figure out the high-level ecosystem, move to a deeper level.

Trace through any kind of dynamic testing and debugging the control flow and data flow. Try to trace how your pieces of code talk to each other. 

If the code is well-decoupled, you’ll notice that the conversation is usually between two or three components. 

Decoupled code means the misunderstanding happened between these two or three components. It will be propagated to the next components, and an E2E test might catch it, but it will cost a lot to get there. So instead of embarking on a very stormy experience with very uncertain outcomes, which is the E2E UI automation, be smart. 

Looking at your user journeys and your architecture at the same time, ask yourselves:

• What could go wrong with each part of the conversation?
• What part of this conversation must we always keep alive?

Answering these two questions is where you should start your automation initiative. 

This approach will help you become aware of 

• shady and legacy parts of the systems, where most of the bugs are clustered
• crossroads with not enough capacity suffering from very heavy traffic jams,
• completely forgotten components that are not used anymore, but you must support them because they still exist, or 
• wrongfully designated components.

The list doesn’t end here, but any drawing of your code would help you identify various violations of design and code quality principles. These violations are the most often cause of non-functional and production bugs. 

The reason is simple – the impact of a change is the most challenging thing to determine, especially if you don’t have anything to look at. We forget. Team members change, the software evolves, and even if it grows the right way, without a way to visually trace our evolution, we remain blind to the risks hidden in any change we make until they become facts.

Staying aware of your technology evolution is a very effective way to prevent severe production issues and untangle team dependencies. 

If you get used to drawing diagrams, the next application of flow diagrams is the flow of your next user story.

Draw the flow of your story

It’s again about asking questions and projecting their answers to a suitable canvas. 

Where is your user? Did they have a good sleep last night? Are they in a hurry? Where do they click first? Why? What do they want to do next? What do they expect from the system? Do they want just to get the job done or have small talk with the system first? 

The difference between a persona and a test user is that it’s always Monday morning for the persona and Friday evening for the test user. 

Draw their journey in a flow diagram. Start simple from wherever you like. 

The superpower of this technique is that it makes you ask the What if…? question. What if the user doesn’t do what we expect them to do? What if something happens in the meantime, and the system can’t respond to the user’s action? What will happen with the data in this case? Do we clean it, so the user will have to start all over again when we’re up, or do we preserve the progress? 

This technique is actually called State-transition diagrams, but you’ll get to it organically. Draw the flow first. See the example below.

We can test everything with a logical table

Last but not least, on the contrary – here’s my favourite testing technique. 

Whatever user story comes across your to-do column, you can illustrate the expected system and user behaviour with the so-called decision table. It’s nothing but a table where every column is a condition, and each row becomes a test case – a combination of several conditions.

I’m so used to decision tables that the first thing to do when I see a new piece of software is to start filling column headers. I probably draw the flow first, but then or at the same time, I jot down my suspects. These are my test conditions. The factors that determine the behaviour of the feature. The variables that make a difference in the result.

The conditions could be of any kind – something is or isn’t, something else happened or didn’t. Any representation of the program in execution – data created, deleted, changed, performed calculation, a decision taken – is a test condition. All these conditions affect the expected result, so we need to examine their possible states – true or false. 

Always attempt to minimise the number of significant conditions. 

Start by writing down all conditions that you think matter. Then build the logical table. See the example below.

You’d notice that it grows very quickly to an unmanageable size. Well, this is when you’d scratch every state of a combination of states that are not possible. 

Then you’d scratch those that don’t change the expected result.

Then you’d combine those that must always be in place together to achieve the expected result. 

And you’d end up with a neat set of combinations – test cases that you’ve generated relatively quickly that will build very high confidence about the functional quality of the feature.

If the number of the test cases is still very high, decide their priority and the execution strategy as a team.

This is not everything about software testing

These recommendations are not exhaustive, of course, and not very structured either, but if you’re a team wondering how to start with software testing, these are the major topics you need to consider first:

• Early requirements analysis, validation and shift-left testing
• Troubleshooting, changes impact, regression risk, testability and maintainability
• Test cases and building high confidence in the quality of the software with reasonable efforts
• Remember, we are customers ourselves

If you have any questions, find me on LinkedIn or Twitter and let’s talk about it. 

Take care, 

Hris