This week at work I was asked “How do you decide which test cases to automate and in what order?” It’s a common question, a great question, and one you should always be prepared to answer. There is no right answer, but there are plenty of wrong answers. What follows is my opinion from years of experience and reading. Feel free to follow it to a T, adopt bits and pieces, or don’t.

The Value Metric

The answers to the following four questions can be combined into what I like to call a “Value” metric, a score from one to five with one being of low value. If you find thinking about these questions too difficult to consider for a single value, score them individually and average them.

What do your users do most often?

Bugs are missed. You should strive to ensure they’re not in code that is used frequently or constantly, such as login or general connectivity.

What can your users not live without?

This is similar to the previous question, but different. In the case of my current work, the product can be used for managing your drones and also flying them. Flying, in fact (I don’t have access to analytics yet, but let’s make an assumption for the sake of my point) is rarely used, but say you do get into the field and find it broken. That’s bad, real bad.

How long does it take to perform manually?

Reducing test execution time is a primary goal of automation. When faced with two similar cases, one taking 30 seconds and the other taking four hours, well, you know.

How often are you testing this?

Contrasting the above, if you run the 30 second test case 1000 times more often than the one taking four hours, it’s suddenly twice as valuable (30k seconds versus 14.4k seconds) to automate.

The Complexity Metric

The answers to the next two questions make up your “Complexity” metric. Again, I score from one to five with five being most difficult. It’s little more than “How many moving pieces are involved?”

How difficult is this to automate?

Will it involve integrating multiple systems? Have you done similar before? Is third party code involved? Are you using bespoke UI components? Are the building blocks for getting to this step already in place? Are there time components that cannot be expedited with existing tools?

Will this add flakiness?

This is a difficult metric to measure. Some things to consider would be how long the test takes to execute or does it have to do anything “hacky” (or high risk) in order to succeed?

Scoring

Divide your complexity by your value and you’re left with a number that can be used to rank your tests cases. A higher number means you should probably automate it sooner. 

Example #1: Login

Everyone does it and users cannot live without it. Easy five for value. It’s a couple of text fields; as easy as they get so a 1 for complexity. Score? Five. This will almost always be the first test case you automate.

Example #2: Verify the images all appear on the login screen 

Value? Every user will see this so that’s embarrassing, but it won’t cause the app to not function. Three or four. This is pretty easy to do so complexity again a one. Three and a half divided by one? Likely to be done just after login.

Example #3: Verify the images properly animate into view

Value? Again, every user will see this, but assuming you’ve already created the test above you at least know they’re displaying. Would a user even know if they were supposed to animate in? Unlikely. Value? Two. Testing animations is not simple. Complexity? Three or four. As is hopefully clear, with a score that’s a fraction, you’re unlikely to automate this anytime soon.

That’s not all!

The above is a very simplistic way of looking at this. There are many other considerations to take into account, and the following are only the tip of the iceberg:

  • Is the code required for this test fundamental to other test cases? Add value.
  • Are we trying to build a smoke test and want to touch a test or two from various parts of the application? Sort your tests by feature and grab the top x number of cases from each.
  • Has your application existed a long time so that certain parts of the code are rock solid and not being affected by current or future development? Add value to tests for the newer, less robust code.

Related reading

https://kevintuck.co.uk/how-to-decide-which-tests-to-automate/