Automated Testing – How we (hopefully) only make mistakes once

Posted by:

Despite our best efforts we make mistakes – it’s human nature. Our customers count on us to fix those mistakes quickly. Automated testing is one of the cornerstones in our software development lifecycle that ensure that when we make a mistake we only make it once.

Types of application testing & why you should care

There are generally two types of testing methodologies in software – ‘automated testing’ and well, uh, not-automated testing, call it regular or manual testing.

Manual testing is when someone who is developing the software or performing quality analysis on the software clicks through the application like a user of the application would be expected to. For example a user that wants to test a backflow device in C3 would need to:

  1. Login to the app with their username & password
  2. Search for a customer
  3. Find the device they want to test on that customer’s page
  4. Fill out the test details
  5. Click finish and receive confirmation

Did that test path work? great! What about if on step 3 in the above example a device didn’t exist? Now the person doing the testing needs to check that a different pathway through the application also works. Or how about a test where the device didn’t pass the first test and needed to be cleaned before a second test showed it past?

You can see how you quickly end up with very large amount of test paths that can take a lot of time to work through to ensure there aren’t any problems in the software.

This gets even worse when you need to make a change to a piece of the application. If you’re doing this work with regular testing you need to understand what was changed and think about what portions of the application you may have affected with the change – and then you need to go manually check that none of those areas were broken by your change.

Most of the previous companies that I’ve worked for did their software testing this way. It isn’t ideal but it works – you always need to have a person go through application paths and beyond just making sure they ‘work’ they also need to flow smoothly and make sense to people that aren’t the original developer. Many changes we make to an application are very safe and only occur in one or two places in an application – the size of some text on a page, or how a button on a page or two works. These changes tend to be easy to manually test – they only exist on at most a few pages and those can all be looked over manually.

Where manual testing falls down is when you start making changes beyond one or two pages. An example: in C3 one of the core concepts of is of course keeping track of backflow tests. The data we associate with these tests has a lot of tie-ins with other components – devices, water purveyors, testers, etc. When we have to make a change to our backflow test we’re making a change that probably touches fifty to eighty pages and operations!

If we only used manual testing anytime we made a change we would need to manually validate what we changed and that we didn’t unintentionally break anything else. As C3 changes and becomes more complex over time that takes more and more time and has greater and greater risk of missing something that was seemingly unrelated.

This is where automated testing saves the day. 

With automated testing as you build an application you build a test suite along side of it. Then when you make a change to your application your run this test suite and see if anything breaks. When it does you know that you need to review the change you made – sometimes you need to adjust your change and sometimes you need to adjust your test suite to support your new change. They coexist and support each other.

Our test suite currently has about 350 individual tests in it and is growing daily. These tests vary from short and sweet to long and drawn out. Examples from both ends of the spectrum include tests that:

  • Make sure that a user can successfully login to the application
  • Validates that the Dashboard properly calculates the displayed information correctly
  • A test for a device can’t be marked as passing if one of the checks was marked as leaking
  • A customer can successfully create & test a Route
  • validate that each of our test report forms get properly created and saved when they have valid information

Why do we use it?

Another example – Earlier this week we discovered a bug that prevented a specific type of test report form from properly generating after a user completed a backflow test. The fix was straight forward – a one line code change. Make the change, push it to production so our customer is happy, easy! Oh look at that – our test suite failed, let’s look a little bit deeper. Oh, of course, that one line code change changed the behavior of how we generated a different test report form. Our test suite saved us from trading one problem for a different one!

Nearly every problem that our customer’s ask us about gets a new test created to ensure that when we fix it that we’ll never inadvertently reintroduce that problem again.

The drawbacks to automated testing

Everything has it’s downside – automated testing isn’t without it’s problems:

  • We have to make sure that where we run our test suite matches where we run our application (operating system, patches, other constraints)
  • Managing the test suite takes time

So is it worth it?

Absolutely! Automated testing takes more time day to day but it allows us to safely make changes without fear that we’ll break something we’ve previously fixed. Long-term this saves a lot of time that we aren’t just chasing down unexpected bugs.

Most importantly it means that every mistake we make we’ll only make once. That’s something we’re proud of and we hope our customers appreciate.

1

Comments

Add a Comment