How to Approach
Test Automation

Unit Tests, UI Automation, and other Automation Areas

Disclaimer

This is not going to be a coding training rather a training how to approach test automation from a pure planning and design point of view.

See it language agnostic, despite the fact that we will show Java examples.

  • Feel inspired
  • See the need to think about test automation in a structured fashion
  • Learn that test automation never finishes
  • Learn about the limits of automation
  • Learn to use code coverage correctly
  • Learn to approach refactoring correctly

An Example

Let's see an example first

Simple helper for CSV Records

Simple CSV record handler

  • Splits up CSV records using a custom field separator
  • That is all for now in terms of documentation!
/**
 * Encodes the given fields to a CSV-encoded data record 
 * using the given field separator.
 * 
 * @param s
 *            the plain fields
 * @param fieldSeparator
 *            the field separator to use
 * @return the CSV-encoded data record
 */
public static String[] decode(final String s, final char fieldSeparator)

Test Automation

What is this test automation thing?

Test Automation

What is test automation?

...test automation is the use of special software (separate from the software being tested) to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Test automation can automate some repetitive but necessary tasks..., or perform additional testing that would be difficult to do manually [1].

And...

  • Unit testing is not test automation
  • Unit testing is often only possible automated
  • Automation increases reach
  • Automation speeds up testing
  • Automation permits replacing boring testing by innovative testing

Test Types and Automation

What can be automated?

  • Unit Test: Smallest testable unit
  • Component/Module Tests: Larger functional units
  • Integration Tests: Communication between modules including communication to other parties (machine-to-machine)
  • End-to-End Tests: Full process tests
  • Functional Tests: Any functional unit independent of its representation
  • UI Test: Any functional unit through automation of UI

What is Test Automation?

In a nutshell

  • Replaces manual labor by means of automation
  • Creates easily repeatable tests
  • Quick result to qualify changes
  • Will pay off in the long run
  • Require predictable outcome
  • Upfront investment
  • One line description: Automation freezes behavior and API (when done properly)

Unit Tests

What is unit testing and what is the misunderstanding?

Unit Tests

What are unit tests?

In computer programming, unit testing is a software testing method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures, are tested to determine whether they are fit for use [2].

UNIT TESTING is a level of software testing where individual units/ components of a software are tested. The purpose is to validate that each unit of the software performs as designed. A unit is the smallest testable part of any software. It usually has one or a few inputs and usually a single output [1].

  • TESTABLE(!) part
  • SMALLEST testable part
  • Test in conjunction with data, usage and operating procedures
  • Can be a set of one more more program modules
  • Nothing about the how!
  • Nothing indicates automation, it just says test for now

The Misunderstanding

There is a conceptional misunderstanding of unit testing

By the Books

  • Unit test is a test type
  • It defines what to test
  • It doesn't define how to test
  • It doesn't say that this is by code or for code

What most People think

  • A unit test is what a unit test framework will test
  • That is not correct
  • Unit test frameworks are a technical tool to automate things
  • They don't say what to test or how to test, they are just a tool but strangely enough most people understand this as unit tests

Questions to ask

What to ask first?

  • Is it the smallest part?
  • How can I reach it? Is it just code?
  • What are the possible ways to test it?
  • Does it have dependencies?
    • Database
    • File system
    • Some other state or data container
    • System dependencies aka OS or setup
  • Most likely we consider automation, unit testing is often close to the code
  • API testing might be unit testing, but not necessarily
  • Complicated setup does not prevent unit testing, just makes it impractical for CI/CD
  • Unit test don't have to be fast, it is just nice to have

Feel confused?

Let's simplify the view on unit testing and test automation once and for all

Test Automation

  • Finding ways to make testing quick, repeatable, predictable, and effortless
  • How? Does not matter
  • Use frameworks that fit

What to automate?

  • Select what is suitable
  • Can be units, components, integrations, UI...
  • Select effortless areas that can be covered easily
  • No idea? The architecture might be wrong

Automation freezes behavior in time.

How to test - Step by Step

Let's assume we settled for test automation

  • How would we develop a test plan now?
/**
 * Encodes the given fields to a CSV-encoded data record 
 * using the given field separator.
 * 
 * @param s
 *            the plain fields
 * @param fieldSeparator
 *            the field separator to use
 * @return the CSV-encoded data record
 */
public static String[] decode(final String s, final char fieldSeparator)

Base Information

What do we need to know before we start?

  • Is this a public API?
  • Is this an internal API?
  • Is this just a piece of code that helps my feature and not for anyone else?
  • Is the unit standalone enough?
  • Does it have dependencies (code, data)?
  • Do I have access to the code?
  • What are the requirements?
  • Or as cheap question: What does it do?

Black Box

Just evaluate what you see and know

  • Approach as blackbox first
  • Use documentation to plan
  • If it is not documented, use implicit knowledge aka your expectations
  • Helps you to step back and see it from a different angle
  • Cover the API and behavior completely:
    • When official API
    • When just your code, you can shortcut according to context
    • P.S. Always expect that someone finds that code and uses it.
  • Don't peek into the code!

Official API

What to do when this is public or seriously shared

Public API Customer Facing

  • Think JDK style
  • Break it and hell breaks loose
  • Ensure signature, documented and undocumented behavior is frozen
  • Order, naming, return values, exceptions, error behavior and more

Public API Between Teams

  • Nobody will notice that you change things
  • Talking about decoupled APIs
  • You have to cover all behavior
  • Similar to customer API despite not having that many "customers"
  • As long as you don't test dependencies all the team, this is a real public API

Internal API

An internal API that is not decoupled

  • Like a public API, but tests will reveal quickly when something is broken as part of the built, even though it might not be full API coverage
  • You can make usage assumption because you know the consumers well
  • When you change and compile, you might see consumers
  • You might find behavior changes more easily without full test coverage
  • Play safe and think as of it as public API

No API

Only you use it

  • Hard to avoid that someone else uses your code
  • Mark your code clearly that is is not open for consumption
  • Test what you consider necessary to make sure the small unit works

White Box

When you think you have covered everything

  • Run code coverage
  • Under 100%, still work to do
  • 100%, darn... not that easy anymore
  • Review code for suspicious comments
  • Check for assumptions made
  • Check for hardcoded values
  • Check for behavior that is not documented
  • Check for all potential error conditions
  • Check for state information
  • Check for OS dependencies
  • Check for multi-threading requirements

Important Hints

What to obey, no matter what it takes

  • Black box first, documentation first
  • Forget the code and forget what your ideas were
  • Never make assumptions about the later usage
  • If an API (aka shared code), make sure the behavior is frozen by tests
  • Don't forget state
  • Vary your test data
  • Make sure you know where the later data comes from
  • Your test can influence the outcome (timing, synchronization)

Automation Criteria

Things you need to succeed or fail... it depends

Approach it again

Things to consider

  • Application has to be designed for automation
  • The better the architecture the easier to automate
  • Select by priority first
  • Select what is hard to test manually next
  • Select by module or component
  • Select by unit within
  • You might have to refactor to get testable code
  • No clear separation of concern, split it
  • Presentation and state has to be separated
  • Clear APIs for yourself and others

Automation Knowledge

Things learned over time

  • Automation is testing, have a plan
  • Writing finds more defects than running
  • Vary by data
  • Never forget edge-cases
  • Hard to automate means either incorrect design or not really worth the effort
  • Mocking can turn things into a self-fulfilling prophecy
  • Do what humans can't do
  • Change something, fix something? Add a test before and one after
  • Never run behind, fix failures immediately
  • Shaky tests need immediate attention
  • Automation only uncovers problems you foresee
  • Always verify results extensively

A last reminder

Just some last words...

  • Automation can easily create a false impression
  • Working tests don't mean that much if you have not well written and designed AND reviewed tests
  • Changes to tests should make your nervous
  • Adding tests is ok, removing tests is mostly not ok
  • Automation is coding, hence it can contains the same defects
  • Review your automation like any other code
  • Document automation code like any other code
  • Don't rely on automation as only qualification input
  • Automation is one criteria of many
  • Automation is a static opinion

Code Coverage

Is it any good?

What is Code Coverage?

The basic idea of code coverage

  • Measure which code has been reached when executing the code
  • How the code is executed does not matter
  • Typically used for checking test suites
  • Used to measure test coverage (sigh!)
  • "All reached, all good."
  • Line, branch, statement, and method coverage are typical metrics

Example

/**
 * Split a string at spaces
 * @param s the string to split
 * @return a list with substrings without the dividing space
 */
public List<String> split(final String s)
{
    final List<String> result = new ArrayList<>();
    
    int lastPos = 0;
    for (int i = 0; i < s.length(); i++)
    {
        final char c = s.charAt(i);
        if (c == ' ')
        {
            result.add(s.substring(lastPos, i));
            
            lastPos = i + 1;
        }
    }
    
    // deal with remainder
    if (lastPos != s.length())
    {
        result.add(s.substring(lastPos));            
    }
    
    return result;
}
@Test
public void test()
{
    split("AB");
}
  • Line Coverage: 100%
  • Branch Coverage: 83%
  • Resulting Test Coverage: 0%

And now?

Do we need coverage at all?

  • 0% coverage tells us something
  • <100% tells us something
  • 100% tells us that it is not simple anymore to add tests
  • It cannot be used to judge test coverage
  • It cannot be used to judge test quality
  • If you cannot reach 100% code coverage, you code logic is broken

Why do we need it?

  • < 100% coverage helps us to discover insufficient test coverage
  • 100% means you have to think harder
  • That's it! Period... nothing more.

Refactoring

How to approach the test part of it?

Refactoring?

What is it and why do we do it?

Code refactoring is the process of restructuring existing computer code ... without changing its external behavior. Refactoring improves nonfunctional attributes of the software. ...improved code readability and reduced complexity; ...improve source-code maintainability and create a more expressive internal architecture ... to improve extensibility [1].

  • Make things better
  • Don't add and don't change the external view
  • Prepare for the future

Challenges and Solutions

Why is refactoring difficult?

  • When you touch things, you break things
  • Ensure that the behavior and functionality stays the same
  • Ensure that the result is better code, not worse
  • Better code is hardly measurable
  • Test cases have to exist
  • The more automated the better
  • Tests might have to be changed too
  • Changing tests can change them for the worse
  • Self-betrayal problem

Protect your Investment

How to ensure safe refactoring

  • Define inner and outer behavior
  • Fix up outer behavior with tests
    • Unit tests
    • API tests
    • Integration tests
    • Whatever is necessary
  • Try to automate as much as possible
  • Not everything can be automated or is feasible to be automated
  • Before you refactor, review and add tests
  • Group your tests by have to stay unchanged and can be adjusted
  • Ensure that the new code is tested extensively
  • And all existing (unchanged) tests run as before
  • Make the call when behavior can change or sadly has to change

Golden Rules

Just a few simple rules and ideas

All green with test automation, especially with unit tests, does not mean the application is working.

Red tests in the contrary always mean that the application is not working.

There are never enough tests.

Unit tests alone are not enough. Test automation alone, is mostly not enough.

An Larger Example

Let's have something less simple

A Simple Map

Yes, just another HashMap.

  • Yet another map
  • Simpler interface for learning
  • Not compatible to JDK Collections
  • Goal is to show API and unit test design
public V get(K key);
public V get(K key);
public V put(K key, V value);
public V remove(K key);
public int size();
public void clear();

// optional
public boolean isEmpty();
public boolean containsKey(K key);
public void putAll(SimpleMap<K, V> map);
public Set<K> keySet();

Questions and Answers

Let's discuss what we have learned