Testing in a Nutshell

How to Design, Plan, and Execute Tests

Disclaimer

Testing is not science, testing is art. Testing is similar to programming where many terms and definitions exist, as well as many approaches can be used to solve the same problem.

The terms of this document are intentionally not fixed up strongly but often kept vague due to the different meaning across the industry. Their meaning and usage should still be clear.

  • Feel inspired
  • See the need to think about testing in a structured fashion
  • Learn that testing never finishes
  • Learn that there is no universal truth
  • Learn about the limits of testing

Garbage in, garbage out!

Example 1 - x2

Let's try something together

To the Power of 2

Nifty UI for a Trivial Task

  • Let's come up with manual functional tests
  • What should be automated and why?
  • What did we miss?

As simple as it seems...

The International Power of 2

Now with localization!

  • What test cases do we have to add?
  • What is the major difference?
  • What did we forget in the first place?

Localization adds complexity quickly.

Just with more insides...

Now we see the code

/**
 * Square of a number transported via a string
 *
 * @param s the input
 * @param locale the locale to use
 * @return
 *       the square of the input as localized string or
 *       "N/A" of the input was invalid
 */
public static String squareIt(final String s, final Locale locale)
{
    final NumberFormat stringParser = NumberFormat.getInstance(locale);
    final NumberFormat formatter = DecimalFormat.getInstance(locale);

    try
    {
        // convert to double
        final Number number = stringParser.parse(s);
        final double x = number.doubleValue();

        // square it
        final double result = x * x;

        // string it again
        return formatter.format(result);
    }
    catch(ParseException e)
    {
        return "N/A";
    }
}
  • Extend the UI test cases with the knowledge of the code
  • Any chance to test more?
  • How about automation on code level?

We just went from black box to gray box to white box.

What is Testing?

A Simple View on a Complex Challenge

The Academic Testing View

ISTQB[1]

Software testing is a process of executing a program or application with the intent of finding the software bugs.

It can also be stated as the process of validating and verifying that a software program or application or product:

  • Meets the business and technical requirements that guided it’s design and development
  • Works as expected
  • Can be implemented with the same characteristic.

Wikipedia[2]

Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test.

...can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation.

Test techniques include the process of executing a program or application with the intent of finding software bugs (errors or other defects), and verifying that the software product is fit for use.

[1] http://istqbexamcertification.com/what-is-software-testing/
[2] https://en.wikipedia.org/wiki/Software_testing

The Quality View

But what is quality? We just used that word?!

ISO 8402 (retired)

The totality of features and characteristics of a product or service that bear on its ability to meet stated or implied needs.

Philip B. Crosby: “Quality is free!"

We define quality as conformance to requirements. Requirements must be clearly stated. Measurements determine conformance... non-conformance detected is the absence of quality.

Quality as term stays vague.

The Real Life View

Let's combine all definitions into one nice little sentence.

Testing consists of all activities that increase our confidence that the system will do what it should do and won't do what it shouldn't.

As result of testing, the behavior aka the state is frozen in time.

The Challenges

  • What to test?
  • Why to test?
  • What are the limits?
  • What limits to set?
  • How much and what should be automated?

Keep it in mind, because it will direct your testing into two directions, while other definitions typically leave one out.

Example 2 - MAX

Let's Try Something Together

A new Max method

For no apparent reason, we got a new and better java.util.Math.max replacement.

/**
 * Returns the greater of two int values. That is, the result is the argument
 * closer to the value of Integer.MAX_VALUE.
 * If the arguments have the same value, the result is that same value.
 *
 * @param a argument one
 * @param b argument two
 * @return the larger of a and b
 */
public int max(int a, int b);
  • New int max(int, int) method
  • No ideas about the internals, just the API
  • No requirements besides the documentation
  • Don't focus on the How rather on the What

Dead simple, but still we will get plenty of test cases.

A requirement change

Your boss dropped by and said that it has to be float not int.

/**
 * Returns the greater of two float values. That is, the result is the argument
 * closer to the value of Float.MAX_VALUE.
 * If the arguments have the same value, the result is that same value.
 *
 * @param a argument one
 * @param b argument two
 * @return the larger of a and b
 */
public float max(float a, float b);
  • What do we have to change?
  • What can we keep?
  • Do we see additional test fields/ideas beyond these simple test cases?

Don't question your boss` decision and yes, we can go really esoteric.

Requirements

What Input is needed

What are Requirements?

The input to development and testing

...a requirement is a singular documented physical or functional need that a particular design, product or process aims to satisfy.

It is a broad concept that could speak to any necessary (or sometimes desired) function, attribute, capability, characteristic, or quality of a system for it to have value and utility to...

Requirements are also an important input into the verification process, since tests should trace back to specific requirements. Requirements show what elements and functions are necessary for the particular project.

  • No standard available
  • Often either overstated or understated, dosage is mostly never right
  • Term is often abused
  • Not really clear what to write down
  • Most e-commerce requirements are incomplete
  • A review is often missing, hence errors and missing requirements are found late, too late... if found at all

https://en.wikipedia.org/wiki/Requirement

Requirements and Test Areas

Requirements are just half the truth, especially with standard processes and even more in e-commerce

  • Typically only new stuff is stated: explicit requirements
  • Many things are not stated aka the obvious: implicit requirements
  • Mostly the should work is mentioned
  • Shouldn't work is generally forgotten

Test Areas

Test areas are often not reflected in requirements. These are the typical test areas for e-commerce testing (list is incomplete and unsorted):

  • Usability
  • Security
  • Performance
  • Documentation
  • Functionality
  • Compatibility
  • API design
  • Styleguides

Test areas are an unspecified and open topic. Yes, we could fill up an entire training just talking about requirements and how to write them...

Testing 101

Basic Testing Knowledge

Structure By Test Type

Let's start with test types and areas first

Definition

  • What to test because only this is reachable
  • How to test because only this is available
  • How much to test because not all ends are connected all the time
  • Focus on topics by test type

Test Types

  • Unit Test
  • Component Test
  • Integration Test
  • API Test
  • End-To-End Test
  • Security Test
  • System Test
  • User Acceptance Test (UAT)
  • Performance Test

All this is topic, reach, and how to test at once. Don't overthink that. The books can drive you nuts here.

Really, don't ride the books, if you want to keep things simple and elegant. Testing is about results and not about terminology.

Unit Tests != Test Automation

Removing one common misunderstanding first

  • Unit testing does not mean test automation
  • Test automation is a way to execute tests
  • Unit tests describe what is tested
  • It might be practical to automate unit tests but it is not mandatory
  • Unit test are often very close to the code, hence more code is only the solution hence we automate
  • Any test type can be automated
  • Not everything can be automated
We will get to unit tests and their definition later again.

Test Execution

Test Type != Test Execution

Test Types

  • Unit Test
  • Integration Test
  • API Test
  • End-To-End Test
  • Security Test
  • System Test
  • UI Test
  • User Acceptance Test (UAT)
  • Performance Test

Test Execution

  • Manual
  • Semi-Automated
  • Fully Automated
  • Any test type can see any test execution even mixed.

Structure by Test Area

Areas again, because they match requirements nicely

Definition

  • Topics of interest
  • Areas the software plays in
  • Areas the software should not play in

Test Areas

  • Functionality
  • Usability
  • Security
  • Performance
  • Documentation
  • Compatibility
  • API
  • Styleguides
  • Conformance
  • Lifecycle
  • Operations

This list is not complete.

Structure By Knowledge

What knowledge can you apply to write and execute tests?

Black Box Test

  • No knowledge about the implementation
  • Easy to act as an end user
  • Helps to preserve independence and objectivity
  • Risk of too little testing due to hidden aspects
  • Classical requirement based testing

White Box Test

  • Full knowledge about the implementation
  • Hard to be independent
  • Additional knowledge can help to extend reach, coverage, or make the test more efficient
  • Risk of ignoring aspects due to too much knowledge
  • Classical unit test approach

Of course, there is something in between - the gray box test. Know some internal aspects, but approach as black box first, when no further testing is possible, reach into the white box.

This is not a rule to follow, just an aspect to know.

Test Cases

What are test cases and how much expressive are they?

A test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective... to verify compliance with a specific requirement.

  • Many test cases are needed per requirement typically
  • Positive testing (what should work) and negative testing (what should not work)
  • Can be a precise execution definition or instructions with a lot of freedom
  • Are not necessarily manual steps or something that is a direct unit test
  • A test case is really hard to define formally...

Of course, there is plenty of wiggle room here.

Example Test Cases

Remember our x2 application?

Classic Test Case

  • Open the application
  • Enter an INPUT value
  • Click the "Calculate" button
  • Verify that "The Square of X is RESULT" is displayed, where RESULT is the matching value from the data table
INPUT 2 5 7 12
Expected RESULT 4 25 49 144

With More Freedom

  • Calculate several positive natural numbers larger 1

The longer the test case, the more boring the maintenance.

Test Cases - Content

The Classic Approach

  • A test ID for later reference
  • A title
  • A description
  • A reference to requirements if any exist and a list of implicit requirements that are tested
  • Preconditions
  • Test steps that define what to do
  • Test expectation to define the expected outcome
  • Test data
  • Tear down information

The Compact Approach

  • A title or description to make clear what the test case verifies, mostly relates to requirements either implicit or explicit
  • Test steps including the expectation
    • Typically a compact statement what to do
    • Keep it open to get test coverage, such as don't tell how data should be entered or the application used
    • Optionally a more elaborate definition of the expectation
  • Test data for testing variations of the topic

There are many variations possible and often is defined by policies.

How to Write Test Cases

All about what format you can use to state a test case

The Classic

  • ID: 001 Title: Positive Natural Numbers
  • Open the application
  • Enter 2
  • Click the "Calculate" button
  • Verify screen   
  • Repeat with 3 (9), 10 (100), and 1211 (1,466,521)

BDD Style

Scenario: Positive natural numbers

    Given that the application is open
    When I enter <Input>
     And click the "Calculate" button
    Then "The Square of X is <Result>" is displayed

    |Input|   Result|
    |    2|        4|
    |    3|        9|
    |   10|      100|
    | 1211|1,466,521|

The Short

  • Test four positive natural numbers > 1

Pure Data

  • 2/4; 3/9; 10/100; 1211/1,466,521

These are just examples.

Test Plan

What is a test plan, what should it contain and how much planning is needed?

A document describing the scope, approach, resources and schedule of intended test activities.

It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice,and any risks requiring contingency planning [ISQTB].

  • Contains test areas, test types, test data, test cases and more
  • Defines completion criteria
  • Adds priorities
  • Might define order
  • Might define resources
  • Can contain any further information if needed

Once again, the dosage is important.

A Guide to Composing Tests

How to Compose a Test Plan and Write Tests
based on a Login Example

Example 3 - Login

A More Complex Example

A Login Screen

A simple login functionality for an application

  • Just a regular login screen
  • Assume a web application
  • Application details behind are not important
  • All other details will follow later

Of course, we are simplifying things here.

Questions to Ask

Ask these questions first before your start anything

Application

  • What is the state of the application?
  • Will we have access to the implementation details?
  • Do we have any written (explicit) requirements?
  • What are the implicit requirements?
  • Is it a UI only or also an API?
  • Can we automate anything? Do we want to?
  • Are there dependencies?

Environment

  • What is the development model?
  • What environments do we have available later?
  • What test data do we have or can we get/generate?

Test

  • What are the test areas?

This list is incomplete of course.

The State

Just to know where we are overall to assess urgency and reach

The Questions

  • What is the development state?
  • In which development cycle are we?
  • How serious is all of that?
  • What is the importance?
  • When will testing be involved?

Our App

  • First implementation
  • Not a prototype
  • Implementation is final
  • We got it when it was almost ready to test
  • Common screen for many applications later
  • Central to the application

The sooner the better.

Implementation Details

Assess if we are black box only and possibly user view only

The Questions

  • Can we access the code?
  • Can we access implementation details such as specs?
  • Will the application log information for us and will it be accessible?

Our App

  • We can access the code
  • It is Java on the server-side and JavaScript on the client side
  • Spec is accessible
  • We will be able to access logs

Explicit Requirements

Which requirements exist? Mix this with state and you get priorities

The Questions

  • What is stated explicitly in requirements that describes the overall behavior and expectations?
  • Ensure to look for non-functional requirements as well, such as performance, usability, code quality, and documentation
  • Security is a functional requirement and "the application must be secure" is not a requirement, just an insufficient statement
  • How high or low-level are the requirements? Is it rather a business level view or an implementation view?

Our App

  • A user should be able to log on with matching e-mail and password
  • A user should be presented with a meaningful error message when the password does not match
  • A user should be presented with an error message when the e-mail does not exist when trying to recover the password
  • A user's e-mail is case insensitive
  • HTML and CSS should be clean and minimal, naming of elements should aid automation
  • 200 logins per second and 20 password recoveries per second should be possible

Too long to list all of them here.

Implicit Requirements

What we know but have not stated

The Questions

  • What are implicit requirements in this area?
  • What previous knowledge do we have?
  • What is the target audience and what might they know?

Our App

  • Spaces should be trimmed for e-mail
  • Incorrect inputs should be highlighted
  • Tab should move to next field
  • Enter should trigger login process
  • No browser password completion
  • Password should not be displayed when inputed
  • Performance of login routine is identical whether or not the e-mail is known
  • ...

Well... writing down implicit requirements is kind of odd...

Automation

Shall we and can we automate?

The Questions

  • Is automation possible?
  • Is automation desired?
  • How much should be automated?
  • Will the automaton be beneficial?
  • Will the application be adjusted to support automation?

Our App

  • Yes, end-to-end automation
  • Happy Path

Automation captures state

Dependencies

Is there anything other that we might have to consider?

The Questions

  • Is there any other functionality that requires or depends on this feature?
  • What came first and what depends on what?
  • Do we have to add test cases here?
  • Do we have to add and execute tests for the other feature?
  • Do we have to create a new integration test?

Our App

  • There is a user import/export
  • Users can sign up
  • Users can be deleted
  • Similar UI elements already exist

There are certainly more connected features.

Development Model and Involvement

How do we get to our feature and how does the test fit into that?

The Questions

  • Do we do Scrum?
  • Is this a waterfall approach?
  • At which moment did the involvement or planning of test start?

Our App

  • Too late, the implementation is done

Too late is always bad. Requirement reviews can produce quite some feedback. Not necessarily because of incorrect rather missing requirements.

Environments

Where can we test and what can we see?

The Questions

  • Can we access logs or technical data?
  • Can we install locally?
  • Can we install at all?
  • Where can it be installed?

Our App

  • Local version possible
  • Logs will be written and accessible
  • We can install and setup ourselves

If we cannot test freely, we might have to cut down on test coverage.

Test Data

Not all test data is created equal

The Questions

  • When is data created?
  • Who can create data?
  • Can we get rid off data again?
  • What about a lot of data?

Our App

  • Login cannot exist without any side features
  • Assume we can create and import users for now

One of the examples where a feature alone is worthless.

And now... a Different Angle

View it from the Test Concept Side

Test Method

Separate the test by what you can do or see

The Questions

  • Is this a black box test?
  • Is this a white box test?
  • Is this something in between?
  • When is the right time to turn the test "white"?

Our App

  • All user facing activities are black box
  • All activities with code knowledge are white
  • Performance test is probably grayish
  • White box is used for all reviews and security

This is more a question to shape the view than to come up with new test cases.

Test Areas/Types/Processes

Use the guidance of test processes and areas to thing about poaaible tests

The Questions

  • Unit Test
  • Component Test
  • Integration Test
  • Regression Test
  • System Test
  • User Acceptance Test (UAT)
  • End-To-End Test
  • Performance Test
  • Security Test
  • Usability Test
  • Code and API Review
  • Documentation Review

Our App

  • We are not going to write that down here...
  • Let's practice and discuss!

Our large exercise.

Criteria

Use goals to focus better

The Questions

  • Define goals in the test plan
  • Goals per test type or area
  • A large goal for the test plan, such as:
    • Only minor defects
    • No performance and security problems
    • Final API state

Our App

  • Performance passed
  • Security, no defects
  • Usability and UI matches rest of the application
  • Only minor defects

Mostly, everything you planned should pass of course.

Priority

Use importance to focus better

  • Importance helps to focus and invest time better
  • Start with basic functionality first aka happy path
  • Target most important test area or type next
  • Continue with the area where most defects are expected
  • Continue the planned tests from here
  • Functional tests for requirements first
  • Security and usability next (login is the first impression)
  • Integration tests aka how it plays with other areas
  • Performance
  • And all other ideas

Some things can be done in parallel, not all.

A List of Tips and some Advise

Just a Few Things that Might Help

Tips

Things to keep in mind

  • Always start with zero inside knowledge
  • Never assume anything
  • Never rule out anything
  • Nothing is obvious
  • Always apply previous test knowledge
  • Tests never relate to requirements in a 1:1 fashion
  • If this worked yesterday, it might not work today
  • Never stop updating test cases and plans
  • If you find a defect, check if you have a test case for that
  • If you find a defect, there is most likely another one in that area
  • If you have not found anything, question yourself!

There is more...

Questions and Answers

Let's discuss what we have learned