Software Testing and Quality Assurance

A Motivation in 60 min or less

About Xceptance

  • Founded 2004, Offices in Jena and Erfurt, Germany
  • Subsidiary in Cambridge, MA, USA
  • 50+ Employees
  • Focused on Software Test and Quality Assurance
  • Performance Testing, Test Automation, Functional Testing, QA, and Test Process Consulting
  • Mostly Active in Ecommerce and Internet
  • Own Open Source Tool XLT () -

About René Schwietzke

  • Co-Founder and Managing Directory Xceptance
  • Master of Computer Science (in German: Dipl.-Inf.)
  • Programmer Since 1992
  • In QA and Test Since 1998
  • Performance Tester Since 1999
  • @ReneSchwietzke
  • #java #qa #test #performance #performancetest #quality

I don't want to be a tester!

Why this presentation might still be useful for you

This presentation is also helpful for: Developers, Product Managers, Product Owners, Content Designers, Campagin Managers, Marketing Specialists, Writers, Designers, Trainers, Mechanics, Influencers, you name it...

Because life revolves around quality, impression, acceptance, and happiness. All that is brought to you by quality assurance and testing.

What is Quality?

A very much overused term

Mother of all Bugs!

macOS does not require a root password

2017: In one of Apple's biggest security blunders in years, a bug in macOS High Sierra allows untrusted users to gain unfettered administrative control without any password.

The Academic View

But what is quality? We just used that word?!

The totality of features and characteristics of a product or service that bear on its ability to meet stated or implied needs.

ISO 8402 (retired)

We define quality as conformance to requirements. Requirements must be clearly stated. Measurements determine conformance ... non-conformance detected is the absence of quality.

Philip B. Crosby: “Quality is free!"

The Real World View

One sentence to sum it up

"What should work, will work - what is not supposed to work, won't."

René Schwietzke

  • Addresses the shortcomings of our typical thinking, we mostly forget the "not supposed to work"
  • One typically only says what is needed, not what MUST not work
  • Example: You log on with user name and password and MUST not be able to log on without password under any circumstances.

Money vaporized

Reuse can be devastating

1996: The failure of the Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition sequence (30 seconds after lift-off).

...reused the code from ... reference platform of Ariane 4... caused a data conversion from a 64-bit floating point number to a 16-bit signed integer value to overflow and caused a hardware exception. ... The supplier ... was only following the specification given to it, ... that in the event of any detected exception the processor was to be stopped.

What is Testing?

A Simple View on a Complex Task

The Academic Testing View


Software testing is a process of executing a program or application with the intent of finding the software bugs.

It can also be stated as the process of validating and verifying that a software program or application or product:

  • Meets the business and technical requirements that guided it’s design and development
  • Works as expected
  • Can be implemented with the same characteristic.


Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test.

...can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation.

Test techniques include the process of executing a program or application with the intent of finding software bugs (errors or other defects), and verifying that the software product is fit for use.

The Real Life View

Let's combine all definitions

Testing consists of all activities that increase our confidence that the system will do what it should do and won't do what it shouldn't.

As result of testing, the behavior (or state) is frozen in time.

The Challenges

  • What to test?
  • Why to test?
  • What are the limits?
  • What limits to set?
  • How long does it take?
  • Do we know enough?

Reasons for Testing

Why do we test?

  • Avoid Death, Insuries
  • Dissatisfaction
  • Social Responsibilities
  • Personal Interests
  • Money






Euro conversion at Bank 24

"Customers of Bank 24 … were astonished on April 6th, 1999 evening to find that their securities accounts appeared to be overdrawn to the tune of 4 billion euro ($4.32 billion) ... which affected 55,000 customers."


"But the problem actually had more to do with human error than with a bug in Bank 24's software... Although the bank had tested and planned to use a new, euro-compatible program to carry out the quarterly calculation, because of human error, the old, pre-euro program calculated the amount."


What do we need first?

What are Requirements?

The input to development and testing

...a requirement is a singular documented physical or functional need that a particular design, product or process aims to satisfy.

It is a broad concept that could speak to any necessary (or sometimes desired) function, attribute, capability, characteristic, or quality of a system for it to have value and utility to...

Requirements are also an important input into the verification process, since tests should trace back to specific requirements.

  • No standard available
  • Often either overstated or understated, dosage is mostly never right
  • Term is often abused
  • Not really clear what to write down
  • E.g., most e-commerce requirements are incomplete
  • A review is often missing, hence errors, and missing requirements are found late, too late... if found at all


Requirements are just half the truth

  • Typically only new stuff is stated: explicit requirements
  • Many things are not stated aka the obvious: implicit requirements
  • Mostly the should work is mentioned
  • Shouldn't work is generally forgotten



Metric Pounds

Why Units of Measurement Will Cause WW-III

Simple confusion over whether measurements were metric or not led to the loss of a $125 million spacecraft last week as it approached Mars, the National Aeronautics and Space Administration said on Thursday.

An internal review team at NASA's Jet Propulsion Laboratory ... that engineers at Lockheed Martin Corporation, which had built the spacecraft, specified certain measurements about the spacecraft's thrust in pounds, an English unit, but that NASA scientists thought the information was in the metric measurement of newtons.

The resulting miscalculation, undetected for months as the craft was designed, built and launched, meant the craft, the Mars Climate Orbiter, was off course by about 60 miles as it approached Mars.


Where Requirements and Testing Meet

  • Requirements should be written topic-based
  • Requirements should be written user-centric
  • Requirements are often redundant on purpose
  • Usability
  • Security
  • Performance
  • Documentation
  • Safety
  • Cost
  • Testability
  • Functionality
  • Compatibility
  • APIs
  • Maintenance
  • Supportability
  • Sustainability
  • ...

Testing 101

Basic Testing Knowledge

How do they test these?

Structure By Test Type

Let's start with test types and areas first


  • What to test
  • How to test
  • How much to test
  • How long to test
  • How often to test

Test Types

  • Unit Test
  • Component Test
  • Integration Test
  • API Test
  • End-To-End Test
  • Security Test
  • System Test
  • User Acceptance Test
  • Performance Test
  • Load Test
  • Smoke Test
  • Compatibility Test
  • Endurance Test
  • Recovery Test
  • Reliability Test
  • ...



It was involved in at least six accidents between 1985 and 1987, in which patients were given massive overdoses of radiation.[1]: 425  Because of concurrent programming errors (also known as race conditions), it sometimes gave its patients radiation doses that were hundreds of times greater than normal, resulting in death or serious injury.

Multidata Systems International

A software product of the company was involved in an accidental overexposure of patients in Panama in 2001 when the treatment planning software RTP/2 (vers. 2.11, 1991) reportedly contributed to 28 patients receiving excessive amounts of radiation at the Instituto Oncologico Nacional in Panama City.

...finding that the software permitted incorrect forms of data entry which in turn had led to miscalculation of treatment times.

Test Execution

Test Type != Test Execution

Test Types

  • Unit Test
  • Integration Test
  • API Test
  • End-To-End Test
  • Security Test
  • System Test
  • UI Test
  • ...

Test Execution

  • Manual
  • Semi-Automated
  • Fully Automated
  • Continously
  • Staged
  • Any test type can see any test execution even mixed.

Structure by Test Area

Areas again, because they match requirements nicely


  • Topics of interest
  • Areas the software plays in
  • Areas the software should not play in

Test Areas

  • Functionality
  • Usability
  • Security
  • Performance
  • Documentation
  • Compatibility
  • API
  • Conformance
  • Lifecycle
  • Operations
  • ...

Structure By Knowledge

What knowledge can you apply to write and execute tests?

Black Box Test

  • No knowledge about the implementation
  • Easy to act as an end user
  • Helps to preserve independence and objectivity
  • Risk of too little testing due to hidden aspects
  • Classical requirement based testing

White Box Test

  • Full knowledge about the implementation
  • Hard to be independent
  • Additional knowledge might help extend coverage
  • Risk of ignoring aspects
  • Classical unit test approach

Of course, there is something in between - the gray box test.


The world was broken

RFC 6520 Heartbeat Extension tests TLS/DTLS secure communication: consisting of a payload, a text string, along with the payload's length as a 16-bit integer. The receiving computer then must send exactly the same payload back to the sender.

Nobody checked if the sent text and the sent text sizes matched. The server just returned all data matches the size data, hence sending back more and often with sensitive information.

Test Cases

What are test cases and how expressive are they?

A test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective... to verify compliance with a specific requirement.

  • Many test cases are typically needed per requirement
  • Positive testing (what should work) and negative testing (what should not work)
  • Can be a precise execution definition or instructions with a lot of freedom
  • Are not necessarily manual steps or something that is a direct unit test
  • A test case is really hard to define formally...

How a test case might look like

The Classic

  • ID: 001 Title: Positive Natural Numbers
  • Open the application
  • Enter 2
  • Click the "Calculate" button
  • Verify screen   
  • Repeat with 3 (9), 10 (100), and 1211 (1,466,521)

BDD Style

Scenario: Positive natural numbers

    Given that the application is open
    When I enter <Input>
     And click the "Calculate" button
    Then "The Square of X is <Result>" is displayed

    |Input|   Result|
    |    2|        4|
    |    3|        9|
    |   10|      100|
    | 1211|1,466,521|

The Short

  • Test four positive natural numbers > 1

Pure Data

  • 2/4; 3/9; 10/100; 1211/1,466,521

Expensive does not mean good

While attempting its first overseas deployment to the Okinawa, Japan (2007), a group of six F-22 Raptors flying from Hickam AFB, Hawaii, experienced multiple computer crashes coincident with their crossing of the 180th meridian of longitude (the International Date Line). The computer failures included at least navigation (completely lost) and communication.

Tips and some Advise

Just a Few Things that Might Help


Things to keep in mind

  • Always start with zero inside knowledge
  • Never assume anything
  • Never rule out anything
  • Nothing is obvious
  • Always apply previous test knowledge
  • Tests never relate to requirements 1:1
  • If this worked yesterday, it might not work today
  • Never stop updating test cases and plans
  • If you find a defect, check if you have a test case for that
  • If you find a defect, there is most likely another one
  • If you have not found anything, question yourself!

Jobs at

Your Presentation Will Continue Momentarily

Jobs, Internships, and More

Load and Performance Testing
Software Engineering
Functional Testing
Test Management
Test Automation
Bachelor and Master Thesis (RnD)

Example 1 - x2

Let's try something together

To the Power of 2

Nifty UI for a Trivial Task

  • Let's come up with manual functional tests
  • What should be automated and why?
  • What did we miss?

The International Power of 2

The requirements changed!

  • What test cases do we have to add?
  • What is the major difference?
  • What did we forget in the first place?

Example 2 - Punch it

Testing is a General Concept

A Hole Puncher

Questions and Answers

Let's discuss what we have learned