Software Testing and Quality Assurance
A Motivation in 60 min or less
About Xceptance
- Founded 2004, Offices in Jena and Erfurt, Germany
- Subsidiary in Cambridge, MA, USA
- 50+ Employees
- Focused on Software Test and Quality Assurance
- Performance Testing, Test Automation, Functional Testing, QA, and Test Process Consulting
- Mostly Active in Ecommerce and Internet
- Own Open Source Tool XLT (
) - https://github.com/Xceptance/XLT
About René Schwietzke
- Co-Founder and Managing Directory Xceptance
- Master of Computer Science (in German: Dipl.-Inf.)
- Programmer Since 1992
- In QA and Test Since 1998
- Performance Tester Since 1999
-
@ReneSchwietzke -
@reneschwietzke@foojay.social #java #qa #test #performance #performancetest #quality
I don't want to be a tester!
Why this presentation might still be useful for you
This presentation is also helpful for: Developers, Product Managers, Product Owners, Content Designers, Campagin Managers, Marketing Specialists, Writers, Designers, Trainers, Mechanics, Influencers, you name it...
Because life revolves around quality, impression, acceptance, and happiness. All that is brought to you by quality assurance and testing.
What is Quality?
A very much overused term
Mother of all Bugs!
macOS does not require a root password
2017: In one of Apple's biggest security blunders in years, a bug in macOS High Sierra allows untrusted users to gain unfettered administrative control without any password.
The Academic View
But what is quality? We just used that word?!
The totality of features and characteristics of a product or service that bear on its ability to meet stated or implied needs.
ISO 8402 (retired)
We define quality as conformance to requirements. Requirements must be clearly stated. Measurements determine conformance ... non-conformance detected is the absence of quality.
Philip B. Crosby: “Quality is free!"
The Real World View
One sentence to sum it up
"What should work, will work - what is not supposed to work, won't."
René Schwietzke
- Addresses the shortcomings of our typical thinking, we mostly forget the "not supposed to work"
- One typically only says what is needed, not what MUST not work
- Example: You log on with user name and password and MUST not be able to log on without password under any circumstances.
Money vaporized
Reuse can be devastating
1996: The failure of the Ariane 501 was caused by the complete loss of guidance and attitude information 37 seconds after start of the main engine ignition sequence (30 seconds after lift-off).
...reused the code from ... reference platform of Ariane 4... caused a data conversion from a 64-bit floating point number to a 16-bit signed integer value to overflow and caused a hardware exception. ... The supplier ... was only following the specification given to it, ... that in the event of any detected exception the processor was to be stopped.
What is Testing?
A Simple View on a Complex Task
The Academic Testing View
ISTQB[1]
Software testing is a process of executing a program or application with the intent of finding the software bugs.
It can also be stated as the process of validating and verifying that a software program or application or product:
- Meets the business and technical requirements that guided it’s design and development
- Works as expected
- Can be implemented with the same characteristic.
Wikipedia[2]
Software testing is an investigation conducted to provide stakeholders with information about the quality of the software product or service under test.
...can also provide an objective, independent view of the software to allow the business to appreciate and understand the risks of software implementation.
Test techniques include the process of executing a program or application with the intent of finding software bugs (errors or other defects), and verifying that the software product is fit for use.
The Real Life View
Let's combine all definitions
Testing consists of all activities that increase our confidence that the system will do what it should do and won't do what it shouldn't.
As result of testing, the behavior (or state) is frozen in time.
The Challenges
- What to test?
- Why to test?
- What are the limits?
- What limits to set?
- How long does it take?
- Do we know enough?
Reasons for Testing
Why do we test?
- Avoid Death, Insuries
- Dissatisfaction
- Social Responsibilities
- Personal Interests
- Money
Inflation
Euro conversion at Bank 24
"Customers of Bank 24 … were astonished on April 6th, 1999 evening to find that their securities accounts appeared to be overdrawn to the tune of 4 billion euro ($4.32 billion) ... which affected 55,000 customers."
"But the problem actually had more to do with human error than with a bug in Bank 24's software... Although the bank had tested and planned to use a new, euro-compatible program to carry out the quarterly calculation, because of human error, the old, pre-euro program calculated the amount."
cnn.com
Requirements
What do we need first?
What are Requirements?
The input to development and testing
...a requirement is a singular documented physical or functional need that a particular design, product or process aims to satisfy.
It is a broad concept that could speak to any necessary (or sometimes desired) function, attribute, capability, characteristic, or quality of a system for it to have value and utility to...
Requirements are also an important input into the verification process, since tests should trace back to specific requirements.
- No standard available
- Often either overstated or understated, dosage is mostly never right
- Term is often abused
- Not really clear what to write down
- E.g., most e-commerce requirements are incomplete
- A review is often missing, hence errors, and missing requirements are found late, too late... if found at all
Requirements
Requirements are just half the truth
- Typically only new stuff is stated: explicit requirements
- Many things are not stated aka the obvious: implicit requirements
- Mostly the should work is mentioned
- Shouldn't work is generally forgotten
Explicit
Implicit
Metric Pounds
Why Units of Measurement Will Cause WW-III
Simple confusion over whether measurements were metric or not led to the loss of a $125 million spacecraft last week as it approached Mars, the National Aeronautics and Space Administration said on Thursday.
An internal review team at NASA's Jet Propulsion Laboratory ... that engineers at Lockheed Martin Corporation, which had built the spacecraft, specified certain measurements about the spacecraft's thrust in pounds, an English unit, but that NASA scientists thought the information was in the metric measurement of newtons.
The resulting miscalculation, undetected for months as the craft was designed, built and launched, meant the craft, the Mars Climate Orbiter, was off course by about 60 miles as it approached Mars.
Areas
Where Requirements and Testing Meet
- Requirements should be written topic-based
- Requirements should be written user-centric
- Requirements are often redundant on purpose
- Usability
- Security
- Performance
- Documentation
- Safety
- Cost
- Testability
- Functionality
- Compatibility
- APIs
- Maintenance
- Supportability
- Sustainability
- ...
Testing 101
Basic Testing Knowledge
How do they test these?
Structure By Test Type
Let's start with test types and areas first
Definition
- What to test
- How to test
- How much to test
- How long to test
- How often to test
Test Types
- Unit Test
- Component Test
- Integration Test
- API Test
- End-To-End Test
- Security Test
- System Test
- User Acceptance Test
- Performance Test
- Load Test
- Smoke Test
- Compatibility Test
- Endurance Test
- Recovery Test
- Reliability Test
- ...
Radiation
Therac-25
It was involved in at least six accidents between 1985 and 1987, in which patients were given massive overdoses of radiation.[1]: 425 Because of concurrent programming errors (also known as race conditions), it sometimes gave its patients radiation doses that were hundreds of times greater than normal, resulting in death or serious injury.
Multidata Systems International
A software product of the company was involved in an accidental overexposure of patients in Panama in 2001 when the treatment planning software RTP/2 (vers. 2.11, 1991) reportedly contributed to 28 patients receiving excessive amounts of radiation at the Instituto Oncologico Nacional in Panama City.
...finding that the software permitted incorrect forms of data entry which in turn had led to miscalculation of treatment times.
Test Execution
Test Type != Test Execution
Test Types
- Unit Test
- Integration Test
- API Test
- End-To-End Test
- Security Test
- System Test
- UI Test
- ...
Test Execution
- Manual
- Semi-Automated
- Fully Automated
- Continously
- Staged
- Any test type can see any test execution even mixed.
Structure by Test Area
Areas again, because they match requirements nicely
Definition
- Topics of interest
- Areas the software plays in
- Areas the software should not play in
Test Areas
- Functionality
- Usability
- Security
- Performance
- Documentation
- Compatibility
- API
- Conformance
- Lifecycle
- Operations
- ...
Structure By Knowledge
What knowledge can you apply to write and execute tests?
Black Box Test
- No knowledge about the implementation
- Easy to act as an end user
- Helps to preserve independence and objectivity
- Risk of too little testing due to hidden aspects
- Classical requirement based testing
White Box Test
- Full knowledge about the implementation
- Hard to be independent
- Additional knowledge might help extend coverage
- Risk of ignoring aspects
- Classical unit test approach
Of course, there is something in between - the gray box test.
Heartbleed
The world was broken
RFC 6520 Heartbeat Extension tests TLS/DTLS secure communication: consisting of a payload, a text string, along with the payload's length as a 16-bit integer. The receiving computer then must send exactly the same payload back to the sender.
Nobody checked if the sent text and the sent text sizes matched. The server just returned all data matches the size data, hence sending back more and often with sensitive information.
Test Cases
What are test cases and how expressive are they?
A test case is a specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective... to verify compliance with a specific requirement.
- Many test cases are typically needed per requirement
- Positive testing (what should work) and negative testing (what should not work)
- Can be a precise execution definition or instructions with a lot of freedom
- Are not necessarily manual steps or something that is a direct unit test
- A test case is really hard to define formally...
How a test case might look like
The Classic
- ID: 001 Title: Positive Natural Numbers
- Open the application
- Enter 2
- Click the "Calculate" button
- Verify screen
- Repeat with 3 (9), 10 (100), and 1211 (1,466,521)
BDD Style
Scenario: Positive natural numbers
Given that the application is open
When I enter <Input>
And click the "Calculate" button
Then "The Square of X is <Result>" is displayed
|Input| Result|
| 2| 4|
| 3| 9|
| 10| 100|
| 1211|1,466,521|
The Short
- Test four positive natural numbers > 1
Pure Data
- 2/4; 3/9; 10/100; 1211/1,466,521
Expensive does not mean good
While attempting its first overseas deployment to the Okinawa, Japan (2007), a group of six F-22 Raptors flying from Hickam AFB, Hawaii, experienced multiple computer crashes coincident with their crossing of the 180th meridian of longitude (the International Date Line). The computer failures included at least navigation (completely lost) and communication.
Tips and some Advise
Just a Few Things that Might Help
Tips
Things to keep in mind
- Always start with zero inside knowledge
- Never assume anything
- Never rule out anything
- Nothing is obvious
- Always apply previous test knowledge
- Tests never relate to requirements 1:1
- If this worked yesterday, it might not work today
- Never stop updating test cases and plans
- If you find a defect, check if you have a test case for that
- If you find a defect, there is most likely another one
- If you have not found anything, question yourself!
Jobs at 
Your Presentation Will Continue Momentarily
Jobs, Internships, and More
www.xceptance.com/jobs/
Example 1 - x2
Let's try something together
To the Power of 2
Nifty UI for a Trivial Task
- Let's come up with manual functional tests
- What should be automated and why?
- What did we miss?
The International Power of 2
The requirements changed!
- What test cases do we have to add?
- What is the major difference?
- What did we forget in the first place?
Example 2 - Punch it
Testing is a General Concept
A Hole Puncher
Questions and Answers
Let's discuss what we have learned