That was stupid idea, wasn't it?
René Schwietzke, Xceptance GmbH
@ReneSchwietzke
@reneschwietzke@foojay.social
#java #performance #tuning #qa #test #performancetest #loadtest
https://github.com/Xceptance/XLT
https://www.xceptance.com/xlt/
I Care and Share
c b a
This presentation is licensed under
Creative Commons Attribution-ShareAlike 4.0 International License.
Third-party content could use a different license.
Most images come from Pexels under a Free Use License.
History Before the History
2000 to 2004
* Windows Server or else
The First Load Tests
Back to Load Testing with a New Company
Wasn't There Something Open Source?
A First Version
The How and What of the First Version
Not Everything Was Working Yet
[java] ===================================== Total =======================================
[java] Timer List Size: 16
[java] RegisterUser: 356 (1924msec) within 00:14:58,096
[java] SimpleSearch: 45920 (174msec) within 00:15:45,088
[java] Storefront: 11283 (36msec) within 00:15:43,762
[java] Checkout.OrderSummary: 2642 (891msec) within 00:15:22,356
[java] ViewCart: 2642 (587msec) within 00:15:23,727
[java] ProductDetails: 25625 (59msec) within 00:15:34,437
[java] Checkout.Unregistered.ShippingMethod: 2642 (218msec) within 00:15:22,400
[java] AddToCart: 19378 (585msec) within 00:15:35,220
[java] Checkout: 2642 (211msec) within 00:15:23,579
[java] Checkout.Unregistered.Addresses: 2642 (303msec) within 00:15:22,644
[java] Checkout.Unregistered: 2642 (117msec) within 00:15:22,617
[java] Checkout.Unregistered.Payment: 2642 (682msec) within 00:15:22,545
[java] MyAccountPage.LoggedOff: 356 (46msec) within 00:14:57,560
[java] MyAccountPage.RegisterWithUs: 356 (43msec) within 00:14:56,522
[java] SelectVariationProductDetails: 35306 (226msec) within 00:15:34,566
[java] SimpleBrowsing: 860800 (67msec) within 00:15:33,816
[java] Total Requests: 1017874
A Cool Tool Needs an Abbreviation
YART - Yet Another Regression Test Tool
Things We Wanted To Do Right
It's a Trap, Luke!
Trust Me, They Are Lying!
Test time: 1 h
- Total: 112,695
Mean: 501 ms
- P50: 250 ms
- P95: 1,950 ms
- P99: 5,170 ms
- Max: 16,689 ms
- P99.9: 7,830 ms
The Mean is Not Your Friend
Test Time: 3h 30 m
- Total: 114,386
Mean: 504 ms
- P95: 550 ms
- P99: 2,660 ms
- Max: 6,169 ms
- P99.9: 4,280 ms
One Stone, Two Birds, and a Costly Mistake
The Right Idea, But...
Failed for Two and a Half Men Reasons
One Thing Survived
Load Testing is About a Ton of Data
A Standard Load Test Result of a Large US Customer
Runtime | 3 hours |
User Scenarios | 17 |
Visits | 5,266,130 |
Page Interactions | 55,462,101 |
Total Requests | 122,185,828 |
Orders | 677,606 |
Errors | 70,491 |
Datacenters | 7 |
Load Generators | 50 / 800 Cores / 1.6 TB RAM |
Test Cases | 17 |
Transactions | 5,266,130 |
Actions | 55,925,554 |
Requests | 122,185,828 |
Events | 124,519 |
Custom | 5,232,721 |
Agent | 53,409 |
Data Lines | 189,751,960 |
11,313 requests per second, about 1 Gbps
How Many Data Points Are Captured?
For Transactions | 47,395,170 |
For Actions | 279,627,770 |
For Requests | 2,810,274,044 |
For Custom Data | 622,595 |
For Event Data | 26,163,605 |
For Agent Data | 1,228,407 |
Total | 3,165,311,591 |
Uncompressed Data | 48.72 GB |
Compressed Data | 4.10 GB |
Lines per Second | 17,569 |
Datapoints per Second | 293,084 |
Transaction
: Scenario execution Action
: Any kind of user interaction Request
: Obvious, isn't it
P.S. This is default logging, not extended logging with IPs, DNS, request methods, parameters, and more.
Open Data for Custom Analytics, Modification, and Reporting
R,ProductDetailsPage.1,1666819841884,2759,false,1345,40942,200,
https://acme.org/p/soap-126303030.html?cgid=foaming-hand-soap,text/html,0,0,2749,10,2749,2759,,GET,,,0,,
R,ProductDetailsPage.2,1666819844769,993,false,1858,429,200,
https://acme.org/en_US/__Analytics-Start?url=https...,image/gif,0,0,992,0,992,992,,GET,,,0,,
R,ProductDetailsPage.3,1666819845762,940,false,1305,1259,200,
https://acme.org/authiframe,text/html,0,0,940,0,940,940,,GET,,,0,,
R,ProductDetailsPage.4,1666819846703,1008,false,1350,2050,200,
https://acme.org/en_US/Cart-MiniCartContent,text/html,0,0,1008,0,1008,1008,,GET,,,0,,
A,ProductDetailsPage,1666819841883,7968,false
T,TAddToCart,1666819778626,72846,false,,,,
Perfect for searching things outside the ordinary. Formatted for screen.
Things That Stand Out
and Are All Based on Learnings
What Many Get Wrong in the First Place
Surprising Use Cases
A list of small but important things
Don't Give Me Excuses, Give Me Details!
When Randomness is Predictable
Concurrent Users
Avoid Hard Coding Data
public class ViewCart extends PageAction<ViewCart>
{
@Override
protected void doExecute() throws Exception
{
// Get mini cart link.
final HtmlElement cartLink =
GeneralPages.instance.miniCart.
getViewCartLink().asserted().single();
// Click it.
loadPageByClick(cartLink);
}
@Override
protected void postValidate() throws Exception
{
// this was a page load, so validate
// what is important
Validator.validatePageSource();
// basic checks for the cart
CartPage.instance.validate();
}
}
What Also Hit Us Often
// If there's only 1 party we don't have to compute anything
if (numberOfParties == 1)
{
return data;
}
else
{
// Contains all data for the current party
final List<T> partition = new ArrayList<T>();
// Compute the block numbers
final int blockSize = numberOfParties;
final int completeBlocks = (dataSize / blockSize);
// Check for remainder
final int mod = dataSize % blockSize;
final int blocks = completeBlocks + (mod > 0 ? 1 : 0);
// To get the partition data, fetch the current party's piece from each block
for (int blockIndex = 0; blockIndex < blocks; blockIndex++)
{
final int dataIndex = (blockIndex * blockSize) + currentPartyIndex;
if (dataIndex < dataSize)
{
partition.add(data.get(dataIndex));
}
else
{
break;
}
}
return partition;
}
Testing is Dynamic
Infinite Possibilities and Regex - Apply Retroactively, When Needed
Every feature was built, because we needed that at some point. Sometimes, we were ahead of the curve.
Success due to Functionality
Errors became a thing
Performance Was Turning South Quickly
Occationally Requests Got Lost
Some Business Things
Try to Extend the Reach
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Get We Get the Check, Please?!
So Much Competition,
So Much More to Do
Features, Features, Features
We can do things we couldn't have done without.
We never broke even but it was worth the money.
We are damn proud of the tool we have built.
Just pointers more information