Introduction to XLT and XTC

In a Nutshell

René Schwietzke, Xceptance

What are XLT and XTC?

XLT - Xceptance Load Test

Load Testing Tool
Performance-Test-As-Code (Java)
Real-Browser Support

XTC - Xceptance Test Center

Hosted Performance Testing
Real-User-Scenario Monitoring
Organizations/Users/Auditing/Sharing...

History

Just some random history facts

2005 - Prototype
2007 - Product offer
2010 - Test automation support added
2014 - Heavily improved test automation capabilities
2016 - Version 4.8
2017 - Script Developer¹ discontinued
2018 - XLT becomes free of charge, Apdex support, result coloring
2018 - Jenkins Plugin for Pipelines, Success Criteria Validation

2019 - Version 4.13
2020 - XLT 5.0 goes Open Source (APL 2.0)
2021 - XTC - SaaS offering
2023 - XLT 7.0, improved reporting
2024 - XLT 8.0, more reporting features
2024 - XLT 8.3, scorecard
2024 - XLT 9.0, HtmlUnit 4, JDK 21

But Why?

Why did we write our own toolchain?

XLT

Commerical offerings were too expensive
Restrictive licensing
Linux was not well supported
Insufficient reporting out of the box
Unusal or no scripting languages
JMeter, limited scale and reporting
Better debugging needed
Enhance trust in the results

XTC

Improve load test project efficiency
Centralize information
Reduce cost
Easier result sharing with customers
Archiving of projects (more than 150 a year)
Hand-over of projects to customers
Allow unattended load testing

Performance-Test-as-Code

Target Audience

This tool is for you, when...

you are a Software Engineer (SE)
you are a Software Engineer in Test (SET)
you love detailed data with the ability to postprocess
you need strong validation
you have an preferred Java-IDE
your tests should be flexible but predictable
you might want to postprocess data

This isn't your tool, when...

you need Capture & Replay
property files, source code, and version control scare you

Tool Design and Features

Open and Flexible

General Features

Platform independent
Java application stack and trusted open-source components*
Test suite concept
Multiple ways to write tests
Ready to use reports
Compare runs, trend runs
Flexible load profiles
Real-time test monitoring**

Open data formats
Adjustable reports in content and layout
Custom data and reporting
Filter and group data
Highly scalable
Easy to deploy
CI ready - Jenkins module
Command line (CLI) driven for local development

Components

The High-Level Overview

Test Suite Code and configuration
MC Mastercontroller
Agents The test executors
Grafana: Real-time test information*

Test Suite Concept

Makes test projects more easily manageable

Holds code, data, and configuration
Made for version control
A true development project (Maven, Gradle)
Other JVM languages possibles (e.g. Kotlin)
Suite design is flexible (e.g. Websocket with custom protocol)

Centralizes data
Centralizes configuration
Maintains a history
One can work truly in parallel
One can use their preferred IDE

Agents

The scaled-out workhorse

A JVM that runs the tests
Each user is a thread with subthreads
MC calculates user distribution across agents
Work is precalculated
Automated data and code upload
Automated result download

Any machine possible, Linux preferred
Homogenous machines preferred but not required
Single open port (custom) required
TLS and credentials for communication
JDK 17 or higher
Everything can be tested locally
Tested with up to 600 agents

Test Approaches

Many ways to write tests

DOM mode
- Simplified Browser
- No JavaScript
- No CSS
Request Level
Real Browser (Chrome)
Automatic or manual static content handling

XLT API - Action API
Extended XLT API
No Coding: YAML
REST
Real Browser LoadTest

Extended XLT API

public class ViewCart extends PageAction<ViewCart>
{
    @Override
    protected void doExecute() throws Exception
    {
        // Get mini cart link.
        final HtmlElement cartLink =
            GeneralPages.instance.miniCart.getViewCartLink().asserted().single();

        // Click it.
        loadPageByClick(cartLink);
    }

    @Override
    protected void postValidate() throws Exception
    {
        // this was a page load, so validate what is important
        Validator.validatePageSource();

        // basic checks for the cart
        CartPage.instance.validate();
    }

    /*
     * getViewCartLink()
     * Header.instance.locate().byCss("#miniCartMenu").byCss("a.goToCart");
     */

}

Automatically handles previous page
Less programming to get to data
Page and component concept
Separates activities from pages
Sophisticated data handling including exclusive data
Available on GitHub (OSS)

REST


// Send a post request which contains form data parameters.
final String lastId = Actions.get("Post Form Data Parameter", t ->
{
    // let's submit some post parameter (form data)
    final HttpResponse response = new HttpRequest().timerName(t)
                    .baseUrl("https://postman-echo.com")
                    .relativeUrl("/post/")
                    .param("id",  String.valueOf(XltRandom.nextInt()))
                    .param("foo", "true")
                    .param("bar", "false")
                    .param("parameter", "value")
                    .method(HttpMethod.POST)
                    .fire();
    response.checkStatusCode(200);

    final String responseContent = response.getContentAsString();

    // Response content validation, uses the jsonpath query language
    final ReadContext ctx = JsonPath.parse(responseContent);
    Assert.assertTrue(Boolean.valueOf(ctx.read("$.form.foo", String.class)));
    Assert.assertFalse(Boolean.valueOf(ctx.read("$.form.bar", String.class)));

    // Also let's return the id from the response.
    final String responseId = ctx.read("$.form.id", String.class);

    return responseId;
});

Testsuite template on GitHub (OSS)
One possible way
Changes and extensions possible
Other API designs applicable
Working on a Postman suite

No-Coding YAML

- Action:
    Name : Homepage
    Request :
       Url : ${host}/posters/
    Response :
       Httpcode : 200
       Validate:
            - Title:
                XPath: id('titleIndex')
                Matches: "\\w out our *"
            - Button:
                XPath: id('btnShowLoginForm')
            - MenuList :
                XPath : id('login-form')/ul/li[@name='Women']
                Count : 17
            - Header :
                XPath: id('titleIndex')
                Text: "Check out our new panorama posters!"
            - ShowCard-Button :
                XPath : "/html/body/div[1]/div[2]/div/button[1]"

    Subrequests:
        - Static:
            - ${host}/posters/assets/css/bootstrap-2.3.1.min.css
            - ${host}/posters/assets/css/bootstrap-responsive-2.0.4.css

No code required
XPath and Regex possible
Pure request based approach possible
Dynamic data and code access possible too
Used by a large European retailer, autogenerates test suites
Example for a custom approach with XLT
Suite on GitHub

https://github.com/Xceptance/testsuite-nocoding

Load Profiles

Flexible load profiles

Constant user rate
Arrival rate
Rampup, Warmup
Freely configurable per test scenario/user
Capping of users possible
Any runtime possible
Easy to scale (load factor)

Result Browser

See the data transferred in detail

Shows actions and requests of a transaction
Includes all HTTP headers
Shows unaltered responses
Displays approximated rendering
Displays test settings
Helps to find errors
Automatically generated in case of errors

Reporting

Standard Reports and Extra Features

Open Data

Open data for custom analytics, modification, and reporting

CSV holds all measured data
XML for intermediate data

XSLT for transformation into HTML
CSS for styling

R,Homepage.1,1440421580217,267,false,255,2830,200,http://localhost:8080/posters/,text/html,1,0,0,0,19,19,
R,Homepage.1.8,1440421580687,28,false,457,306680,200,http://localhost:8080/posters/img/products/XXL/XXL_4.jpg,image/jpeg,0,0,7,13,7,20,
A,SelectCategory,1440421583179,1094,false
T,TBrowse,1440421578625,8189,false,,

Load Test Report

Ready to use reporting

Single run test report
Best practice setup
Detailed data as HTML report
Easily sharable
Content and design modifiable
Can be recreated at any time from the results
Machine readable XML version

Comparison Report

Compare two runs

Compares two test runs
Quantifies differences
Detailed data as HTML report
Easily sharable
Content and design modifiable

Trend Report

See the trend

Compares many test runs
Quantifies differences over time
Detailed data as HTML report
Easily sharable
Content and design modifiable

Additional Data

What can be added

Custom timers for additional measurements
Custom values for gathered data during test
External data for data from other systems
Comments per test to aid archiving

Reporting Magic

You can do more at any time

Split, join, rename, filter requests
Adjust chart Y-axis
Adjust chart resolution
Set SLAs
Modify time periods aka zoom in
Split test results
Possible at any time after the test

Real-Browser Performance Testing

A real browser as engine

Real-Browser Support

Why real browser support helps

Real browser render timing differs
Loading decisions depend heavily on visual state
JavaScript might contain logic
Complex and dynamic client behavior
Perfect for monitoring!

Runs with Chrome and Firefox
Uses Navigation Timing and Performance API
Needs way more resources
Full scaled performance tests possible
WebVitals support

Client-Side Statistics Chrome

Web.Vitals

SaaS for XLT

What is XTC?

Xceptance Test Center

SaaS for XLT
Real-user monitoring and
Hosted load testing with XLT
Organization, user, and role management
Planning, execution, and evaluation of load tests
Designed for project work (project assignments)
GCP, AWS, Hetzner as datacenters
Runs on Google Cloud
GIT-based flow

Structure

The basic building blocks

Organization A client or other logical entity, sets the security scope
Project Monitoring or load test
User A real person with access to XTC
Role Set per organization and project to form a permission scope
Assignment Users are assigned with roles to organizations and projects

Security and Privacy

Basic features

Credentials based login
Password hashing PBKDF2
SSO (Google, Microsoft), enforceable
Two-Factor Authentication (2FA), enforceable
Different roles per assignment possible
Lock out per org
Last login tracking
Audit logging per org, project, and user

No analytics, no tracking
No cookies besides session state
No third-party services for user data
Data storage at Google Cloud Frankfurt
Audit log 180-days retention
30-day deletion protection

Monitoring

Real Scenario Monitoring

Advanced Synthetic Monitoring

Use real test cases with full test stack

Uses regular test automation approach
Connects test automation and monitoring
WebDriver-code (Java)
Enables performance monitoring
Full validation of possible
Third-party validation can be integrated

Full control over flow
Covers non-page loads as well
Able to react to state
Uses WebDriver automation stack
No third parties or instrumentation needed

Features

Features in a nutshell

Multiple datacenters
Schedules, alarms and notifications per scenario
Retention of good and bad test results
Raw XLT data available
Quiet-periods for target-system maintenance
Metrics overview
History including export
Local debugging and development
Code in GIT, properties can be supplied
Email, text, and Slack notifications

Performance Testing

Load and Performance Testing

Load Test Project

A nutshell view

Projects, User, Roles
Easy datacenter and machine-type selection
Results and report hosting
Public sharing with expiration when desired
Anytime report creation
Full data download
GIT-based, config overrides possible
Slack notifcations
Duplication and scheduling
Resource usage tracking
Easy rating of results and note taking of configs

Special Features for Client Projects

Things built to aid professional services

XLT

DNS resolver support/reporting
IP-address reporting
Custom header and agents
Dynamic header field tracking
Scale for broad IP ranges - Akamai
Validations and detailed error reporting
Merge-rules for more detailed filtering
Custom data/values
Predictable randomness
Site/Country/Localization support

XTC

User invitation workflow
Public sharing with expiration
Reviewer role
SSO/2FA
Rating criteria support
Scheduling
GIT-branch support

Roadmap

What is next?

Realtime load test data
Continuous UI-refinements
Integration of additional tooling
Private agents
Jmeter support
RESTassured support
Additional datacenters (Digital Ocean)

Extra Tooling

Tools for Load Testing

XCMailr

"Emails for Testing" as a Service

Email addresses at scale
Xceptance-owned domains
Forwarding for functional testing
Email intake at scale without bouncing for load testing
Random local-part generation/use (a81717q88728q2@varmail.net)
API for use with test automation
Able to access emails via automation (account confirmation, password reset, coupons...)

Performulator

Real-world runtime mocking

External services are often limited
Helps mocking of external services
Simulates realistic runtimes
Emulates certain response behavior
Small framework running on Google Cloud


@Controller("/")
public class WorldpayController
{
    @Inject
    LatencySimulation latency;

    @Post(uri = "/jsp/merchant/xml/paymentService.jsp")
    @Consumes(MediaType.TEXT_XML)
    @Produces(MediaType.TEXT_XML)
    public CompletableFuture<HttpResponse<String>>
            payment(HttpRequest<String> request,
                    @Body final String body, final String version)
    {
        return latency.latency(request, () -> new Worldpay(body));
    }
}

public class Worldpay implements ResponseInformation
{
    String requestBody;

    @Override
    public String render() {
        final var responseData = new WorldpayResponseData(requestBody);
        return Freemarker.render(
                        "/worldpay/worldpay.ftl",
                        Map.of("data", responseData));
    }

    @Override
    public ResponseTime responseTime() {
        return DEFAULTDYNAMICRESPONSETIME;
    }

    public static final long LATENCY_MIN = 1000;
    public static final long LATENCY_AVERAGE = 1300;
    public static final long LATENCY_P95 = 1500;
    public static final long LATENCY_MAX = 1600;
    public static final int LATENCY_JITTER = 150;

    private static final DynamicResponseTime DEFAULTDYNAMICRESPONSETIME =
        new DynamicResponseTime(LATENCY_MIN, LATENCY_AVERAGE, LATENCY_P95,
                                LATENCY_MAX, LATENCY_JITTER);
}