Introduction to

...in a Nutshell

René Schwietzke, Xceptance

Outline

Just to get an idea what is coming

• What is XLT?
• Design and Features
• Test Suites and Scripting
• Test Execution
• Reporting
• Special Features

• This is not a programming class

Feel free to ask any question, anytime

History

Just some facts about the history of XLT.

1) Neodymium as spin-off for test automation.

Target Audience

Tool Design and Features

Open and Flexible

General Features

• Platform independent
• Java application stack and trusted Open Source components*
• Test suite concept
• Multiple approaches to write tests
• Ready to use reports
• Compare runs, trend runs
• Flexible load profiles
• Realtime test monitoring**

• Open data formats
• Adjustable reports in content and layout
• Custom data and reporting possible
• Massage data to group it
• Highly scalable
• Easy to deploy
• CI ready - Jenkins module
• Command line (CLI) driven for local development

*) HtmlUnit, HttpClient, JUnit, **) either standalone or soon in XTC

Platform Independent

Open Source Core

Do not reinvent the wheel. Use what is proven.

• HtmlUnit
• Apache HttpClient, OkHttp
• Apache Commons
• Selenium
• Jetty
• Hessian
• Freemarker
• ...

• We patch
• We fix
• We contribute back

Components

Agents can be anywhere. HTTPS communication and authentication, just one open port needed.
*) Optional

Test Suite Concept

Makes test projects more easily manageable

• Holds code, data, and configuration
• Perfect for version control
• Perfect for sharing
• One execution engine, many projects

• Leverage version control and:
  • Centralized and share data
  • Centralized and share configuration
  • Have a history
  • Work truly concurrently
• Use your preferred IDE

Requires GIT or similar tools

Mastercontroller

Your console for test execution - short name MC

• Command line console
• Interactive or direct execution mode
• Mostly local to the user
• Upload, Start, Stop, Download
• Create reports from last result
• Get agent information

• You can (re)attach to running tests
• MC has a full command line too
• MC usable in scripting
• Your MC can be remote too (ssh)
• Technically a running JVM
• XTC is hosting this for you

Only the interactive mode permits (re)attaching, even to tests of others.

Agentcontroller

Waits, receives, and distributes workload - short name AC

• MC communicates with AC
• Usually one AC per hardware box
• AC listens typically on port 8500
• Communication HTTPS and authenticated
• Spawns the test agents
• No direct interaction needed

• Port and authentication can be changed
• More than one AC per machine possible
• Does not know anything about other ACs
• Technically a running JVM

The AC is usually a don't-care component.

Agent

The workhorse of the load test and a component to watch

• A JVM that runs the users
• Most of the time more than one per box
• Each user is a thread with subthreads
• Inactive threads are possible
• MC calculates user distribution across agents

• Memory tuning is important
• Total size and size in relation to box
• A stalling agent doesn't block other agents

The tuning and sizing part is important for good and consistent results. We will get back to that topic later.

Test Approaches

Many ways to write tests

• DOM mode
  • Simplified Browser
  • No JavaScript
  • No CSS
• Request Level
• Real Browser (Chrome)
• Automatic or manual static content handling

• XLT API - Action API
• Extended XLT API (https://github.com/Xceptance/posters-advanced-loadtest-suite)
• No Coding: YAML (https://github.com/Xceptance/testsuite-nocoding)
• REST (https://github.com/Xceptance/rest-load-test-suite)

XLT Action API

public class Search extends AbstractHtmlPageAction
{
    public Search(AbstractHtmlPageAction prevAction, String searchPhrase)
    {
        super(prevAction);
        this.searchPhrase = searchPhrase;
    }

    public void preValidate() throws Exception
    {
        HtmlPage p = getPreviousAction().getHtmlPage();

        searchField = p.getByXPath("id('q')");
        Assert.assertNotNull(searchField);
    }

    public void execute() throws Exception
    {
        // Fill in search phrase
        searchField.setAttribute("value", this.searchPhrase);

        // Click on 'Search'
        loadPageByClick(searchButton);
    }

    public void postValidate() throws Exception
    {
        // response code = 200 ?
        HttpResponseCodeValidator.getInstance().validate(p);

        List result = p.getByXPath(this.searchResultPath);
        Assert.assertEquals(1, result.size());
    }
}

• Enforces thinking about before, now, and after state
• Encourages validation with assertions
• Little lengthy
• Passing of previous page required

Powerful including great form handling.

Extended XLT API

public class ViewCart extends PageAction<ViewCart>
{
    @Override
    protected void doExecute() throws Exception
    {
        // Get mini cart link.
        final HtmlElement cartLink =
            GeneralPages.instance.miniCart.getViewCartLink().asserted().single();

        // Click it.
        loadPageByClick(cartLink);
    }

    @Override
    protected void postValidate() throws Exception
    {
        // this was a page load, so validate what is important
        Validator.validatePageSource();

        // basic checks for the cart
        CartPage.instance.validate();
    }
}

• Keeps postvalidation enforcement
• Prevalidation gone
• Automatically handles previous page
• Less programming to get to data
• Page concept
• Separates activities from pages
• Available on GitHub (OSS)

Prevalidation returns as isPossible()

REST

// Send a post request which contains form data parameters.
final String lastId = Actions.get("Post Form Data Parameter", t ->
{
    final String id = String.valueOf(XltRandom.nextInt());

    // let's submit some post parameter (form data)
    final HttpResponse response = new HttpRequest().timerName(t)
                    .baseUrl("https://postman-echo.com")
                    .relativeUrl("/post/")
                    .param("id", id)
                    .param("foo", "true")
                    .param("bar", "false")
                    .param("parameter", "value")
                    .method(HttpMethod.POST)
                    .fire();
    response.checkStatusCode(200);

    final String responseContent = response.getContentAsString();

    // Ok, get us some response content for validation and use the jsonpath query
    // language for that.
    final ReadContext ctx = JsonPath.parse(responseContent);
    Assert.assertTrue(Boolean.valueOf(ctx.read("$.form.foo", String.class)));
    Assert.assertFalse(Boolean.valueOf(ctx.read("$.form.bar", String.class)));
    Assert.assertNotNull(Boolean.valueOf(ctx.read("$.form.parameter")));

    // Also let's return the id from the response.
    final String responseId = ctx.read("$.form.id", String.class);
    Assert.assertEquals(id, responseId);

    return responseId;
});

• Testsuite template on GitHub (OSS)
• One possible way
• Change and extensions possible

Prevalidation returns as isPossible()

No-Coding YAML

- Action:
    Name : Homepage
    Request :
       Url : ${host}/posters/
    Response :
       Httpcode : 200
       Validate:
            - Title:
                XPath: id('titleIndex')
                Matches: "\\w out our *"
            - Button:
                XPath: id('btnShowLoginForm')
            - MenuList :
                XPath : id('login-form')/ul/li[@name='Women']
                Count : 17
            - Header :
                XPath: id('titleIndex')
                Text: "Check out our new panorama posters!"
            - ShowCard-Button :
                XPath : "/html/body/div[1]/div[2]/div/button[1]"

    Subrequests:
        - Static:
            - ${host}/posters/assets/css/bootstrap-2.3.1.min.css
            - ${host}/posters/assets/css/bootstrap-responsive-2.0.4.css

• No code required
• XPath and Regex possible
• Pure request based approach possible
• Dynamic data and code access possible too
• Suite fully Open Source

https://github.com/Xceptance/testsuite-nocoding

Local Execution

Result Browser

Remote Test Execution

user@machine ~/xlt-4.10.1/bin $ ./mastercontroller.sh

Xceptance LoadTest 4.10.1 (r28afdd4d06)
Copyright (c) 2005-2018 Xceptance Software Technologies GmbH. All rights reserved.
Basic License (5 virtual users). This license does not expire.

Checking for agent controller reachability and XLT version conflicts ... OK

-----------------------------------------------
 What do you want to do?
-----------------------------------------------
 (u) Upload test suite
 (s) Start test
 (r) Report test status
 (d) Download test results
 (c) Create test report
 (a) Abort test
 (p) Ping agent controllers
 (i) Show agent controller information
 (q) Quit
=> u

Uploading test suite...
    Preparing:    0% .. 10% .. 20% .. 30% .. 50% .. 60% .. 70% .. 80% .. 100% - OK
    Uploading:    0% .. 20% .. 40% .. 60% .. 80% .. 100% - OK

Starting agents...
    0% ... 100% - OK

Load Profile
-------------------------------------------------------------------------------
Test Case | Arrival Rate [eff] | Users [eff] | Load Factor | Measurement Period
-------------------------------------------------------------------------------
TOrder    |                n/a |        1..5 |         n/a |            0:05:00
-------------------------------------------------------------------------------
                             0 |           5 |         n/a |            0:05:00

Test Properties

Infinite test setups with reuse possible

• Each test setup is set of property files
• All setups can be prepared upfront
• One will be referenced for the test
• Can have reusable components (includes)
• Everything can be varied or overwritten

That is not all of course.

User Settings

What the test setup permits

• Per test case rampup, warmup, delay, and shutdown
• Per test case user and arrival rate configuration
• Per test case runtime
• Iteration model available
• Run the same test case with different configs

• Thinktimes adjustable
• Keep-alive, timeouts, proxies
• Error rate limits can be defined
• Include and exclude requests based on patterns

That is not all of course.

Load Profiles

Scale up or scale down easily by using the factor feature.

Open Data

Open data for custom analytics, modification, and reporting

• CSV holds all measured data
• XML for intermediate data

• XSLT for transformation into HTML
• CSS for styling

R,Homepage.1,1440421580217,267,false,255,2830,200,http://localhost:8080/posters/,text/html,1,0,0,0,19,19,
R,Homepage.1.8,1440421580687,28,false,457,306680,200,http://localhost:8080/posters/img/products/XXL/XXL_4.jpg,image/jpeg,0,0,7,13,7,20,
A,SelectCategory,1440421583179,1094,false
T,TBrowse,1440421578625,8189,false,,

Perfect for searching things outside the ordinary.

Comparison Report

Realtime Test Monitoring

We have an AWS image available.

That is not all of course.

Reporting Magic

What you can do to get more details

• Split, join, and rename request timers
• Charts can be changed in size
• Charts can be capped or displayed logarithmically
• SLAs can be set
• Modify time periods aka zoom in
• Split test results later
• Change timezones

That is not all of course.

The need for CPT

The need for Client Performance Testing (CPT) explained

• Mostly JavaScript is not executed
• Real browser renders and times differently
• Real browser performance is not incorporated
• Third parties can block first parties

• A full browser is heavy
• Resource issues when running larger load tests
• Limited access to features to test everything

We want to see the whole picture and not just pure network.

Features

Use WebDriver API scripts and measure with a real browser

• Runs with Firefox and Chrome
• Measures using Navigation Timing and Performance API
• Accurate behavior simulation due to third party loading
• Accurate transaction handling aka full flow of user interaction
• Needs more resources than plain performance testing
• Tests execution is harder to control due to the timing and loading in the browser
• You can run even run full performance tests!

Chrome has even more events.

CPT Chrome

Lot of data.

Filter and Join Results

Results can be further worked on.

• Join test results together
• Split up tests by agents or test cases
• Use manual rules to combine things

All due to the open storage format.