Load Profiles

Load Test Configuration

User and Their States

What is a user and what states can it have?

  • Technically a thread on an agent
  • You could name it an executor
  • Simulates a real world visitor
  • Executes one transaction at a time
  • Will always run the same test case as transaction
  • Inactive and idle: No test execution in progress
  • Active and working: Test case code is executed
  • Active and waiting: A request is open and response is awaited
  • Active and thinking: Thinktime period runs
  • Active and blocked: Trying to execute code but cpu, memory, or IO does not permit that

Load Profiles

What kind of load profiles can you run

  • Change the load factor
    • By User
    • By Arrival Rate
  • Change the pattern
    • Steady
    • Ramp Ups
    • Fully Flexible

Fully flexible is effectively a combination of steady and ramp-ups.

Base Load Factors

Two factors and time determine the load profile

User Count

  • How many users could execute a transaction
  • Determines concurrency
  • User number must always be defined

Arrival Rate

  • Defines the work load for users
  • Number is transactions per hour
  • Can be undefined and therefore means infinite
  • See it as workload queue for the user count you defined

Basically you always define how often a test case should be executed aka a transaction should run.

Arrival Rate

What is arrival rate and why is it so useful

Imagine a super-market. You visit the store despite the fact that the checkout lines are long, because you do not know that when entering. Real world users do not know that the site is down or parts of it are slow. Therefore the traffic hits the infrastructure.

  • Mimics real world behavior
  • Server behavior does not influence visitors
  • Load increases if server does not keep up
  • Start points for transactions are pre-calculated
  • Users can be capped to limit maximum load

This is the most preferable way of running tests.

test.properties

A nice way to define individual workloads by having ready to run files configured. Avoids accidental overwriting of configurations.

  • Create a copy of test.properties
  • Name it
  • Modify the content
  • Use it by referring to it, share it, commit it

project.properties

## The file with test run specific settings.
com.xceptance.xlt.testPropertiesFile = test-max.properties

Command Line

./mastercontroller.sh -testPropertiesFile=test-max.properties

See the property hierarchy concept.

Base Definition

The minimal set of definitions for a test

## Typically in project.properties
## Map the test case aka name the test case and connect it
## with the test class. Name and class name often match but
## don't need too.
# com.xceptance.xlt.loadtests.TCheckout.class = com.company.tests.TCheckout
com.xceptance.xlt.loadtests.TTest.class = com.company.tests.TCheckout
com.xceptance.xlt.loadtests.TSearcher.class = com.company.tests.TSearch
## Announce the test to the load test setup
## This is a list of tests typically.
com.xceptance.xlt.loadtests = TTest TSearcher

## Define the test execution time and we will measure the entire
## time as well as the load comes up immediately.
## The 'default' indicates that this is for all tests configured.
com.xceptance.xlt.loadtests.default.measurementPeriod = 10m

## Set the load for the test mapping TTest
## Constant user rate set
com.xceptance.xlt.loadtests.TTest.users = 5

## TSearcher is arrival rate based
com.xceptance.xlt.loadtests.TSearcher.users = 50
com.xceptance.xlt.loadtests.TSearcher.arrivalRate = 1000

This is about the minimum. Everything extra is fun.

Time Dimensions

How to set and change the different runtimes

  • You can delay a test start with initialDelay
  • MeasurementPeriod states how long you want to collect data
  • WarmupPeriod and shutdownPeriod can remove noise
  • With rampUpPeriod, you can gradually increase the load automatically
## In case the test should not start immediately
com.xceptance.xlt.loadtests.default.initialDelay = 1h

## No measurement will take place during the first five minutes
com.xceptance.xlt.loadtests.default.warmUpPeriod = 5m

## We will measure for 10 minutes
com.xceptance.xlt.loadtests.default.measurementPeriod = 10m

## and shutdown the test over period of 5 minutes. Again without
## measuring anything during that time.
com.xceptance.xlt.loadtests.default.shutdownPeriod = 5m

## Over what period of time the load should be gradually increased?
## This clock starts after the initialDelay is over, so we rampup during
## warmup and half of measurement.
com.xceptance.xlt.loadtests.default.rampUpPeriod = 10m

Warmup is not really helpful, because you can remove any time period from reports during report creation if desired. This feature is here to match other tools.

Global vs. User Settings

Fine-tune the behavior with individual settings per user

## Announce the test to the load test setup
## This is a list of tests typically.
com.xceptance.xlt.loadtests = TTest TSearcher

## Define the test execution time and we will measure the entire
## time as well as the load comes up immediately.
## The 'default' indicates that this is for all tests configured.
com.xceptance.xlt.loadtests.default.rampUpPeriod = 5m
com.xceptance.xlt.loadtests.default.measurementPeriod = 10m

## Set the load for the test mapping TTest
## Constant user rate set
com.xceptance.xlt.loadtests.TTest.users = 5

## TSearcher is arrival rate based
com.xceptance.xlt.loadtests.TSearcher.users = 50
com.xceptance.xlt.loadtests.TSearcher.arrivalRate = 1000
## Adjust the users to start later and rampup differently.

## Announce the test to the load test setup
## This is a list of tests typically.
com.xceptance.xlt.loadtests = TTest TSearcher

com.xceptance.xlt.loadtests.default.rampUpPeriod = 5m
com.xceptance.xlt.loadtests.default.measurementPeriod = 1h 30m

## TTest should start later
com.xceptance.xlt.loadtests.TTest.initialDelay = 30m
com.xceptance.xlt.loadtests.TTest.users = 50

## TSearcher should rampup longer to be less aggressive
com.xceptance.xlt.loadtests.TSearcher.rampUpPeriod = 30m
com.xceptance.xlt.loadtests.TSearcher.users = 50
com.xceptance.xlt.loadtests.TSearcher.arrivalRate = 1000

Don't overuse this, because it is hard to keep track and the more differences in load over the entire period the harder to evaluate and make sense of the collected data.

Steady User Load

The setup of choice for best control

  • All users are active all the time
  • Can have all of the substates of active
  • Load goes down when server goes slow
  • Perfect for controlled experiments
  • Unsuitable for real world testing
com.xceptance.xlt.loadtests.TTest.users = 50

Don't run this for real user load tests!

Steady Arrival Load

Arrival rate is a steady load profile despite the base concept

  • Client is steady differently (not users, but workload)
  • Traffic goes up, when server goes slow
  • Real world simulation at its best
  • When user count is too small for arrival rate, you get steady user load
com.xceptance.xlt.loadtests.TTest.users = 50
com.xceptance.xlt.loadtests.TTest.arrivalRate = 1000
  • User count = ${arrival rate} / (3600s / ${length of scenario}) * ${safety factor}
  • Safety factor should be 2 at least, better higher to avoid running all user all the time

If you don't need the cap, give it more safety margin before being held back by too few users.

Add Ramp Up

Make the traffic come up slowly

  • Linear increase of load over time
  • Not influencing the total runtime
  • Either users or arrival rate is increased
  • Other steps sizes possible
  • Rampup per test case possible
com.xceptance.xlt.loadtests.default.measurementPeriod = 1h 30m
com.xceptance.xlt.loadtests.default.rampUpPeriod = 15m

com.xceptance.xlt.loadtests.TTest.users = 50
com.xceptance.xlt.loadtests.TTest.arrivalRate = 1000
com.xceptance.xlt.loadtests.TTest2.rampUpPeriod = 5m
com.xceptance.xlt.loadtests.TTest2.users = 500
com.xceptance.xlt.loadtests.TTest2.arrivalRate = 20000

Ramp up time can be equal to measurement time.

Variable Profiles

Simulate changing traffic over longer periods of time

  • Simulates ups and downs easily
  • Perfect for 24h runs or flashsales
  • You define traffic changing points
  • For users or arrival rate
  • Tool will interpolate the traffic between points
  • Works for default or per user
...arrivalRate = 0h/0 1h/1000 2h/500 4h/500 4h/1000 12h/0

Looks more difficult than it actually is.

Load Factor

Coolest way to adjust the defined workload quickly

  • The load factor is a multiplier to your current settings
  • Great for: "Let's do 50%."
  • You set your mix and 100% mark and the factor varies it
  • Per user or as default
  • See the test suite folder tools/LoadProfileEditor/loadprofile.html
## Run half of it
com.xceptance.xlt.loadtests.default.loadFactor = 0.5

## Run twice as much
com.xceptance.xlt.loadtests.default.loadFactor = 2

## Turn it into a fancy day simulation
...measurementPeriod = 24h
...loadFactor =  0h/0 1h/1 2h/1 2h30m/1 3h30m/1 4h/2 4h30m/1 5h/1 6h/1 6h30m/2 \
7h/2 7h30m/3 8h/4 8h30m/4 9h/4 9h30m/3 10h/3 10h30m/5 11h/5 11h30m/8 \
12h/5 12h30m/5 13h/5 13h30m/4 14h/3 14h30m/3 15h30m/3 16h/3 16h/4 16h30m/5 \
17h/6 17h/7 17h30m/10 18h/10 19h/10 19h30m/9 20h/7 21h/9 21h30m/9 22h/9 \
22h30m/8 23h/7 23h30m/6 24h/2

You can also apply this to default.

Iterations

If you need absolute execution certainty

  • Runs the exact amount of test repetitions
  • Fixes up transactions, not what happens in the test
  • Number is per user
  • Does not work with arrival rate
  • Not meant to be used with load factor
  • Measurement period can limit the test
## This will result in a total of 500 transactions
com.xceptance.xlt.loadtests.TTest.users = 5
com.xceptance.xlt.loadtests.TTest.iterations = 100

The measurement period is the ruling factor. If the iterations do not fit the time granted, the test might end without the total count reached.

Dual run test cases

Reuse a test case with different config

com.xceptance.xlt.loadtests.TStore1Checkout.class = \
	com.company.tests.TCheckout
com.xceptance.xlt.loadtests.TStore2Checkout.class = \
	com.company.tests.TCheckout

start-url = http://www.southfoobar.com/
TStore1Checkout.start-url = http://www.test1.com/
TStore2Checkout.start-url = http://www.test2.com/
## Announce the test to the load test setup
## This is a list of tests typically.
com.xceptance.xlt.loadtests = TStore1Checkout TStore2Checkout

## Define the test execution time and we will measure the entire
## time as well as the load comes up immediately.
## The 'default' indicates that this is for all tests configured.
com.xceptance.xlt.loadtests.default.measurementPeriod = 10m

## Config load
com.xceptance.xlt.loadtests.TStore1Checkout.users = 20
com.xceptance.xlt.loadtests.TStore1Checkout.arrivalRate = 100

## Store2 gets more traffic
com.xceptance.xlt.loadtests.TStore2Checkout.users = 50
com.xceptance.xlt.loadtests.TStore2Checkout.arrivalRate = 1000

Only makes sense if the test case supports different properties of course.

Questions & Answers

Your questions, your feedback.