Reproducible Randomness

A Power-Feature to Aid Testing Quality

What is Randomness?

A big question for a simple problem

Randomness, in its most common usage, refers to the apparent or actual lack of a definite pattern or predictability in information or events.

When something is random, individual outcomes are unpredictable.

  • Randomness in real life is based on physical properties
  • A computer, by default, does not have access to physics
  • So, randomness is modelled by an algorithm
  • PRNG: Pseudo-Random Number Generators
  • HRNG: Hardware Random Number Generators

Under The Hood

Just a quick view under the hood - Math incoming

PNRG

  • Start with a number, called seed
  • Apply some math
  • Get a pseudo-random number
  • Repeat
  • Same seed, same sequence
  • Each number advances the sequence
  • An RNG has an internal state
  • Most have equally uniform distribution

Linear Congruential Generator (LCG)

X n + 1 = ( a ⋅ X n + c ) mod m

  • a = 13
  • c = 41
  • m = 97
 0 41 89 34	95 15 42  5  9 61 58 19 94  2 67 39 63 84 66 26 88 21 23 49	96
 1 54 64  0 41 89 34 95 15 42  5  9 61 58 19 94  2 67 39 63 63 84 66 26 88
 2 67 39 63 84 66 26 88 21 23 49 96 28 17 68 52 38 50 12  3 80 14 29 30 43
 3 80 14 29 30 43 18 81 27  4 93 86 92 73 20 10 74 33 82 40 76 59 32 69 65

42  5  9 61 58 19 94  2 67 39 63 84 66 26 88 21 23 49 96 28 17 68 52 38 50

Under The Hardware Hood

Make things more secure

HNRG

  • Physical Entropy Source
  • Digitize analog signal
  • Conditioner: Remove bias and improve randomness
  • Health test
  • Your number
  • HNRGs are slow
  • Sources, e.g. Thermal Noise, Clock Jitter, Atmospheric Noise
  • PNRGs can be enhanced with entropy sources

A camera is a good source

A digital camera sensor creates perfect random noise

SHA-1: 4f93ef067ac75a15fc449dba838734e9f1bb9989

SHA-1: 149c6a70434fe11bf691c5bea318779f9c669332

The Testing Challenge

Why we need but don't want randomness

Fixed Testing

  • Caching effects (on any layer)
  • Data access effects
  • Might cause rhythm
  • Low coverage
  • More or less stress
  • Ordering effects
  • Errors are simpler to reproduce

Random Testing

  • More stress on components
  • Broader coverage
  • Discover ordering effects
  • Finding of functional side-effects
  • Errors are harder to reproduce

Randomness in XLT

Randomness but Reproducible

Concept Seeding

Concept Rounds

Load Test vs. Development

The difference that makes things reproducible

Load Test

  • com.xceptance.xlt.random.initValue goes into the seeding process as part of a calculation
  • If not set, the current time millis will be used

Local Test

  • com.xceptance.xlt.random.initValue is the only seed source
  • If not set, the current time millis will be used
  • If multiple iterations could run, they continued the load testing way with 31 * seed1 + 1

Debugging

How to reproduce randomness

  • Open a result browser
  • Click the test case name
  • Copy the initValue
  • Paste it into your dev.properties
  • Run your local test case


## Initial value to use the same randomness for local development
com.xceptance.xlt.random.initValue = 3541978581159734743

Limitations

Limitations might apply depending on context

  • If the system under test changes, the random sequence might be different.
  • If you change your code and you draw one more or less number, the result will change.
  • If you use random in class initialization code, things might be different (will be fixed).
  • Use a fixed seed for a load test as well to get a similar series replayed.
  • Due to the timing of users, the system under test might exhibit different behavior.
  • Fixed seeds for a load test might backfire.
  • A different XLT version might create different series.

Real Randomness

When you cannot afford repeatable randomness

  • Something must be unique and cannot repeat
  • Such as emails, account names, and IDs
  • You must not use XltRandom
  • java.util.Random and a random seed
  • Seed: System.currentTimeMillis() or System.nanoTime()
  • Or UUID.randomUUID()
  • Or java.security.SecureRandom but it is slower, ok for a few numbers such as an emails
  • Secure random uses a cryptographically strong pseudo-random number generator (CSPRNG) that includes system entropy.