Direct Goals

• Verification of Availability
• Functional Verification
• Performance Monitoring
• Issue Notification
• Location Comparison
• First and Third Party Checks

• Single url monitoring
• Does not use a browser

• Single url monitoring
• Uses a real browser
• Typically only checks for response code and some text
• Tracking of page load performance possible

• Run a business flow
• Scripted or url approach
• Can cover activities that require interactions

• Don't code, just measure
• JavaScript client extension to measure in a real browser
• Use real users and real traffic to measure and monitor

• Use a real flow to trigger RUM capturing
• Permits measurement on test instances

• Use regular test automation approach
• Use a DevOps approach
• Connect test automation and monitoring
• Maintain the test case as part of the test automation
• Real code possible

• Time to first byte
• Time to last byte
• Interesting because we know how long server processing and networking took
• Interesting because we know how long downloading took

Users don't care about technical details

• Users judge by impression
• No interest in DOM numbers
• Browsers expose Paint Timing API: first-paint, first-contentful-paint
• Alternative measurements are "first meaningful paint" aka something the user waited for, not just content (no API)
• The page is complete and does not load or move anymore ("visual complete") (no API)

• SaaS Offering
• Multiple Projects
• Roles and Permissions
• WebDriver based test automation or request based
• API testing possible as well
• Can instrument a browser for measurement
• Automatic metric capturing
• Browser based or request based, mix possible
• Service testing possible as well
• Ability to develop and test locally
• Uses GIT to centralize and communicate code

Tools

• XTC: the SaaS UI
• XM: the monitoring core
• XLT: our load testing framework for the automation and later measurement

Users

• Users are independent of an organization
• Users are identified by e-mail and authenticated by password
• User can be project and organization member
• User can have different roles in each project

• Metrics: The collected data dashboards
• History: The test executions in detail for diagnostics
• Configuration: Where the basic setup is done

• Name: shown everywhere
• Short Name: used internally (think JIRA short code)
• Description: Some verbal project purpose
• Avatar: A logo for the project used here and there

• GIT is the source of truth
• Holds test suite and basic(!) test config
• Code is automatically updated from GIT and recompiled
• Ability to switch branches dynamically
  • Site deployment has a text file
  • File defines branch
  • Site change will automatically switch monitoring

• A scenario is a test case setup
• It is not a test case, because you can map a test case multiple times and use it in many scenarios
• Each project has defaults that are inherited by each scenario
• Defaults simplify the setup

• A name for the scenarios to use
• The test class - JUnit
• A short description
• An execution schedule
  • How often such as every 5 min
  • In case of a failure, should it be retried quicker?
• Custom properties
• Notifications to send, text and e-mail
• Criteria to evaluate
  • Request runtime and errors
  • Transaction, action runtime
  • DomContent, OnLoad, First Paint, First Contentful Paint

Who can see and manage

• Project members and their roles
• Not all roles are fully utilized yet
• Membership is not needed for a notification (but to diagnose it)
• Membership is explicit by project

• Informs about the current data retention
• For result storage only, does not include metrics
• Good runs: No failures or criteria misses
• Bad runs: Anything that raised an alarm
• Read-only view

Questions

• What do we want to monitor?
• Why do we monitor?
• What can we use to prove it?
• What states can we have?
• When do we want to know a state change?
• Who should know?

• JUnit style test
• XLT interface to WebDriver
• Action block for proper naming
• Opens an url and waits for the dialog to show
• Verifies that we have not been redirected

• Notifications are optional
• Via e-mail and text
• Per Scenario setup possible or from project defaults
• Can be deactivated per entry or all together
• Notifications can be postponed to events happening in a row
• You can set a reply-to if needed

• When something bad happens
  • Cannot execute at all
  • Any unknown error came up
  • When an assertion occurred
• When a criteria is triggered
• When the threshold is up: How often should a failure occur before a notification is sent (default: 1)

A General Overview

• Time-series based data storage
• Captures action and page metrics as well as failure count and total test runtime
• Supports time based view, filtering, and split by location

Actions are wrappers

• Actions group logical activities
• They run their own timers
• Useful for non-page load activities

// open homepage
Action.run("Homepage", () ->
{
    Commands.open("https://www.foobar.com");
    Commands.waitForVisible("css=#js-data-privacy-save-button");
    
    Assert.assertTrue("Redirect missing", 
        Commands.getWebDriver().getCurrentUrl().endsWith("/"));
});

What detailed browser timings are collected

• XLT extends Chrome and Firefox with a plugin
• Captures Navigation Timing and Performance API data
• Timings published
  • FirstPaint
  • FirstContentFullPaint
  • DomContentLoaded
  • Load (OnLoad)

Metrics can be filtered and grouped

• Most metrics can be filtered and grouped
• Filter By Scenario: Show only one relevant test
• Filter By Action: Show only these actions
• Filter By Location: Show only selected locations
• Group by Scenario: Show chart lines by scenario
• Group by Location: Show chart lines by location

View by time and narrow down

• Select by last X hours or days
  • Last Hour
  • Last Day
  • Today
  • Last Week
  • ...
• Custom time ranges by date and time
• Data is preserved at least 30 days
• Working on 365 days
• Auto refresh can be set

• Extensive Starter Template
• Extensive Documentation
• Selection of Locations per Scenarios
• Google OAuth
• User based phone and email for notifications
• PXX numbers
• SLA definitions and monitoring
• Y-Axis limitation