XLT – User Manual

Introduction

About this Document

This document describes all the components and features of the regression and load testing tool Xceptance Load Test (XLT).

To get the most out of this document, and to be able to work effectively with XLT, basic knowledge about web technologies, the Java programming language, and the JUnit concept is recommended. You definitely don’t need to be an expert, but being familiar with these topics will help you better understand the next sections.

What is XLT?

XLT is an abbreviation for Xceptance LoadTest. XLT offers an easy way to write and run regression and load tests for web applications. Nearly every software that provides access via HTTP/HTML can be tested using XLT. There is also extensive JavaScript support for testing applications that use Web 2.0 technologies. XLT supports not only web testing but also SQL tests, RCP-based application tests or any other test that should run as a load or performance test in a distributed manner. Since XLT is written in Java it can be run on any platform that supports Java.

XLT Script Developer

The easiest way to start is with the XLT Script Developer. It operates as a Firefox add-on and provides an easy-to-use user interface for developing and running test cases and entire test suites. XLT Script Developer can record the page flow while navigating through a website and features a wide range of validations on the web page.

XLT Framework

Scripting test cases using the Script Developer is easy, but keep in mind that you are limited by the strictly linear approach and the set of available commands.

The XLT Framework provides different approaches for writing test cases in Java. Starting with Java code automatically generated from the recorded Script Developer test cases, you can extend your test suite by using the XLT API. This features a higher level command scripting API with a very intuitive syntax as well as the WebDriver API and also the lower level XLT Action API based on HtmlUnit.

XLT API provides a programming paradigm to translate all test scenarios into JUnit4 tests. The principles of JUnit4 and its annotations are used to implement and tag test cases. This way, each XLT test is a JUnit test, so XLT tests can be executed just like any other unit test within an existing build process.

Installation Instructions

It is recommended that you install Xceptance LoadTest before continuing with this manual.

System Requirements

Hardware

Typically, XLT requires a CPU running at 1.5 GHz at least and about 1 GB of RAM. XLT can run on slower hardware but this depends on the tests themselves. The default installation requires about 85 MB of hard disk space, but load test results might require more free space, so at least 1 GB of free disk space is recommended.

Software

XLT runs on any operating system for which a Java 6 (or higher) virtual machine is available, such as Microsoft Windows, Linux, Oracle Solaris, HP-UX, or Mac OS X. We recommend using Oracle’s JDK if it is available for your platform. XLT will also run in JVMs provided by other vendors (e.g. BEA, HP, IBM), but this has not been tested extensively. In order to view the HTML load test reports generated by XLT, a modern web browser is required, such as a recent Firefox or Chrome, Internet Explorer 8, or Safari 5. Please note that JavaScript needs to be enabled to fully utilize all functionality. If you plan to use the XLT Script Developer, you will need Firefox 4 or higher.

Installing XLT

The installation of XLT is simple. Just unzip the XLT archive¹ to a file system location of your choice. The root directory is part of the archive, so you do not need to create it separately. Please note that although spaces in the path are supported by XLT, it is easier to code tests when the path is free of them.

On Unix-like systems, you need to grant execution rights to all *.sh files in the bin directory.

chmod u+x <XLT>/bin/*.sh

If you received a license file, copy it to the directory <XLT>/config. A Basic License for XLT does not require the installation of a license file. In this case, XLT is not limited in time or functionality, but the number of virtual users is restricted to five. Also notice that in this case the Basic License Terms apply. See the file <XLT>/doc/license.html for more information.

Finally, make sure that the executable directory of your Java installation is listed in your PATH environment variable so that the XLT start scripts can find the virtual machine runtime.

In order to install the XLT Script Developer, an extension to the Firefox browser, you need to carry out the following steps:

1. Start Firefox.
2. Click File > Open File....
3. Navigate to the <XLT>/tools directory and select the .xpi file. The Add-On installation dialog should appear.
4. Click Install to finish the installation of the Script Developer extension.

Alternatively, drag the .xpi file and drop it onto the Firefox window to install it.

¹ The XLT archive can be either obtained from our website or from our Maven-compatible repository which allows Maven and Ivy users to include XLT and all of its dependencies conveniently. Please see Writing Web Tests for detailed information how XLT can be integrated in your Maven/Ivy project.

Updating XLT

Before you update to a newer XLT version it is highly recommend that you back up all modified files and project-specific or customized settings. In particular this includes:

• All your test suites (especially if they are stored in a sub-folder of the XLT installation directory)
• Result files, which are stored in <XLT>/results by default
• Generated load test Reports, which are stored in <XLT>/reports by default
• Modified XLT properties files from <XLT>/config
• License file license.xml from <XLT>/config

After creating a backup of all these files and directories you can download the latest version from the Xceptance website and install XLT as described in the previous section of this chapter. You can have multiple XLT installations in parallel as the name of the unpacked installation folder includes the version number by default.

Now you can copy all the backed up files and directories mentioned above to the corresponding place in the new XLT installation directory.

New available test suite settings are provided in the default.properties file of the demo test suite testsuite-pebble that is part of the XLT installation. Copy this default.properties file from <LatestXLTversion>/samples/testsuite-pebble/config to the config directory of your test suites <YourTestSuite>/config.

In order to install the latest version of XLT Script Developer just open the latest .xpi file with the Firefox File | Open file menu. The file is located at <LatestXLTversion>/tools.

If you work with Java-based test cases, you have to add the updated XLT libraries to the Java build path of the Eclipse project. See the section Importing the Pebble Test Suite into Eclipse for a detailed description of the necessary steps.

When configuring your test project to use a newer version of XLT, do not forget to update XLT on your load machines as well. The version you have used to develop your test scripts must match the execution version of your load test environment.

Uninstalling XLT

Before uninstalling, make sure that you have backed up all the test results and test reports that you want to keep. To uninstall XLT simply delete its installation directory. If you installed the Script Developer Firefox extension, use the Firefox Add-On dialog to remove the extension.

Script Developer

Introduction

XLT Script Developer, which is a Firefox extension, is used to create test case scripts. Script test cases are based on a simple syntax and a reduced set of operations, which makes them a perfect fit for non-programmers. Besides the Script Developer, no other tool is necessary to create, edit, and manage basic script test cases.

Script Developer is used to record test cases. That means the tester simply uses the application under test. All interactions with the application are recorded in the background and stored to an XML script file. While recording, you can also add commands to perform validations on the page. Any recorded value can later be extracted out of the script into a test data file to separate test data from script code. Scripts may also be exported as ordinary Java code.

Via the Script Developer, script files can be replayed in Firefox at any time to quickly check whether the test case still runs successfully.

Before describing how to configure and use the Script Developer, we have to give a brief introduction into the basic concepts.

Basic concepts

Selenium is widely used for test automation of web applications. To ease the start with Xceptance Load Test and the Script Developer, similar basic concepts are used. In this section we describe this concepts to provide a basic understanding. In the following, if not mentioned elsewhere, the terms Action, Assertion, Command and Element identificator refer to the basic concepts described here.

An Action is an atomic collection of Commands in a test. More correctly, it can be split in sub parts but it can be seen as a mathematically closed part of the test. See also here because the concept of an Action is the same in the XLT framework and the Script Developer. However, be aware that the semantic of the term Action completely differs from that used by Selenium.

An Assertion verifies that a condition still holds. So an assertion contains always a formulation of an expected value which is compared with the actual value. If both values are different such that the condition is violated, the test aborts with an error.

A Command in the Script Developer is the same as in Selenium. It is a single statement in the test. So a command can be something that causes the test to sleep for some time or that simulates a click on an element or that waits for an element to become visible and so on.

An Element Locator or Element Identification Strategy is the approach used to identify an element on a page. While in simple examples different strategies lead to the same result causing the same effort, each of the different strategies has its own advantages and disadvantages in the power it provides and the convenience with which it can be used.

These concepts will be used in the next sections, especially those about the editing of tests. An overview of the commands complete with examples is provided in the Appendix.

Settings

Before you start, make sure the Script Developer Firefox extension is installed correctly. Open the Script Developer window, either via the Firefox Tools menu or via clicking the XLT icon in the status bar.

Next you need to change some basic settings. In the Script Developer window, open the configuration dialog by clicking on XLT Script Developer|Settings in the upper left and configure the settings appropriately.

Script Developer Settings

Java Code Settings

Generate JUnit wrapper class for test cases: Activates the generation of JUnit wrapper classes needed to run the recorded scripts in your Java IDE. This box has to be checked if you plan to run load tests using recorded test cases or if you want to automate these tests, e.g. in a build process.

Source Directory Name: The name of the directory where the generated Java source code will be saved (src in most cases). The path is relative to location of the current test suite.

File Encoding: The character encoding scheme for the JUnit wrapper classes. This encoding should match the settings in your IDE when running the test cases as JUnit tests.

Recording Settings

Element Identification Strategies: Use the check boxes to select element identification strategies available during the recording of a test case. Each web element type has a default identification strategy. If the related checkbox has not been selected in the settings dialog, then one of the other available identification strategies will be used to identify the element. For example, if you deselect the Name checkbox, then the recorder will avoid command targets like name=xyz and also avoid the name attribute in XPath expressions. Please note that XPath is used as a fall-back strategy and cannot be disabled.

Attribute Filtering: Include Patterns and Exclude Patterns provide a possibility to filter the attributes used in element locators. If an Include Pattern is defined, then only those attributes that match this filter pattern are used in element locators. If the attributes do not match the Include Pattern, then the attributes will be avoided. On the other hand, an attribute will be avoided in element locators if it matches one of the Exclude Patterns. Exclude Patterns always take precedence over Include Patterns , which means that the attribute will definitely be avoided if it matches the Exclude Pattern, even if it also matches any Include Pattern. Include Pattern and Exclude Pattern are available for ID, Name and Class attributes and must be given as regular expressions. Several Include Patterns and Exclude Patterns per attribute can be defined, separated by whitespace.

Replay settings

Command Timeout (in sec.): Defines the default timeout for replaying WaitForXyz commands. When this time has elapsed and the condition is still false, the command will fail with an error message. This value might be changed for particular scripts by using the setTimeout command.

Editor settings

Display line numbers: If checked, line numbers are shown in front of a command when a test case or module is open for editing.

User Interface Elements

The main window of XLT Script Developer features the following sections and elements.

Script Developer

Toolbar and Record/Replay Section

Project view

The project view contains a tree view, the script explorer, that lists all available test cases and modules structured in packages. XLT Script developer loads all test cases and modules from <test-suite>/scripts and its sub-folders. The package structure of the script explorer is reflected by the structure of the sub-folders.

The package structure in the script explorer also reflects the package structure of the generated JUnit wrapper classes.

Furthermore, there is a Create/Import button and a drop-down box. The drop-down box contains all test suites known to the Script Developer which allows easy switching between projects. With the Create/Import button test projects that are already existent but unknown to the Script Developer yet, can be imported.

When importing a project you have to choose the parent directory of the scripts directory, not the script directory itself. To create a new project just choose an already existing directory from the file dialog which pops up after you clicked the Create/Import button.

Below the script explorer you can find an input field to filter the displayed list of test cases or modules. By clicking on the small arrow icon on the left, you can filter the list either by name, description, tag, or any combination of these. Typed characters are case insensitive. By clicking the x icon on the right side of the input field or by deleting all characters, you can reset the search result.

It is possible to change the alphabetic sorting (A-Z or Z-A) of the packages by clicking on the header of the Name column. The second column shows a description for each test case or module. You can add or edit the description in the Edit Details dialog.

Editor Tabs

By double-clicking on a script (or via context menu option Edit) you can open a test case or module for editing. The script’s commands will be listed inside the work area on the right-hand side of the developer window. When clicking Edit Details in the context menu you can edit the test case’s / module’s description and so on but the test case/module is also opened in the editor (to make users recognize that something has been changed). It is possible to open more than one script at a time. You can select the active test case or module by clicking on the related tab above the command list. The script editor tab displays the name, target and value attributes of each command in three separate columns, whose width can be adjusted.

Recording and Replaying Test Cases

Recording

The easiest way to create test scripts is to record the steps a user takes to interact with a web application. To do this:

1. Open the web page you want to start with in a Firefox tab and make sure this tab remains the active tab, i.e. the foreground tab.
2. Switch to the Script Developer and create a new test case via the context menu in the script explorer. Provide a meaningful name. As a result, an empty script editor tab opens.
3. Click the Start Recording icon in the tool bar to start recording.
4. Switch back to your web page and start using it. All your interactions with the page will be recorded.
5. Once you are done with the test scenario, switch back to the Script Developer and stop recording by clicking the Stop icon.
6. Do not forget to save the new script by clicking the Save icon or using the Ctrl+S shortcut. A list of all available shortcuts is provided in the Appendix.

If you have the Firefox addon Firebug installed, then please make sure that Firebug isn’t active during recording since Firebug’s highlight mechanism may cause unnecessary interactions to be recorded.

Note that beside recording the normal web page flow you might also validate the correctness of pages by using assert commands. When interacting with a web page typically a new or changed page is displayed which should be checked.

To record assertions:

1. Open the Firefox context menu and choose XLT Script Developer (available only while recording). A sub-menu opens.
2. Choose an appropriate assertion from the sub-menu. (See the Appendix for a complete list of all available assertions.)
3. Continue interacting with the web page as defined by your test scenario.

Record commands using the context menu

For every assertion there is also a variant that checks whether the respective condition is not true. You can record as many assertions as you like.

Replaying a Test Case

Once you’ve recorded a test case, it can be executed inside the browser. In the script explorer, right-click the test case to open its context menu and click the Run menu item or:

1. open the respective test case script in a test case editor tab or activate the tab if it is already open and
2. click the Play icon in the tool bar.

The script will be replayed in the current Firefox browser tab.

While the script runs, you can watch the results of the script execution. You will see how links are clicked, input fields are filled, buttons and checkboxes are clicked, etc. The element on the page the script is currently dealing with is highlighted (yellow for actions and orange for validations) to make it easier for you to follow the script execution. In the script tab, the commands are marked with a special status icon. As long as the command (or module) is being executed, the status icon is displayed in yellow. Once the command is finished, the icon turns either green in case of success, or red in case of errors. Note that script execution stops immediately if a command fails.

Sometimes it is difficult to follow the script execution. In this case, use the speed slider in the tool bar to reduce the replay speed. On the other hand, if you do not want to follow the interactions and need the result as quickly as possible, increase the replay speed. Note that the actual commands are not executed slower or faster. The slider influences only the time elements are highlighted.

You can stop or pause script execution at any time. To stop a script, just click the Stop icon in the tool bar. As a result, the script is terminated. The situation is different when you click the Pause icon. In this case, script execution is suspended (after finishing the currently running command). To continue script execution, click the Continue Replay icon. Alternatively, you may click the Single Step Forward icon which causes only the next command to be executed.

You can set a start point and one or more breakpoints for a test script. See the sections Start Point for Replaying Test Cases and Breakpoints for more details.

Note that clicks on alert, confirmation or prompt boxes will not be recorded. When replaying test cases no alert, prompt or confirmation box will pop up. XLT Script Developer always goes on with replaying the test case, simulating a return value of the alert() or confirm() function which is equivalent to clicking “OK” or “Yes”. For prompt() an empty string will be returned as the user’s input value.

Running Batch Tests

It is possible to run a sequence of test cases or an entire test suite as a batch test. For this you have to select the desired test cases from the script explorer by clicking them while holding the CTRL key. Alternatively you can select one or more packages to run. Now choose Run as Batch Test from the script explorer context menu to start the batch test.

All selected test cases will be executed consecutively and the result is shown in a new tab in the work area. Each test case has a colored bullet showing the current state of the test execution. Grey is for not tested, yellow for currently running test cases, green for passed and red for failed test cases. The total number of selected test cases, the number of already executed test cases, an error count and the elapsed time is visible in the header of the batch test tab.

Batch Test

After the batch test is completed the Rerun failed tests button may become active if at least one test failed. Clicking this button starts a new batch test in a new tab with all the test cases that failed in the previous batch test. The result of a batch test can also be exported to HTML. All you have to do is to press the Export as HTML button and provide a name and target location for the exported file. You can then view the batch execution report with any web browser.

Batch Test HTML Report

Editing Test Suite, Test Cases and Modules

The script test cases and modules can be modified at any time. Often enough, it is necessary to fix mistakes made during the recording of the test scenario, for example when you clicked the wrong link or forgot to add an assertion. In the first case, just go back and continue with your test scenario. The unwanted or incorrect steps can be deleted later on. In the second case, add the appropriate assertion commands manually after recording has finished. You can even modify a test case while recording. Switch to the Script Developer window, make the necessary changes, and return to the web page and continue recording.

Editing Test Suite

Test suite context menu

The following options for editing the test suite and scripts (test cases or modules) are accessible via the context menu in the script explorer:

Editing Details

When selecting Edit Details in the context menu, a dialog opens that allows the user to edit details of the selected test case or module.

Edit Test Case Details

Edit Module Details

Depending on whether the selected element for editing its details is a test case or a module the dialog elements Enable test case specific settings, Generate JUnit wrapper class for test case and Parameters are shown or not shown.

Editing Scripts

After opening a script (test case or module) as a tab in a work area you can edit this script by selecting one of the options from the context menu (right click). A script may consist of actions, commands, and module calls.

The following options for editing a script are available (item = action, command or module call):

Editing Commands

To edit a command select Edit from the context menu and the XLT – Edit Command dialog will appear.

Edit command

Start Point for Replaying Test Cases

By default, the execution of a script will start at the beginning. However, sometimes it might come in handy to start the script execution from a specific command. To start a script at a specific position, select the respective command and press S to set a start point or choose Set Start Point from the context menu. A start-point marker that looks like the replay icon appears next to the command.

When running the script with the Play icon in the toolbar, the execution will now start from the marked command.

Make sure that the right page is displayed in the active Firefox tab, so that the script can be started at the selected command. Otherwise the script is likely to fail.

Breakpoints

In order to automatically pause the script execution at a certain command, set a breakpoint before running the script by selecting the respective command and press B afterwards. A breakpoint marker appears next to the command. When the script is replayed, execution is paused automatically when the command has been reached. You can go on with replaying the script by clicking Continue Replay or Single Step Forward in the tool bar. To clear the breakpoint, mark the command again and press B. A breakpoint may also be set or deleted via context menu or by double-clicking the breakpoint column (the grey leftmost column). When setting a breakpoint for a command inside a module that is used multiple times, the respective command in each module call will be marked with a breakpoint and therefore the replay is paused each time one of these module calls is reached.

Breakpoints exist in memory only. Closing the appropriate script editor tab or reloading the script will clear all of its breakpoints.

Actions

As you may have noticed, the Script Developer inserts Actions automatically while recording. An action is a sequence of steps that belong together. For example, filling in the inputs of a form, submitting the form by clicking the submit button, and checking the resulting page with assertions is typically one action. Actions are primarily used to break the page flow down into atomic steps and to give those steps a name. Action execution times are measured and reported while running load tests.

The Script Developer gives actions generic names, but you can rename them later on to facilitate script maintenance. You can also insert a new action manually at any position of your script (Insert > Action). The start of a new action will automatically end the previous action, also if the following action is part of a module call.

Please note that actions are not to be used to structure your scripts visually since they are not comments. In other words, you should not use actions to structure a long list of validations without loading a new page. Using actions in this context conflicts with the basic concept of an action, namely that it should always load one new page. The problem is that the misuse of actions in script test cases makes it difficult to analyze the test results since the actions do not represent the page flow. Furthermore, remember to give your actions a meaningful name (e.g. Search, Browse, or AddToCart).

Script Modules

Extracting a Module

Like test cases, script modules are sequences of commands, actions and optional calls to other modules. They can be written from scratch, but it is much easier to extract them from an existing test case. That means you record your test script first, and afterwards you identify parts of a script which are reusable. These parts are then factored out into separate modules.

The following steps are involved in extracting a module:

1. Open the original test case.
2. Select the groups of commands you want to extract by clicking them while holding CTRL key.
3. Choose Extract Module from the context menu.
4. Provide a meaningful module name and the script package.
5. Optionally provide tags, a description, and module parameters.

After confirming with the OK button you will find the new module in the script explorer. Additionally, the original test case will have been modified so that the selected items are replaced by a call to the new module.

Most test-case editing options are available for modules as well, such as edit module properties and edit commands of a module. Therefore the section Edit Test Suite and Editing Commands of this document applies to modules, too. You can open a module in a separate tab and edit the commands of the module there. But it is also possible to edit a modules’s command from within a test case that is using the module. When you do this keep in mind that editing module’s command may also affect other test cases that use the same module.

The script explorer context menu has two options for creating new modules. New Script Module can be used to create a new script module from scratch as an alternative to extracting it from a test case. New Java Module creates a script interface to integrate Java code into a script test case for the purpose of running the test case outside Script Developer later but will be skipped when creating the test case using Script Developer. For more information, see the section Creating Java modules.

Defining Module Parameters

In order to increase the re-usability of modules in other contexts and scenarios, a module can be parameterized. For example, a module that logs in a user should be parameterized with the user name and password. Script module parameters are defined as part of the module’s meta data. To use these parameters later in the module’s script code as target or value, you refer to them by using the special @{} placeholder notation (for example: @{userName} and @{password}) in the target and/or value of a command.

In order to parameterize a module execute the following steps:

1. Select the module from the library and open the Edit Details dialog.
2. Declare the module parameters. You can add parameters with the + button and delete parameters with - (see figure “Edit Module Details” below).
3. After confirming the changes in module details, open the module in an editor tab in the work area.
4. Edit the commands that shall use any of the defined parameters. Where needed, replace literal values in the target or value of the command with @{name} placeholders (see figure “Refer to module parameters in a command” below).
5. Save the module.

Edit Module Details

Refer to module parameters in a command

For all test cases using this module, you have to provide a value for each placeholder.

1. Open the test case in an editor tab.
2. Open the Edit module dialog by selecting Edit from the context menu of the module call inside the test case.
3. Provide a valid value for each placeholder.
4. Close the Edit module dialog and save the test case.

Provide Values for Module Parameters

Modules and Actions

It is possible to extract a module from any part of the script, but there are some basic rules you should follow when extracting modules, especially when it comes to actions. Actions are important when you plan to run load tests because load-test reports are based on actions. Always keep in mind the basic concepts of XLT test cases.

In XLT Script Developer an action starts with an action line in the script and ends with the beginning of the next action, regardless if the next action is part of a module or not.

In most cases, modules are used more than once in a test suite. Be careful of actions that are contained in modules. If a module starts with a sequence of commands followed by an action, then the commands of the module might be part of an action of a completely different module. This will not cause your test scripts to break, but it can cause confusion when analyzing load-test reports.

Bad style: Module call in script test case with hidden action

Bad style: Hidden action in a module call

You can follow some simple rules when extracting modules to achieve a better style and test reports that are much easier to analyze.

• Modules should either contain no actions at all or start with an action if they contain one or more actions.
• If a module contains one or more actions then the first line after the module call should also be an action.

In other words, for each module you should decide if you:

• use a module as a reusable sequence of commands that will always be part of one enclosing action or
• encapsulate one or more complete actions as a module.

Good style : Module calls in script test case

Good style: Modules with no action at all or completely encapsulated actions

Managing Test Data

By default, test data is hard-coded into the test script. In order to simplify changing test data, the data values can be bound to variables and all occurrences can be replaced by references to the appropriate variables. We refer to these bounded test data values as test data mappings since a test data variable is mapped to a certain value. All test data mappings of a script are managed using a separate data file which is named per convention as <name>_data.xml, where <name> is the name of the corresponding script. For example, the test data file of the script TAuthor is named TAuthor_data.xml and stored next to the script file TAuthor.xml.

To manage test data for a script test case or module:

1. Select the script in the explorer.
2. Choose Manage Test Data from the context menu of the script explorer’s tree view, or alternatively press ALT+D. A dialog will come up where you can add new variable mappings, edit existing ones or remove those you don’t want to use anymore. Of course, you can use this dialog when you just want to know what test data mappings exist and whether they override any default value.
3. Close the dialog. The data file will be automatically saved.

Manage Test Data

When you open the dialog the first time it shows an empty map where you can add the mappings you want to use. Each mapping is represented as a 3-column row where the first column specifies the name of the test data variable and the third column specifies the value of the variable. The second column is a read-only field that shows the default value of the test data variable (global test data mapping) if one is set.

Each of the defined test data variables can be referenced several times in the scripts by using the ${...} syntax. For example, ${foobar} references the variable foobar.

Once you have a reference to a test data variable and you rename the variable in the Manage Test Data dialog, the reference is automatically adjusted to reflect the change(s) you’ve made.

Test data mappings can not only be specified for test case scripts, but also for modules. If a module has a test data file and this module is called by a test case that also has a test data file, both test data files are used to access the required values. If both test data files contain a test data mapping with the same name but different values, then the test case data file has priority.

Test data variables are scoped. That means, that a script cannot access test variables defined by a called script. Furthermore, the calling script may overwrite test data variables defined by the called script. This forms a scope chain which is defined by the call hierarchy of your scripts. Furthermore, the scope of global test data variables always forms the end of this scope chain.

Global Test Data

Certain test data (such as user credentials or product data) are often the same for many or all test cases in a test suite. However, if the data needs to be changed, the test data mappings of all affected test cases would need to be adapted one-by-one in the same way. In order to minimize the effort of maintaining such data, you may also define it as global test data.

Each of the global test data mappings is available to all of your scripts in the current test suite, and their values serve as default values that might be overridden by a script if necessary. This way, you can also specify a test data fallback since a global test data value is only used when no script-specific test data value could be found.

To edit the global test data, click XLT Script Developer > Manage Global Test Data. This opens a dialog which is nearly the same as the previously shown Manage Test Data dialog except that each test data mapping is represented as a 2-column row instead of a 3-column one.

All global test data is stored in a file named global_testdata.properties located in the root directory of the respective test suite.

Exporting Test Cases to Java

Scripting test cases using Script Developer is easy, but limited. The scripts are strictly linear and the set of available commands cannot always replace a programming language. In certain cases, it is useful to switch to other programming languages such as Java. Java is a powerful tool that provides the structure to reuse code, as well as a wide range of available libraries. Typically, you will need to switch to Java if your tests need to act randomly, for example if you want the TBrowseCatalog test case to not always open the same catalog, but an available randomly chosen catalog instead. This behavior is especially useful for load tests which shall simulate realistic traffic.

The setting Generate JUnit wrapper class for test cases only allows for the generation of Java wrapper classes to execute the XML scripts from outside Script Developer. This is not a real export.

In contrast, Export translates the test-script file into Java syntax and generates one or more classes which represent the test case and modules. When exporting to Java you have the choice to select the API used for the resulting code. It is possible to generate code based on the XLT Scripting API or code based on the XLT Action API.

To export a script test case or module to Java follow these steps:

1. Select the script(s) you want to export from the script explorer.
2. Select Export from the content menu. The XLT – Script Export dialog will open.
3. Define the source directory, package prefix, and API (or accept the suggested values).
4. Click OK.

As a result, you will find the generated Java code in the specified packages.

Export Script to Java

• Source Directory: The name of the directory where the generated Java source code will be saved (src in most cases). The path is relative to the location of the current test suite.
• Package prefix: The package prefix for the generated Java classes. The resulting package is composed of this prefix and the script package of the exported test case.
• API: The API for the resulting Java code. XLT Scripting API or XLT Action API.

XLT Action API does not support Java modules. When exporting to XLT Action API possible calls to Java modules are omitted in the resulting code and just a comment is inserted instead.

Export to XLT Scripting API

This example shows the TAuthor test case of the demo test suite exported to XLT Scripting API code:

/**
 * Tests the creation of new blog articles.
 */
public class TAuthor extends AbstractWebDriverScriptTestCase
{

    /**
     * Constructor.
     */
    public TAuthor()
    {
        super(new XltDriver(true), "http://localhost:8080/");
    }


    /**
     * Executes the test.
     */
    @Test
    public void test() throws Throwable
    {

        //
        // ~~~ OpenStartPage ~~~
        //
        startAction("OpenStartPage");
        deleteCookie("JSESSIONID");
        open("/pebble/");
        assertTitle("My blog");
        assertText("id=blogName", "My blog");
        final Login _login = new Login(this);
        _login.execute("username", "password");

        final CreateNewArticle _createNewArticle = new CreateNewArticle(this);
        _createNewArticle.execute(resolve("${title}"), resolve("${subtitle}"), resolve("${excerpt}"), resolve("${body}"));

        _createNewArticle.execute("Another title text", "Another subtitle text", "Another excerpt text.", "Another body text.");

        final Logout _logout = new Logout(this);
        _logout.execute();


    }
}

This is the code for the module CreateNewArticle, which is called by TAuthor:

/**
 * A complete flow to create a new blog article.
 */
public class CreateNewArticle extends AbstractWebDriverScriptModule
{

    /**
     * Constructor.
     *
     * @param caller
     *      the calling script test case
     */
    public CreateNewArticle(final AbstractWebDriverScriptTestCase caller)
    {
        super(caller);
    }


    /**
     * Constructor.
     *
     * @param caller
     *      the calling module
     */
    public CreateNewArticle(final AbstractWebDriverScriptModule caller)
    {
        super(caller);
    }


    /**
     * Executes this module.
     * 
     * @param title
     * @param subtitle
     * @param excerpt
     * @param body
     */
    public void execute(final String title, final String subtitle, final String excerpt, final String body)
    {

        //
        // ~~~ NewBlogEntry ~~~
        //
        startAction("NewBlogEntry");
        clickAndWait("link=New blog entry");

        //
        // ~~~ WriteArticle ~~~
        //
        startAction("WriteArticle");
        type("name=title", title);
        type("name=subtitle", subtitle);
        type("name=excerpt", excerpt);
        type("name=body", body);
        clickAndWait("name=submit value=Preview");

        //
        // ~~~ SaveArticle ~~~
        //
        startAction("SaveArticle");
        clickAndWait("name=submit value=Save");

        //
        // ~~~ PublishArticle ~~~
        //
        startAction("PublishArticle");
        clickAndWait("name=submit value=Publish");

        //
        // ~~~ ConfirmPublishArticle ~~~
        //
        startAction("ConfirmPublishArticle");
        clickAndWait("name=submit");
        assertText("xpath=id('content')/div[@class='contentItem published']/h1/a", title);
        assertText("xpath=id('content')/div[@class='contentItem published']/h2", subtitle);
        assertText("xpath=id('content')/div[@class='contentItem published']/div[@class='contentItemBody']", body + "*");

    }
}

As you can see, the XLT Scripting API Java code is very similar to the script code. You can refactor the code as desired and use lower-level APIs directly, such as the WebDriver API, to implement more advanced functionality. You can also add code to randomize your tests or to retrieve test data from other sources, such as the GeneralDataProvider. See XLT Scripting API for details.

Export to XLT Action API

When exporting to XLT Action API the resulting code is based on separate classes for each action of a test case. Also, the script modules will be translated into separate classes. If a script module contains more then one action it will be represented by a flow.

To make this approach work without any problems it is very important that users follow the rules for defining actions in combination with modules. If there is an action in a module and it is not the first item of the module, then a automatically generated action will be inserted at the beginning of the exported module code.

Executing Tests Outside Script Developer

Once your test suite is complete, you can run the test cases outside Script Developer, in headless mode. This is especially useful for functional tests or load tests. In this case, the XLT framework is responsible for interpreting the script test cases and sending the respective requests to the system under test.

Note that it is always recommended to have a JUnit wrapper class for each script test case. This makes working with test scripts in your preferred IDE much easier, because these tools typically know how to work with JUnit classes.

Inside an IDE

To check whether your scripts are running successfully in headless mode, open the test project in your favorite IDE, navigate to the JUnit test classes, choose one of them, and run it as JUnit test.

Within Build Scripts

To run all or selected script test cases as part of your functional tests you need to make the corresponding JUnit classes available to your test framework, such as a junit Ant task.

Typically, it is sufficient to add the script test case wrapper classes to the list of classes to be run by JUnit. If you prefer to not have a wrapper class for each test script, you need to add the class com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite to the list of JUnit classes. This class is a generic representative for a set of test scripts. To tell the suite class which scripts are to be executed, use the property com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite.testCases and list the script names:

com.xceptance.xlt.api.engine.scripting.ScriptTestCaseSuite.testCases = TAuthor TVisitor

In any case, each test script will be executed once and the results will be part of the JUnit test report.

See the demo test suite in directory <xlt>/samples/testsuite-pebble for an example of how to configure Ant’s junit task to run script test cases.

As a Load Test

Performing load tests with your script test cases is quite simple. Register the JUnit classes with the XLT load-test framework and use them in your load tests as you would do with regular Java-based tests. Note that in this case, you need a JUnit wrapper around your test script. See the section Load Testing for more details.

Advanced Topics

File Generation

The figure below shows the different types of files generated by Script Developer when wrapper class generation is enabled. The files generated during export to Java are not shown.

XLT Script Developer File Generation

We will use the sample test case TAuthor (available in the demo application testsuite-pebble) as a reference to describe all of these types in detail.

Test Script (TAuthor.xml)

All the script files recorded by Script Developer are saved in an XML format following the naming convention Testcasename.xml. With reference to our sample test case, TAuthor.xml is the test-case script file which consists of all the commands and modules calls being used. The syntax is very simple and easy to understand. See below for an example:

<?xml version="1.0" encoding="UTF-8"?>
<test-case baseURL="http://localhost:8080/" xmlns="http://xlt.xceptance.com/xlt-script">
  <description>Tests the creation of new blog articles.</description>
  <action name="OpenStartPage"/>
  <command name="deleteCookie" target="JSESSIONID"/>
  <command name="open" value="/pebble/"/>
  <command name="assertTitle" value="My blog"/>
  <command name="assertText" target="id=blogName" value="My blog"/>
  <module name="Login">
    <parameter name="userName" value="username"/>
    <parameter name="password" value="password"/>
  </module>
  <module name="CreateNewArticle">
    <parameter name="title" value="${title}"/>
    <parameter name="subtitle" value="${subtitle}"/>
    <parameter name="excerpt" value="${excerpt}"/>
    <parameter name="body" value="${body}"/>
  </module>
  <module name="CreateNewArticle">
    <parameter name="title" value="Another title text"/>
    <parameter name="subtitle" value="Another subtitle text"/>
    <parameter name="excerpt" value="Another excerpt text."/>
    <parameter name="body" value="Another body text."/>
  </module>
  <module name="Logout"/>
</test-case>

Test Data (TAuthor_data.xml)

Separating test data from script code by extracting the recorded values into data files is a very useful feature. The naming convention being used is Testcasename_data.xml, so TAuthor_data.xml represents the data file of the TAuthor test case. A data file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<data xmlns="http://xlt.xceptance.com/xlt-script-data">
  <title>The title text</title>
  <subtitle>The subtitle text</subtitle>
  <excerpt>The excerpt text.</excerpt>
  <body>The body text.</body>
</data>
 

JUnit Test Case Wrapper (TAuthor.java)

This file is generated when you enable wrapper class generation for your test cases. This feature allows you to run your script files outside the browser via the XLT framework, which simulates a headless browser. This mode is suitable for unattended test case execution, such as functional or load tests.

Custom Module (ComplexUserLoggedInCheck.java)

If some special constructs cannot be expressed due to the basic script syntax, you can overcome that by creating your own custom modules and using them inside your scripts. Custom modules are implemented in Java. See Using Java Modules for an example.

Macros

When creating and running regression tests using the Script Developer, it is sometimes necessary to have unique data available to be able to run a test over and over again. For example, user account creation always needs a new email address since the target system accepts each address only once.

For the purpose of generating timestamps and random strings or numbers, the following macros are available:

• NOW: returns the current time as time stamp (number of milliseconds elapsed since 1970-01-01)
• RANDOM.String(length): returns a random string of the given length
• RANDOM.Number(max): returns an randomly chosen integer between 0 (inclusive) and max (exclusive)
• RANDOM.Number(min,max): returns an randomly chosen integer between min (inclusive) and max (inclusive)

Macros can be used in any command using the ${variable} notation. If you want to input a random or unique email address, you could use one of the following code lines as a value in the type command:

${RANDOM.String(5)}@anyserver.com

or

${NOW}@anyserver.com

The resulting email addresses might look like: zghfu@anyserver.com or 1295519733483@anyserver.com

If you use one of these macros as a value for a module parameter, the random string or time stamp is created once for a test run and then can be used in several commands of the module with an identical value. For example, you can fill in a form with a random name, submit the form, and then validate the name on the confirmation page. This would only work if filling in the form, submitting it and the validation is all part of the same module. The name has to be defined as a module parameter and referenced in the relevant commands with @{name}, where ${RANDOM.String(8)} is the provided value for the module parameter name.

Creating Java Modules

The commands supported by the Script Developer are sufficient to create working test scripts. However, some special constructs cannot be expressed due to the restricted script syntax, for example:

• Combining assertions by a logical OR.
• Performing advanced assertions which are not available as commands yet.
• Accessing and validating downloaded files.

To overcome such limitations, custom modules can be created for test cases that are meant to run outside Script Developer (as a load test, as JUnit test in Eclipse, or integrated in a build process). The Java modules will be skipped when running the test case in the Script Developer.

Custom modules are written in Java by implementing the WebDriverCustomModule interface. This interface forces you to implement the required execute method. See below for an example:

public class ComplexUserLoggedInCheck implements WebDriverCustomModule
{
    public void execute(final WebDriver webDriver, final String... parameters)
    {
        final WebElement webElement = webDriver.findElement(By.xpath("id('sidebar')/div[1]/div[1]/span"));
        final String userName = parameters[0];

        Assert.assertTrue("Expected user name not found: " + userName, webElement.getText().contains(userName));
    }
}

Make sure the custom module class is compiled and made available on your test suite’s class path.

In order to be able to integrate Java custom modules in test scripts, you first need to register them with Script Developer. You do so by creating a new Java module script:

1. Choose New | Java Module from the script explorer context menu. The Edit Module Details dialog will come up (see the figure below).
2. Fill out the necessary fields as described below.
3. Close the dialog.

The new Java module is now available in the script explorer and can be used in other scripts (test cases or modules) the same way as any other script module. Please note, that Java modules cannot be edited and will be skipped when running the test case inside Script Developer.

New Java Module

• Name: The name of the new Java module.
• Script package: The script package for the new Java module.
• Full Class Name: The full class name (including package) of the Java class implementing the WebDriverCustomModule interface.
• Tags: Tags to group modules and make them easier to find. Tags have to be separated by a comma. The list of available modules can be filtered by tags.
• Description: A description of the module.
• Parameters: The name of the parameters that the module expects when being executed.

If you defined module parameters you have to provide a value for each parameter when using the Java module in a test case. You access the first of these parameters in your Java code by reading the value from parameters[0], the second one by reading the value from parameters[1], and so on. The name of the parameters defined in the Script Developer Java Module script is not visible in the Java code; only the order of the parameters is relevant.

Note that Java modules should be used only if absolutely necessary, as their execution will be skipped when running the test case in Script Developer. If Script Developer comes across a Java module while replaying a script, it simply ignores it (the status icon turns gray) and continues with the next command. The next command might or might not succeed depending on what the module does with the page. If the module leaves the page unchanged (i.e. it performs advanced validations only), the rest of the script will run successfully. However, if the module changes the page or loads a new page, the commands following the module are likely to fail. This is why Java modules are mainly used for the purpose of complex validations.

Please tell us more about extensions you create, as we might include them in a future release.

Assertions on Form Controls

When doing assertions on form controls it is very helpful to know, when to choose which kind of assertion. In detail, for a given form control you have to decide whether one of the *Value commands or one of the *Text commands has to be used.

The following table shows for each affected form control what exactly will be used as input when checking the element using either a *Value or a *Text assertion. The input can either be the element’s text content T or the element’s value attribute V.

XLT Framework

Basic Concepts

When performing web-based application tests the possible paths from page to page define the web page flow. The web page flow can be represented as a directed graph where the vertices are the web pages and the transitions represent the actions to get from one page to another. Typically, a test scenario covers a certain part of the page flow only, such as a specific path through the application.

XLT provides a programming paradigm that uses a three-level architecture (transactions, actions, and requests). These levels will be described in the following subsections.

Transaction

A transaction is an execution of exactly one test case or test scenario. In order to perform the scenario, the page flow is modeled in code. The test scenario is implemented as a test case, which itself executes a sequence of one or more actions.

Action

An action can be defined as one irreducible step within a test case. So an action interacts with the current page and – as a result – loads the next page. That page is associated with this action and becomes the current page for the next action in the test scenario. Generally, an action triggers one or more requests.

Request

This level is equivalent to the HTTP request level used in web browsers or in any other application that relies on HTTP communication. You do not have to deal with requests directly because they are automatically generated by the underlying HtmlUnit framework when performing actions on HTML elements.

Validation

Because we are testing the functionality of applications or pieces of software, we have to check the correctness of all responses. We strongly recommend that you handle all potential situations and use validations as often as possible. It is better to have too many checks rather than too few! They can’t do any harm, and will increase your confidence that your application works correctly. You should insert as much validation as necessary to detect any abnormal application behavior of the software being tested.

Pre-validation

Each action should have a pre-validation section that checks whether or not all of the required data is available to interact with that page and possible advance to the next one. From the end user’s point of view, we simply look for the information on the page that we need to continue our web experience, such as a form to fill in or a link to click. In case the required information cannot be found, an exception is thrown. If we run a load test, XLT will catch this exception and log all relevant information. This allows you to evaluate the results after running the test and narrow down error conditions.

Post-validation

Post-validations work similarly to pre-validations. They are used to validate the result of the interaction and ensure that the data matches the expectations.

Example

The following example shows a very simple scenario to understand the terminology better. It is based on the demo application that is shipped with XLT. Imagine a typical user visiting a blog. The user will possibly:

• Open the blog’s home page
• Browse the list of articles
• Select items to read
• Add a comment
• Go back to the home page

Test Case And Actions

In this example the test scenario is modeled as test case TBlogVisitor. A single execution of this test case is a single transaction. OpenHomePage, Paging, ViewArticleDetails, AddComment and ReturnToHomePage are the actions of this page flow to go from one page to the next. Validations after each page transition ensure that we arrived on the right page with the right content.

Available Programming Interfaces

Overview

XLT provides different approaches for writing test cases in Java. Several programming interfaces are available when using the XLT Framework. Built on each other, they represent different abstraction levels.

You can extend your test suite by using the XLT Scripting API. This allows you to start with Java code that was automatically generated from the recorded Script Developer test cases. This features a high-level command scripting API with a very intuitive syntax. XLT Scripting API is built on top of the WebDriver API, which can also be accessed directly to write more advanced tests.

XltDriver, as part of the XLT framework, is a WebDriver implementation extending the HtmlUnitDriver. Both in turn are built on the HtmlUnit API. HtmlUnit is a headless browser that offers a low-level API that allows for full control when creating web tests.

To serve as the main framework when creating HtmlUnit based tests, XLT provides the XLT Action API which structures the code in action classes and test-case classes.

XLT Framework API

XLT Test Cases are JUnit4 Tests

All XLT test cases use a Java test case class with one test() method, regardless of the chosen approach for writing tests, except for the pure Script Developer script test cases.

The test() method of each test case class has a @Test annotation (see TBlogVisitor code example line 12). XLT builds upon JUnit4 principles and its annotations to implement and tag test cases. This way each XLT test is in fact a JUnit test, so XLT tests can be executed just like any other unit test in the IDE or within an existing build process. The only difference between XLT and standard JUnit4 tests is that XLT tests can only have one active test method per test class. That means that although there can be an arbitrary number of methods within a class, only one method is permitted to be annotated with @Test. This limitation simplifies things and does not result in any actual limitations in real life.

Implementing the test case as a JUnit4 test also allows you to use standard JUnit assertion to validate the page, mainly when creating test cases by using the HtmlUnit API.

XLT Scripting API

When exporting a script developer test case to Java (see Export test cases to Java) the resulting code can be based on the XLT scripting API (depending on the API selection during export). This API is an easy-to-use programming interface with a simple syntax based on the commands available in the XLT Script Developer.

Test cases can be written from scratch in Java using the XLT Scripting API. However, the most common way is to record a test case with Script Developer and extend the test case using the Scripting API after exporting it to Java.

The following screenshot shows how the test case TBlogVisitor introduced in the section above may look like if it would have been recorded using Script Developer. Note that this test case uses a very poor validation to keep the example short and simple. A real test should have more validations to be sure that the page is displayed correctly.

TBlogVisitor Demo Script Test Case

The automatically generated Java code of the TBlogVisitor test case after the export is shown below:

/**
 * Tests viewing articles and writing comments.
 */
public class TBlogVisitor extends AbstractWebDriverScriptTestCase
{

    /**
     * Constructor.
     */
    public TBlogVisitor()
    {
        super(new XltDriver(true), "http://localhost:8080");
    }


    /**
     * Executes the test.
     */
    @Test
    public void test() throws Throwable
    {

        //
        // ~~~ OpenHomePage ~~~
        //
        startAction("OpenHomePage");
        open("/pebble/");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");

        //
        // ~~~ ViewArticleDetails ~~~
        //
        startAction("ViewArticleDetails");
        clickAndWait("//div[@id='content']/div[1]/h1/a");
        assertElementPresent("link=Send a TrackBack");
        assertElementPresent("//div[@id='content']/div/div[3]/a[1]");

        //
        // ~~~ AddComment ~~~
        //
        startAction("AddComment");
        click("//div[@id='content']/div/div[3]/a[1]");
        type("id=commentBody", "A comment");
        clickAndWait("//*[@id='commentForm']//input[@name='submit' and @type='submit' and @value='Add Comment']");
        assertElementPresent("//*[@id='content']//p");
        clickAndWait("name=submit value=Confirm");
        assertElementPresent("//*[@id='content']//div[contains(@class,'contentItemBody')]");

        //
        // ~~~ ReturnToHomePage ~~~
        //
        startAction("ReturnToHomePage");
        clickAndWait("link=Home");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

    }
}

You can see that the code is structured in blocks representing the actions. Each action starts with an startAction() command. The rest of the commands are very similar to the commands available in Script Developer. Each Script Developer command has an appropriate counterpart in the XLT scripting API. The generated test-case class extends the abstract class AbstractWebDriverScriptTestCase and inherits the methods that represent the scripting commands to interact with the page and perform validations.

See package com.xceptance.xlt.api.engine.scripting for more information about the available scripting commands of that API.

Script Modules

The Paging action in TBlogVisitor is implemented twice, which is not very efficient. Script Developer allows the user to extract parts of the script as a module. This would be a great idea for the Paging action.

When exporting the script to Java, XLT generates a class for each of the modules which extends AbstractWebDriverScriptModule and extracts the code to reuse it in the test case. Using the execute() method, the module can be called in the test case after creating an instance of the module class.

Modules can embed sub-modules. That means modules can be called by test cases or by other modules. This is why the module class needs two different constructors, one that takes a test case and one that takes a module as parameter.

This is the code for the extracted module class that can now easily be reused in any test case:

public class Paging extends AbstractWebDriverScriptModule
{
    //Constructor
    public Paging(final AbstractWebDriverScriptTestCase caller)
    {
        super(caller);
    }

    //Constructor
    public Paging(final AbstractWebDriverScriptModule caller)
    {
        super(caller);
    }

    /**
     * Executes this module.
     * 
     */
    public void execute()
    {
        //
        // ~~~ Paging ~~~
        //
        startAction("Paging");
        clickAndWait("link=<< Previous");
        assertElementPresent("link=<< Previous");
        assertElementPresent("link=Next >>");
    }
}

The resulting test case code instantiates the Paging module class and calls the execute() method twice to perform the paging actions:

        //
        // ~~~ Paging ~~~
        //
        final Paging _paging = new Paging(this);
        _paging.execute();
        _paging.execute();

WebDriver API

WebDriver is a tool for automated tests of web applications. It was originally introduced by Google. WebDriver features a simple and efficient API. The API permits control of real web browsers, such as Firefox, Internet Explorer, Safari, and the HtmlUnit headless browser.

The XLT framework integrates the WebDriver API, so external test cases using the WebDriver API can generally also run in the XLT framework. XLT-based WebDriver tests cannot be executed with a stand-alone WebDriver, because the XLT-WebDriver API integrates the concept of action names.

The Java code generated by XLT when exporting from the Script Developer is based on the XLT scripting API which is built on WebDriver. When you look at the TBlogVisitor example, you may notice that the strictly linear approach can show some limitations. Using the lower-level WebDriver API in combination with the XLT scripting API is a good way to overcome these limitations.

When running the TBlogVisitor example as functional test or load test the level of realism is low because the comments are always added to the same article. It would be nice to randomize the Paging action to browse to different pages and also to select the article randomly.

The exported code uses a XltDriver to simulate users. The WebDriver API also allows you to easily switch to other WebDrivers to simulate real-world browsers like Chrome, FirefoxDriver or Internet Explorer. The XltDriver is an extension of the HtmlUnitDriver implementation of the WebDriver API.

The following code shows an example using the WebDriver API to introduce random factors. To separate the different approaches in this example, the WebDriver functionality is re-factored to a method which also allows it to be reused for random paging and random article selection. It gives an example of a JUnit4 assertion and shows how you can use standard Java functionality like for loops to execute the paging several times.

The re-factored code described above could look like this:

/**
 * Tests viewing articles and adding a comment
 */
public class TBlogVisitor extends AbstractWebDriverScriptTestCase
{
    /**
     * Constructor.
     */
    public TBlogVisitor()
    {
        super(new XltDriver(true), "http://localhost:8080");
    }

    /**
    * This method randomly picks one of the web elements that match the given xpath
    * It uses the WebDriver API
    */ 
    private WebElement findWebElementsAndPickOne(String xpath)
    {
        // get all elements that match the given expression
        final List<WebElement> articleLinks = getWebDriver().findElements(By.xpath(xpath));

        //Make sure there is at least one element; This is a pure JUnit assertion
        Assert.assertFalse("No elements found", articleLinks.isEmpty());

        // grab one of the elements randomly
        WebElement element = articleLinks.get(XltRandom.nextInt(articleLinks.size()));

        //return the element
        return element;	
    }

    @Test
    public void test() throws Throwable
    {
        //
        // ~~~ OpenHomePage ~~~
        //
        startAction("OpenHomePage");
        open("/pebble/");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");

        //
        // ~~~ Paging ~~~
        //
        //Loop the paging twice by using the extracted method implemented using the WebDriver API
        for (int i = 0; i < 2; i++)
        {
            startAction("Paging");
            clickAndWait("link=<< Previous");
            //Find the paging links by xpath, pick one randomly and click the link
            findWebElementsAndPickOne("id('linearNavigation')/a[. != 'Home']").click();
        }

        //
        // ~~~ ViewArticleDetails ~~~
        //
        startAction("ViewArticleDetails");
        clickAndWait("link=Another title text");
        //Find all article headlines by xpath, pick one of the articles randomly and click the link
        findWebElementsAndPickOne("id('content')/div[@class='contentItem published']/h1/a").click();
        assertElementPresent("link=Send a TrackBack");
        assertElementPresent("link=Add a comment");

        //
        // ~~~ AddComment ~~~
        //
        startAction("AddComment");
        //click("link=Add a comment");
        click("xpath=id('content')/x:div/x:div[2]/x:div/x:div/x:a[1]");
        type("id=commentBody", "A comment");
        clickAndWait("name=submit value=Add Comment");
        assertTextPresent("Please click the button to confirm.");
        clickAndWait("name=submit");

        //
        // ~~~ ReturnToHomePage ~~~
        //
        startAction("ReturnToHomePage");
        clickAndWait("link=Home");
        assertTitle("My blog");
        assertElementPresent("link=<< Previous");
        assertNotElementPresent("link=Next >>");
    }
}

The full documentation of the XLT framework with more information about the WebDriver support can be found in the file <XLT>/doc/apidoc.zip.

XLT Action API

In addition to the components already described, the XLT framework supports one more approach for modeling test scenarios in Java code. The test cases and action classes can be implemented in Java from scratch or they can be generated automatically during an export from XLT Script Developer. This approach is based on special action classes, so it is called the XLT Action API.

The foundation of XLT Action API for testing web-based applications is the HtmlUnit framework, which also includes the Mozilla Rhino engine for JavaScript support. Web tests are performed by a low-level web browser simulation. Basically this means that – as in real browsers – a web page is translated to a DOM (Document Object Model) tree. Analysis, validation, and any other access is performed afterward on the constructed DOM tree.

XLT Action API provides a programming paradigm to translate a test scenario into a unit test. The test scenario is implemented as a test-case class, which itself executes a sequence of one or more actions.

Test Case and Action Classes

All test-case classes should inherit the abstract class AbstractTestCase which supplies some basic features like logging the test results to disk and easy access to properties.

Because a test case models a transaction and transactions rely on actions, defining the appropriate actions is the first step.

All actions must inherit the abstract class AbstractAction which forces you to implement the three methods execute(), preValidate() and postValidate(). As mentioned earlier, the preValidate() and postValidate() methods are used to perform validations before and after the execution of that action itself. Therefore, the call sequence of an action generated by the XLT framework will always be:

1. preValidate()
2. execute()
3. postValidate()

This call sequence will be executed exactly once when the instance method run() of an action is called.

As you see, XLT Action API forces us to implement the validation methods and that is the whole purpose of testing: validating data. Therefore, implementing the abstract validation methods in a non-trivial way (i.e. not leaving them empty) is strongly recommended. Otherwise, you will sacrifice test quality.

Each of the three methods may throw an exception which is always an indication of a problem. To check if an action can safely be executed, the abstract class AbstractAction provides a method called preValidateSafe(). This method internally calls preValidate() and catches any thrown exception. If no exception is thrown, preValidateSafe() returns true; otherwise it returns false. This helps you to determine if the prerequisites are fulfilled to continue the page flow in a certain direction. A simple example for that is the flow through a catalog with nested categories. Because you do not know the nesting level up-front when you create a dynamic and random test execution, it might be necessary to call preValidateSafe() before trying to go to the next level of categories.

Note that AbstractAction does not offer any web support. Therefore, any web-based test should inherit the abstract class AbstractHtmlPageAction which is a specialization of AbstractAction and offers support for web testing.

Validation

Assertion

JUnit provides the concept of assertions and we use this concept for all validations. Because XLT does not change JUnit in any way, you can use assertions just as you learned to do with JUnit.

Pre-validation

XLT offers two ways of using the preValidate() method. Any exception on the direct path will stop the test with an error message. In case we just want to check whether or not a requirement is fulfilled, we can call the preValidate in a safe way (by using preValidateSafe()), so that any exception will be caught and no error is reported. Should we accidentally cause any Java exception different to AssertionException, such as NullPointerException or IndexOutOfBoundException, XLT will issue a warning, because the code might contain a problem from a programming point of view. Errors from the application being tested should always come up as assertion failures.

Post-validation

The postValidate() method works similarly to the preValidate() method. It is used to validate the page just loaded in execute() and ensures that the data matches the expectations. The full set of JUnit assertions is available.

The postValidate() method cannot be called explicitly, the framework does it. Additionally, error messages cannot be suppressed. If a page has different outcomes based on random data or states, you have to handle that explicitly in your validation code.

Validators

We strongly encourage you to write individual validation classes for easy reuse. If a certain check has to be done more than once, it becomes a candidate for a validator implementation. This simplifies the maintenance of the tests and makes them less error-prone, because copy-paste is one source of typical programming errors.

Some common validation routines are already covered by default validators, such as a HTTP response code, HTML end tag, and HTTP content length validation. See package com.xceptance.xlt.api.validators in the API documentation for more information about this topic.

Example

To explain the XLT Action API with an example, let’s imagine a web search engine. The most important action would be to “search”: to fill in the search phrase and click “Go”, “Search”, or something similar to load a list of results. The preconditions are the existence of the search input field and of an appropriate button labeled Search or Go. The execute() method should fill in the search phrase and click the button.

After the new page has been loaded, the result should be validated. This validation consists of general validation, performed by validators, and action-specific validation.

The resulting implementation of the search action would look like this:

public class Search extends AbstractHtmlPageAction
{
    // XPath to search input field
    private static final String searchInputFieldPath = "/id('search')/input";

    // XPath to search button
    private static final String searchButtonPath = "/input[@type='submit' and @name= 'Search']";

    // XPath to search result block
    private static final String searchResultPath = "/div[contains(text(), 'result')]";

    // Search input field HTML element
    private HtmlInput searchField;

    // Search button HTML element
    private HtmlInput searchButton;

    // search phrase
    private String searchPhrase;

    /**
     * Performs a new search.
     * @param prevAction
     *            Previous action.
     * @param searchPhrase
     *            Phrase to search for.
     */
    public Search(AbstractHtmlPageAction prevAction, String searchPhrase)
    {
        super(prevAction);
        this.searchPhrase = searchPhrase;
    }

    /**
     * Validation prior to execution.
     * @throws Exception
     *             if some of the required input elements couldn't be found.
     */
    public void preValidate() throws Exception
    {
        HtmlPage p = getPreviousAction().getHtmlPage();

        List result = p.getByXPath(searchInputFieldPath);
        Assert.assertEquals(1, result.size());

        searchField = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchField);

        result = p.getByXPath(searchButtonPath);
        Assert.assertEquals(1, result.size());

        searchButton = (HtmlInput) result.get(0);
        Assert.assertNotNull(searchButton);
    }

    /**
     * Executes the search. Primarily this includes the input of the search
     * phrase and a click on the proper search button.
     * @throws Exception
     *             if some of the inputs have become invalid or setting the
     *             value attribute of the search input field has failed.
     */
    public void execute() throws Exception
    {
        // Fill in search phrase
        searchField.setAttribute("value", this.searchPhrase);
        // Click on 'Search'
        loadPageByClick(searchButton);
    }

    /**
     * Validation after search has become complete.
     * @throws Exception
     *             if no search result block element could be found or the
     *             search result block element doesn't contain the search
     *             phrase.
     */
    public void postValidate() throws Exception
    {
        HtmlPage p = getHtmlPage();

        // response code = 200?
        HttpResponseCodeValidator.getInstance().validate(p);
        // does the length match?
        ContentLengthValidator.getInstance().validate(p);
        // does the page end with </html>?
        HtmlEndTagValidator.getInstance().validate(p);

        List result = p.getByXPath(this.searchResultPath);
        Assert.assertEquals(1, result.size());

        HtmlElement el = (HtmlElement) result.get(0);
        Assert.assertNotNull(el);

        Assert.assertTrue(el.asText().contains(this.searchPhrase));
    }
}

Notice that the constructor of this class has two parameters. One of them is the search phrase that the action has to know about. The other parameter is the action performed previously. All actions that will be used in page flows have to provide a constructor with a parameter representing the previously performed action to enable a flow. Without passing the previous action, each action would be stand-alone and behave as if you had just opened a new web browser. Normally only the start action does that.

You’ll notice that the postValidate() method uses some of the predefined validators. XLT also offers a StandardValidator which performs the most common validations in one go. This includes:

• HTTP response code validation,
• HTML end tag validation,
• content length validation, and
• XHTML validation.

Having the search action at hand, the implementation of a test case using this action is almost done. A very simple test case would be to repeatedly search for some phrases. These phrases can be stored in a data file and obtained using the XLT data provider mechanism.

public class TSearch extends AbstractTestCase
{
    // Container that holds all the search phrases
    private static CustomDataProvider phrases = null;

    @Before
    public void initialize() throws Exception
    {
        // Data container already initialized?
        if(phrases != null) return;
        // No. Go for it.
        phrases = new CustomDataProvider(
                        getProperty("searchphrases.filename", "phrases.txt"),
                        CustomDataProvider.DEFAULT);
    }

    @Test
    public void search() throws Throwable
    {
        // Start on Homepage.
        Startpage start = new Startpage();
        start.run();

        for (int i = 0; i < XltRandom.nextInt(10); i++)
        {
            // Take a random search phrase.
            String searchPhrase = phrases.getRandomRow(false);

            // Search.
            Search search = new Search(start,searchPhrase);
            search.run();
        }
    }
}

The example above also demonstrates the usage of the XltRandom class which offers some nice features regarding randomization. Note that the package com.xceptance.xlt.api.util provides more functionalities that can help implementing tests.

Each execution of the search action requires a proper search phrase which is obtained from a CustomDataProvider object. This class provides a generic mechanism to handle and provide test data that is stored in a text file. The name of the text file, along with a Boolean value that tells the appropriate parser whether whitespace should be removed, are passed as parameters to the constructor. When the class is instantiated, all data is kept in memory, allowing easy and fast access. XLT is shipped with a predefined set of data files that contain email addresses, first and last names, street and city names, and so on. This data can be obtained from the GeneralDataProvider class which uses the appropriate text files located in directory <testsuite>/config/data where <testsuite> refers to your test project directory.

The example also demonstrates the use of JUnit4 annotations which can be used in the standard manner.

Data-Driven Tests

Sometimes a certain test case should be executed not just once, but multiple times, each time with a different set of test data. For example, to check not only the “happy path”, but also some border cases, you might want to specify multiple test data sets, which are automatically recognized by the test framework and used to execute the test case once per specified data set. The test case executions are independent from each other and each produces a separate result in the test report. This concept is also known as data-driven test.

XLT supports data-driven tests for any kind of test case (plain Java and scripts), but only when running them as (a part of) a functional test using a JUnit test runner. Data-driven tests are neither supported when replaying script test cases in the Script Developer, nor can they be used during load tests. For load tests, there are other parameters that define how long/often a test will be executed, such as measurement time and arrival rate.

Typically, test data can be classified as constant or variable test data. Constant test data is fixed for all runs of a test case in a data-driven test. Constant test data is either hard-coded into the test case or kept separate from the test code in a data file. Variable test data is different for each run of the test case, so variable test data is organized as a list of separate data sets. Each data set contains all variable test data needed for exactly one test run. The number of data sets determines the number of test runs.

But where do these data sets come from? For a data-driven test, XLT retrieves test data sets from an additional source. This could be, for example, another data file or a database. XLT accesses a data source via data set providers, which implement a uniform interface and return a data set as a simple key/value map.

During test execution, the framework reads the next data set from the configured data set provider and passes it to the test case instance and runs the test. The test instance is responsible to apply the test data appropriately. This process is repeated until all available variable data sets have been processed.

Data Set Providers

Built-In Providers

XLT supports three common sources for test data sets out of the box, which will be discussed in this section.

XML Files

Data sets can be stored in an XML data file with a three-level element structure. There is a single top-level element (as XML mandates it). The elements on 2nd level define the data sets, while the elements on 3rd level define the values. A data file with two sets of user data might look like this:

<?xml version="1.0" encoding="utf-8"?>
<data-sets>
    <data-set>
        <userName>fred</userName>
        <password>topsecret</password>
    </data-set>
    <data-set>
        <userName>wilma</userName>
        <password>cantremember</password>
    </data-set>
</data-sets>

Note that you can name the root element (here: data-sets) and the 2nd-level elements (here: data-set) as you like since the structure is important only. However, the tag names on the 3rd level always define the parameter names, so these tag names must be used consistently across all data sets.

CSV Files

The data sets can also be stored as CSV data files, organized as lines of separated values. The values in the first line define the parameter names while the values of all following lines define the values of each data set. The previous example would look like this:

userName,password
fred,topsecret
wilma,cantremember

The separator used in the CSV file (typically comma or semicolon) can be configured using an XLT property. For example:

## Sets the field separator character for CSV files (defaults to ",").
com.xceptance.xlt.data.dataSetProviders.csv.separator = ; 

Note that there is no way to specify character encoding information in a CSV file. By default, XLT reads CSV files using UTF-8. To override this default use the following property:

com.xceptance.xlt.data.dataSetProviders.csv.encoding = ISO-8859-1

JDBC Data Sources

For JDBC data sources, there is also a “data file”, but this time the file does not specify the data sets directly, but contains an SQL query, which retrieves the data sets when executed. A query file for user data sets could look like this:

select login as "userName", password as "password" from users;

Each row returned from the database is converted to one data set. The alias names in the query define the resulting parameter names.

Note that the SQL data set provider needs some additional setup before it can be used. First, an appropriate JDBC driver needs to be present on the class path of your test suite. Simply copy the respective JAR file to <testsuite>/lib and you are done. Second, you need to configure the URL and credentials of your JDBC database connection that should be used when executing the query:

com.xceptance.xlt.data.dataSetProviders.jdbc.url = jdbc:h2:tcp://localhost/test
com.xceptance.xlt.data.dataSetProviders.jdbc.userName = sa
com.xceptance.xlt.data.dataSetProviders.jdbc.password = YourPassword

Custom Data Set Providers

If the built-in data set providers are not sufficient, you can write your own. Your custom data set provider must implement the general DataSetProvider interface:

public interface DataSetProvider
{
    public List<Map<String, String>> getAllDataSets(File dataFile) throws DataSetProviderException;
}

To register your own data provider implementation with XLT, you just need to add a property to your configuration:

com.xceptance.xlt.data.dataSetProviders.foo = com.yourcompany.FooDataSetProvider

This will tell XLT to use the class com.yourcompany.FooDataSetProvider for data files with the extension .foo. Note that this way you can also override the built-in providers.

Test Data Set File Lookup

As you have learned in the previous section, there is always some kind of “data” file involved. Typically, these data files are named after the test case (the script name for script-based test cases, or the simple class name for Java-based test cases). For example, if a test script is named TAuthor, then the XLT framework will automatically look for files like TAuthor_datasets.<ext>, where <ext> is one of the file extensions for which a data set provider has been registered. So, the resulting list of file name candidates is: TAuthor_datasets.csv, TAuthor_datasets.xml, and TAuthor_datasets.sql (and TAuthor_datasets.foo respectively).

The lookup order of data files is defined as:

1. .csv
2. .xml
3. .sql
4. (.foo)

However, test data files can also have arbitrary names or paths. In this case, you have to configure which data file belongs to which test case. You do so by mapping the data set file name (or path) to the test case’s Java class using the following notation:

<class_name>.dataSetFile = <data_set_file_path>

For example:

com.mycompany.xlt.tests.MyTest1.dataSetsFile = Test1.xml
com.mycompany.xlt.tests.MyTest2.dataSetsFile = ./subdir/Test2.csv
com.mycompany.xlt.tests.MyTest3.dataSetsFile = c:/tmp/Test3.sql

Please keep in mind, that all property files are Java-style property files. So, when you use backslashes on Windows, you have to quote it with another backslash, e.g. c:\\tmp\\Test3.sql.

Whether or not the test data file name is given explicitly or has been derived from the test case name, the XLT framework looks for such a file in several locations:

1. in the current directory (typically <testsuite>)
2. in the directory specified by the property com.xceptance.xlt.data.dataSets.dir [This property is set to ./config/data), but commented out by default. Remove the hash character when you want to use this location.]
3. in the default script test case directory <testsuite>/scripts (for script-based tests only)
4. in the class path (for Java-based tests only)

As soon as a suitable file has been found, the lookup is stopped.

Note that if the test case has a qualified name, i.e. it has a package part, the data set file also needs to have that package to be found. So, if there is a test case named your.package.TAuthor (be it a script or a Java test case), the framework will search the aforementioned directories for a corresponding data set file using the file path <dir>/your/package/TAuthor_datasets.<ext> or, alternatively, using the file name <dir>/your.package.TAuthor_datasets.<ext>.

Executing Data-Driven Tests via JUnit

Running data-driven tests via JUnit is not different in any way from running normal test cases. You simply add the test cases in question (directly or indirectly) to the list of classes to be run by JUnit. If XLT finds a data set file for a certain test case, it will be passed to the right data set provider, which returns all data sets. The test case will then be executed once for each data set. This works, because all test cases extend the class AbstractTestCase, and this class has all the magic built-in. The same is true for the generic script test case suite class ScriptTestCaseSuite.

See the demo test suite in directory <xlt>/samples/testsuite-pebble for an example how to configure Ant’s junit task to run XLT test cases.

If you want to temporarily disable a data-driven test and let the test cases run only once even though there are data set files, just configure the following setting in default.properties:

com.xceptance.xlt.data.dataDrivenTests.enabled = false

But note that your test case must provide some default test data in this case.

Demo Application and Test Suite

XLT ships with a real-world demo web application (called “Pebble”) as the system under test and a test suite to test this application. Both can be found in the directory <XLT>/samples.

Running Pebble

Pebble is a blog software written in Java. Because Pebble is small and easy to deploy, it is a perfect fit for our test application. Thanks to Simon Brown for this excellent piece of software and licensing it as Open Source.

Please do not use the provided installation for purposes other than testing, because it has been modified to run without problems from the sample directory. Feel free to download the latest version.

To start the demo application, use the following command:

cd <XLT>/samples/app-server/bin
./start.sh

Windows users have to use the appropriate .cmd file, which is located in the same directory, by entering start.cmd in your command prompt.

This will start an application server which contains the Pebble application. To access Pebble, simply open a browser with this URL: http://localhost:8080/pebble/. Please take some time to become familiar with Pebble.

The Pebble Test Suite

XLT supports creating test cases using different approaches. Basically, test cases can be written directly in Java using your favorite IDE or they can be recorded as simple scripts in the Script Developer. A test suite can contain test cases created with either approach, and so does the Pebble test suite. While the number of test scenarios is limited, the sample project demonstrates the differences when using the two approaches, and – for Java-based test cases – what APIs are available to implement them.

The Directory Structure of the Test Suite

An XLT test project has a simple directory structure. The following directories have to exist to make everything run smoothly:

• <project>/classes contains the compiled code of your project. Normally your favorite development environment will do the job and place the files there. Optionally, you can build a JAR and place it in the <project>/lib directory.

• <project>/config contains all the properties files used to configure the project.

• There can be an optional <project>/config/data directory, where you can place any data file you need for the test, such as address data, logins and so on. All files will be uploaded to the agent before a load test takes place. The programming API provides a way to access this data easily.

• You can place all your required libraries in <project>/lib. The content will be uploaded to the runtime agent and included in the class path. For your local development within an IDE, you have to add the libraries manually to the class path of your project.

• The <project>/src directory holds the Java-based test cases of your project. This code will be compiled into classes by your IDE or build environment. The code is organized in main packages, typically one for test cases, one for the actions, and one for utility classes. Make sure that the compiled classes end up under project/classes, because this is the directory that XLT configures as class path for your test.

• The <project>/scripts directory contains test scenarios in the form of script test cases. Again, the code is organized in main packages, one for test cases and one for reusable modules.

Understanding the Test Scenarios

As Pebble is a blog software, the test scenarios cover the typical use cases of a blog:

• Author: The author creates the content for the blog by creating new articles, previewing them to validate them, and finally publishing them.
• Visitor: A visitor visits the blog start page, reads the first two or three articles in detail and posts some comments.

Note that both scenarios share some common steps, allowing us to demonstrate the re-use of code across test cases.

Running Script Test Cases in Firefox

Script test cases can be replayed directly in Firefox via the Script Developer, so we have to import the test project into the Script Developer first.

Importing the Pebble Test Suite into Script Developer

1. Open the Script Developer.
2. Click the folder-like toolbar button to open a file system explorer window.
3. Navigate to <XLT>/samples, mark testsuite-pebble, and click Select Folder.

In the tree view on the left-hand side, you should see all available script test cases and modules now.

Executing Script Test Cases in Script Developer

Open the XLT Script Developer. All script test cases and modules are visible in the script explorer. Double-click the TAuthor test case to open it and see the list of its commands and called modules inside the work area. Click the Play icon to start replaying the script inside Firefox. Alternatively, you can right-click on the test case in the tree view and choose Run from the context menu.

Running Java Test Cases in Headless Mode

Java-based test cases are executed in headless mode, i.e. in a simulated browser which does not perform page rendering. Via Java wrapper classes, also script test cases can be run in this mode. But before we can do so, we have to import the test suite into your favorite Java IDE.

Importing the Pebble Test Suite into Eclipse

After starting Eclipse and creating a workspace, if you have not already done so, you have to import the sample test suite.

1. Open the import dialog (File > Import > General > Existing Project Into Workspace).
2. Select the root directory to search in and point to <XLT>/samples.
3. Select the test suite project testsuite-pebble from the list.
4. Click Finish and you are done.

Because the imported project has dependencies to the libraries of XLT, you have to adjust these dependencies.

1. Right-click on your project and select Properties.
2. Click Java Build Path.
3. Select the Libraries tab and click Add External JARs.
4. Go to <XLT>/lib and select all JARs. Then click Open.
5. A list of all these JARs should be visible now. Close the dialog with OK.

Eclipse will rebuild the project now and should not report any build problems if configured properly.

Users of other IDEs have to carry out similar steps.

Executing Java Test Cases in Eclipse

Any Java test case can be run directly in Eclipse using the headless-browser mode. Go to package com.xceptance.xlt.samples.tests, select the test case class (e.g. TAuthor) and run it as JUnit test via the Eclipse class file context menu.

Writing Web Tests

The following sections will guide you through the essential steps when you want to write the first web tests on your own using XLT. These sections provide useful information that helps getting you started and helps to make essential the decisions regarding your test suite.

We encourage you to study the demo test suite carefully with respect to the approach as well as the code. Since all web tests are similar in structure, you will soon get a feeling for how to write your own tests. The sample test suite can serve as a template for your own test projects.

Creating a New Test Suite

In the XLT installation directory, you will find the demo test project “testsuite-pebble” which is mentioned above. To avoid creating the mandatory directory structure and configuration files from scratch, just copy this existing test project and strip it.

Copy the complete testsuite-pebble directory to a location of your choice and rename it, e.g. c:\test-suites\testsuite-MySite. We will refer to this directory as <testsuite> in the next sections.

It is recommended to store your test suite in a directory that is not a sub-directory of your XLT installation. This makes updating to newer XLT versions much easier as it reduces the risk of overwriting your test cases during an update.

This is the required directory structure of XLT test suites. It cannot be changed or altered.

--+ <testsuite>
  |--- classes
  |--+ config
  |  `--- data
  |--- lib
  |--- results
  |--- scripts
  `--- src

Now it’s time to get rid of the leftovers from the testsuite-pebble project. You can delete the complete source code of the demo tests as well as the classes by emptying the following directories:

• <testsuite>/classes
• <testsuite>/scripts
• <testsuite>/src
• <testsuite>/results

You should by no means delete the configuration files in <testsuite>/config as well as the files in the <testsuite> root. These files are mandatory to run the XLT test cases and must stay in these directories. Some configurations in these files must be changed to fit your project, especially when you plan to run load tests. You can delete the pebble specific properties here. You will find details for all of the properties and settings as comments in the files itself.

Before importing your project into Eclipse you have to edit the name element in the XML file <yourProject>/.project.

<?xml version="1.0" encoding="UTF-8"?>
<projectDescription>
  <name>testsuite-YourSite</name>
  <>..<>
  <>..<>
</projectDescription>

Choosing a Suitable Approach

As described in the XLT Framework section of this document there are a number of different approaches available when you start writing your own test cases. The first step is to choose the suitable approach and API. To make the right decision you should answer yourself a couple of questions regarding your web test project:

• What is the goal of your test suite?
  • Pure functional test suite
  • Functional test suite with some basic load testing
  • Load test project
  • Time critical load test project with heavy load
• How good are your programming skills?
  • Non-programmer
  • Basic programming skills
  • Good Java programming skills
• Do you have to simulate different browsers in a functional test suite?
• What are the available resources to generate enough load during a load test?
• Do you need random factors in your test cases and do you have to handle conditions?

Stand-Alone Script Developer Test Suite

Starting to record your tests using the XLT Script Developer and possibly continuing with just the Script developer and no Java code at all is a good option if...

• you need a functional test suite you want to run from time to time,
• load testing is an interesting option for the future,
• you have no or just basic Java programming skills,
• it is sufficient to use Firefox and there is no need to simulate other browsers,
• you are OK with exactly repeating test cases and you don’t have to handle conditions,
• you don’t need any advanced random factors or data-driven tests.

The test cases can be recorded and manually replayed in the XLT Script Developer. There is no need to write any Java code and there is no need to use any tool or IDE different from the Script Developer.

Executing Plain Script Test Cases Outside Script Developer

When you need to run the tests outside Script Developer, e.g. in Eclipse or during an build process, then XLT Script Developer is the best starting point. You just have to enable generating Java wrapper classes for the script test cases. You can also run a load test with these automatically generated wrapper classes as long as you’re satisfied with the limitations listed above. Still you don’t need to write any Java code. Everything is based on the easy to use Script Developer commands and Firefox context menu validations.

As an additional benefit you can run data-driven tests and you can also simulate different browsers when running the script test cases outside the XLT Script Developer.

Script Developer Export, XLT Scripting API and WebDriver

If you need to overcome the limitations of the script test cases by switching to Java as the programming language, then the Script Developer again is the best starting point because it’s fast and easy to create working prototypes for your test cases and then export these scripts to Java. The result is a test case class using the XLT Scripting API that can easily be expanded by using the WebDriver API. Another option is an export with resulting Java code based on XLT Action API. The TBlogVisitor test case introduced in the XLT Framework section is a good example for that approach with XLT Scripting API as export result. This approach is maybe the best choice if...

• you want to benefit from the advantages of the Script Developer,
• you want to overcome the limitations of the Script Developer,
• you have basic or good Java programming skills,
• you need to simulate different browsers for a functional test,
• performing data-driven tests is required,
• you want to handle conditions and execution branches in you test cases,
• the planned load testing profile is not causing any problems regarding resources and timing.

WebDriver from Scratch

To write your test cases you can also use the WebDriver API directly in the XLT context without using the XLT Scripting API abstraction level. This is a common approach for advanced users and this is a good choice if...

• the web test project meets the requirements listed in the section above,
• you’re happy with writing your test cases from scratch without any recording feature,
• you have good Java programming skills.

XLT Action API and HtmlUnit

Higher level APIs have a certain overhead that can influence the performance of the executed tests in comparison to a low-level API. In some situations it makes sense to skip the higher level APIs listed above and program the test directly using HtmlUnit, for example when running load tests where heavy network traffic is required. For this case the XLT Action API serves as a framework to provide support for the action concept and validations, and also helps to structure your code. You don’t even have to waive the Script Developer recording feature since exporting the script test cases to Java also allows to generate code based on the XLT Action API. The XLT Action API is the approach you should choose if...

• the goal of your test suite is a high performance load test,
• you have to handle the available resources carefully to generate enough load,
• you need full control over requests and responses on the low-level HtmlUnit API,
• it is sufficient to use a headless browser and there is no need to simulate other browsers.

If you decide to use the XLT Action API as the surrounding framework, then you will have to use the HtmlUnit API to code your tests. The other way round this is not true. You still would be able to go down to the HtmlUnit level for sections of your test cases, also if you decided to use one of the other approaches (e.g. WebDriver).

XLT Lightweight Mode

You should consider to choose the so called XLT Lightweight Mode to code your test cases if...

• the XLT Action API and HtmlUnit approach is still not sufficient to meet your requirements regarding resources consumption and test execution speed.

XLT provides the Lightweight Mode to code highest performance test cases. This maximum performance can be reached by omitting the creation of a DOM tree and JavaScript completely. Responses are available as html source and the test cases have to be coded on this level. One of the consequences is that some regular expression knowledge is required to identify elements and perform validations. Also the test cases become more complex and you will have to expect an increased programming effort.

Structuring Your Test Suite and Test Cases

Structuring in Script Developer

Naming and Tags

The easiest way to bring some structure into your test suite is to give your test cases a common naming convention. A common convention is to start the name with a capitalized T and then go on with the so called “camel-case” with each element’s initial letter capitalized within the compound word. A best-practice example is to use the name’s elements to specify the test case’s purpose, e.g. TCartCheckoutCancel, TCartOrder and so on.

XLT Script Developer provides the possibility to tag test cases and modules. Each test case can have several tags. Tags help to group several test cases and make them easier to find. The list of available test cases can be filtered by tag and name.

Script Packages

Script Developer allows you to define packages to structure your test cases and modules in the script explorer. You can use this feature to bundle test cases for similar purposes. The packages can be created with a hierarchical structure but they are always displayed in a flat representation (e.g. testcases.cart.order).

Script Modules to Structure Test Cases

To structure a single script test case you should use modules. A module is a sequence of script commands and can be re-used in several test cases and can also call other modules, which means they can be nested. Using modules will prevent you from writing or recording the same sequence of commands again and again. But it also helps to keep the test case clear and understandable as a module call can be folded to hide the contained commands and show only the module’s name.

Action Commands

Script Developer inserts so-called Actions automatically while recording. You can also insert a new action manually at any position of your test case. Even though actions look similar to comments or general structuring elements they are not to be used to structure your scripts visually. Please read the action-related notes in the Script Developer section of this document.

Java Code Structuring

If you decide to choose an approach that requires to write your test cases in Java, you will have the same possibilities to structure your code as in any other Java program or software development project. Feel free to extract sequences of your test case code to methods, create Java classes and use packages to structure your test suite.

Package Suggestions

The <testsuite>/src directory contains sub-directories with the structure of your Java packages as normal. Your source code should be organized in main packages. Typically you create one package for test cases, one for actions, one for flows, one for validators and one for utility classes.
A typical directory structure after you created your packages might look like this:

• <testsuite>/src/.../actions (only when XLT Action API is used)
• <testsuite>/src/.../flow
• <testsuite>/src/.../util
• <testsuite>/src/.../validators
• <testsuite>/src/.../tests

If you plan to use more than one of the approaches provided by XLT, it is recommended to create packages for each of its test cases. This would result in up to three additional sub directories. You can use the following names as a suggestion for your own packages:

• <testsuite>/src/.../tests/actionbased
• <testsuite>/src/.../tests/scripting
• <testsuite>/src/.../tests/webdriver

Beside these general possibilities, each of the approaches introduces XLT specific framework conditions that will give basic structure to your test suite and test cases. In particular, each test case is necessarily implemented as a Java class that extends a XLT test case class which is specific for the chosen approach and contains one method annotated with @Test. These specific framework conditions and further possibilities are explained in the following sections.

Structuring Scripting API and WebDriver Test Cases

When you export a script test case from Script Developer to Java using XLT Scripting API then it will be converted into a test case class which extends AbstractWebDriverScriptTestCase. Modules are converted to classes extending AbstractWebDriverModule. The test case class and all module classes are created automatically and you normally don’t have to deal with the creation of these classes.

If you decided to write the test cases from scratch using WebDriver API, then your test case class should extend AbstractTestCase. Like all test case classes, this class can have several methods but exactly one method has to be annotated with @Test. This class normally contains the statements and lines of code that define the basic structure of the test case, i.e. the page flow.

For both approaches, Scripting API and WebDriver API, it is recommended to structure the page flow by actions. This is important, especially if you plan to run a load test because the load test reports are designed for analysis and evaluations based on XLT actions.

Scripting API offers a very simple command to start a new action: startAction("MyNewAction"). When using pure WebDriver API, the following line of code can be used to start a new action Session.getCurrent().setWebDriverActionName("MyNewAction").

Again, actions should always be used with the basic concepts in mind. This means they should be used only to represent the page flow.

Structuring XLT Action Based Test Cases

As the name implies, test cases are closely related to actions when using the XLT Action API. Like all other approaches, an action interacts with the current page and as a result, loads the next page. That page is associated with this action and becomes the current page for the next action in the test case. But in contrast to previous approaches an action is implemented as a Java class extending AbstractHtmlPageAction. These XLT action classes can be seen as reusable building blocks to write your test case and define the page flow.

More information about how the API forces you to structure code and validations with methods you have to implement can be found in the XLT Action API section of this document.

Creating a Flow

When creating XLT test cases, sometimes you may want to reuse blocks of code that contain more than a single action. As with modules, you can create an own class with one method that combines a sequence of several XLT actions as a flow. Different test cases can call this method now to reuse the flow. This is a concept for code structuring that can be implemented if needed, but there is no explicit support available or necessary in the XLT framework when creating a flow manually.

Flows will only be created automatically when exporting script modules to XLT Action API containing more than one action.

Test Suite and Framework Configuration

When using just the Script Developer for recording/writing and replaying script test cases, the following section does not apply. But configuring the test suite and framework by changing properties files might be necessary if you write or run test cases from inside Eclipse as JUnit tests or if you want to perform load tests.

To configure the test environment and test suite XLT uses Java properties files. Therefore, the basic characteristics and syntax of that format apply also to the XLT properties files.

When reading the properties, XLT distinguishes between the load test mode and development mode. As the name implies the load test mode is active when test cases are executed by the XLT master controller/agent controller as load tests. When test cases are executed as JUnit tests in Eclipse or any other JUnit test runner, they are run in development mode. Even though development mode is used mainly for developing the test cases it is also active if your test suite’s goal is an automated functional test that will be triggered manually from time to time or is integrated in a build process.

XLT uses a hierarchical file system so that properties can be distributed to several files with different priorities. Properties from different files complement each other. Furthermore, properties from a file with higher priority can overwrite identical properties from a file with a lower priority. This mechanism allows general default values to be specialized for different test run scenarios or projects. Additionally, it is possible to prepare several configurations in different files and activate one of these configurations by switching between the files.

All properties are read from the <testsuite>/config/ directory. The existing properties files are listed below sorted by priority from lowest to highest. See the following subsections for details.

• default.properties: Default configuration of the XLT test framework
• project.properties: Configuration of your test project
• test.properties: Configuration of a specific load test profile
• dev.properties: Properties only read in development mode
• dev-log4j.properties: log4j logger settings used in development mode
• log4j.properties: log4j logger settings used in load test mode
• jvmargs.cfg: JVM setting for the agents and therefore only used in load test mode

Default Framework Configuration – default.properties

The properties in default.properties represent the general XLT framework settings which are all set to their respective default value. They are neither specific for a single test project nor for a single test run.

If you need to change one of these properties, you should copy the considered property to the project.properties or test.properties and change the value there in order to overwrite the value in default.properties. Each of the properties in default.properties can be overridden since they have the lowest priority.

When updating XLT to a newer version, the default.properties file should also be updated because newly available properties can be found there along with their default value and description.
Even though this file is not read-only it should be treated as such. You can use this file as a documentation of available XLT framework properties which is also defining the default values for these properties.

The properties listed in the default.properties file are separated into the following groups (look into the file itself for details):

• HTTP/Protocol Settings
• Browser Emulation Settings
• JavaScript Settings
• CSS Settings
• Test Data Management Settings
• Result Settings
• Test Execution Settings
• Script Engine Settings
• Miscellaneous Settings

Test Project Configuration – project.properties

The file project.properties contains project specific settings. The first and most important property is the reference to the test.properties file that should be applied (e.g. com.xceptance.xlt.testPropertiesFile = test-1.properties). By changing the value for this property you can easily switch between several different load test profiles configurations.

By default, it also contains the test case mapping that maps the test case class onto a load test name. The load test name will be referenced later in the load test configuration.

This file is also the best place for all your test case specific custom properties, e.g. URLs, credentials, search phrases or any other data you want to extract from your test cases as properties. The demo test suite pebble gives several examples how to use properties in test cases.

Load Test Profile Configuration – test.properties

All settings to configure a specific load test profile are collected in a separate file. Please see the chapter about load tests or the properties file itself for details of the available load test settings.

The default name for this file is test.properties. However, this name is variable and several files with different load test profile configurations may exist. The one file applied for a test run is referenced by a property in project.properties, which is mentioned in the section above.

Development Environment Configuration – dev.properties

The file dev.properties contains development mode settings. Use this file to modify the configuration such that it better suits your needs during development of test cases, i.e. when you create and debug the test cases from within your IDE.

This file is read in development mode only, but not during load testing. For development mode the values in this file have highest priority. Any setting defined here will overwrite the corresponding setting from the other properties files: “default.properties”, “project.properties”, and the test run specific properties file, e.g. “test.properties”.

A typical example for differing development setting is com.xceptance.xlt.loadStaticContent = true to enable loading images and other static content in development mode for debugging. Default value ( =false ) for this property is switching off loading static content to save resources during load test.

If the default values are sufficient as development settings for your test suite, the dev.properties file can also be empty.

Additional Configuration Files

In addition to the already described files there a three more files in <testsuite>/config/:

• dev-log4j.properties: log4j logger settings used in development mode
• log4j.properties: log4j logger settings used in load test mode
• jvmargs.cfg: JVM settings for the agents and therefore only used in load test mode. E.g. settings for Java garbage collector tuning.

For more information about log4j settings also see Apache Log4j API Docs

Property Replacements

In all XLT properties files you can work with property replacements based on a ${} syntax. You can define a property and then assign a value to another property by referring to the first property.

This is a helpful feature especially for project.properties where properties are often defined for each test case to gain flexibility but e.g. the login data is identical for all test cases by default.

username = MyUsername
password = MySecretPassword
com.xceptance.xlt.samples.tests.TAuthor.username = ${username}
com.xceptance.xlt.samples.tests.TAuthor.password = ${password}
com.xceptance.xlt.samples.tests.webdriver.TAuthor.username = ${username}
com.xceptance.xlt.samples.tests.webdriver.TAuthor.password = ${password} 

Integrating XLT

In order to better support test projects that use dependency management systems like Maven or Ivy, we have made a public repository available that hosts all published XLT releases together with their corresponding POM files. This makes updating the XLT version used in test projects much easier. See below for some examples how to configure your test project to use the Xceptance repository.

When configuring your test project to use a newer version of XLT, do not forget to update XLT on your load machines as well. The version you have used to develop your test scripts must match the execution version of your load test environment.

Integration into Maven

All necessary information to integrate XLT into your Maven project can be found below. Just copy and paste them into your pom.xml file.

<repositories>
    <!-- Declare Xceptance repository that hosts the XLT artifacts -->
    <repository>
        <id>xceptance-releases</id>
        <url>https://lab.xceptance.de/nexus/content/repositories/releases/</url>
    </repository>
</repositories>
...
<dependencies>
    ...
    <!-- Declare XLT dependency -->
    <dependency>
        <groupId>com.xceptance</groupId>
        <artifactId>xlt</artifactId>
        <version>4.2.3</version>
    </dependency>
    ...
</dependencies>

Integration into Ivy

The setup necessary to integrate XLT into your Ivy project consists of two steps:

1. Declare the XLT repository as Maven compatible resolver in your ivysettings.xml file.
2. Tell Ivy that your project depends on the artifact xlt of group com.xceptance. This is usually done in your ivy.xml file.

ivysettings.xml

<ivysettings>
    <settings defaultResolver="resolver-chain" />
    <resolvers>
        <chain name="resolver-chain">
            <filesystem name="local">
                <artifact pattern="${ivy.default.ivy.user.dir}/.ivy/local/[artifact]-[revision](-[classifier]).[ext]" />
            </filesystem>
            <!-- Declare Xceptance repository that hosts the XLT artifacts -->
            <ibiblio name="xceptance-releases" root="https://lab.xceptance.de/nexus/content/repositories/releases" m2compatible="true"/>
        </chain>
    </resolvers>
    <conflict-managers>
        <compatible-cm/>
    </conflict-managers>
</ivysettings>

ivy.xml

<dependencies>
    ...
    <!-- Declare XLT dependency -->
    <dependency org="com.xceptance" name="xlt" rev="4.2.3" />
    ...
</dependencies>

Load and Performance Testing

Overview

Load Models

A load model describes what basic characteristic you can influence to reach a certain load and performance behavior. XLT supports two different models:

• a user count model and
• an arrival rate model.

Both load models have different characteristics and use cases, which will be described in more detail in the next chapters.

User Count Model

The user count model is a static and non-feedback based load model.

When using the user count model the actual load is determined by the number of concurrent users. For example, when you configured a load of 10 users, XLT will run 10 threads that execute this scenario repeatedly. At any given time during the test, the target system has to handle 10 concurrent users, no more, no less. The number of executions (transactions) that can be achieved during a certain period of time directly depends on the time the target system needs to respond.

This model is suited best for:

• a simple base line test (single-user test) to assess the base performance of the system under almost no load,
• a real load or performance test to assess the performance under a high, but predictable load,
• a test that should be easily repeatable and its load factor is not influenced by the system under test.

Arrival Rate Model

The arrival rate model is a feedback based load model.

When using the arrival rate model, the target number of transactions per hour is the base for the load generation. This means for an arrival rate of 1,000 transactions per hour, XLT will run the respective scenario 1,000 times, equally distributed across a period of one hour. XLT will use as many concurrent users as necessary to fulfill the given arrival rate, but no more than specified. So if the time needed for a transaction is very short, even one user might be sufficient. On the other hand, if a transaction takes longer, more users will be used in parallel.

The number of concurrent users is not static and influenced by the response time. If the response time increases temporarily, for example due to a server-side background job, the user count may increase as well. When the response times get better again, the number of concurrent users will decrease automatically. This way, the generated load is somewhat unpredictable, at least in terms of concurrency.

This relationship between response times and concurrent users can lead to situations where more users cause more load, which, as a result, causes longer response times, that, of course, requires even more users to run. This might overload the server at the end. Even though this behavior sounds pretty aggressive, it is also more realistic. Compare that to a real world situation where a lot of people are waiting at the cash register at the end of the supermarket, but there are still people coming in, because they do not know that people are waiting already. Or transferred to the online world: When you visit an online presence, you do not know that the system behaves poorly before you started your visit or your action.

Even though the number of concurrent users is rather a result than an input value for this load model, XLT requires you to specify a user count. This number is used to impose an upper limit to the number of concurrent users. This can help you to restrict the total load on the system if you want to avoid a total overload as a result of the feedback loop.

The arrival rate load model is best used if the load test should prove that a system is indeed able to handle a certain number of transactions per hour. Since this is the primary purpose of a load and performance test, the arrival rate load model is the best choice for most of your test tasks.

Load Profiles

While the load model defines what you can modify to achieve a certain load factor, the load profiles defines how you apply this values over a period of time. XLT supports three different load profiles:

• static,
• ramp-up, and
• variable load.

The following sub-sections will discuss each profile in detail.

Static Load Profile

The load parameter remains unchanged during the test. This is the simplest profile. Note that the target systems must be able to handle the full load right from the beginning.

Ramp-Up Load Profile

When using a ramp-up load profile, the load parameter is steadily increased. This gives the target system time to warm up before the full load hits the system, for example to compile and optimize code or to fill caches.

The ramp-up behavior of the load parameter can be controlled via the following settings:

• initial value: the load parameter value to start with,
• target value: the final load parameter once the ramp-up period has finished,
• step size: the increment that is added to the load parameter after each ramp-up step,
• ramp-up period: the length of the ramp-up phase,
• steady period: the period to keep the current parameter value until the next ramp-up step occurs.

The steady period and the ramp-up period settings are mutually exclusive.

Use the steady period if you want to keep the load at a certain level for a defined time, no matter how long the total ramp-up phase will be. On the other hand use the ramp-up period if you want to finish the ramp-up process after a certain time, no matter how long the resulting steady phases will be.

If an arrival rate is defined, the ramp-up parameters are applied to the arrival rate, otherwise to the user count.

Variable Load Profile

XLT supports a variable load profile. It gives you full control and you can vary the load parameter freely during the test. This means the load may not only be constantly increased (compare ramp-up), but it can be increased and decreased at any time.

The variable profile comes in handy when you want to combine different load levels within one test run, for example a test where phases with regular load alternate with peaks of much higher load. You can also imagine a test that models the load profile of a typical 24 hour day, maybe squeezed into a shorter period of time for a faster test turn-around.

In order to define how the load parameter varies over time, you have to specify a load function. You do so by defining a sequence of time/value pairs, each defines a point in time when the slope of the load function changes. When you connect the dots by a straight line, the final shape of the load function evolves.

Multiple time/value pairs can be specified by separating them using one or more space, comma, semi-colon or tab characters. The time part can be given in all formats that are supported for time periods.

Imagine the following load function:

0/10, 60m/10, 60m/20, 70m/5

This sequence of time/value pairs defines a function that keeps the load parameter constant at 10 for an hour, doubles it to 20 in an instant, and immediately starts to decrease it over the next ten minutes to 5, keeping it at 5 for the rest of the test.

Load /\
     |
  20 +                         *
     |                         **
     |                         * *
     |                         *  * 
  10 +**************************   *
     |                              *          
   5 +                               **************************
     |
     +-------------------------+-----+--------------------------->
     0                        60    70                       Time

Note that the time/value pairs must be sorted by their time in ascending order. There can also be two pairs defined for a certain time, which is useful when the load parameter should change immediately. If no pair is given for time 0, a pair “0/1” will be inserted automatically (implicitly causing a ramp-up behavior). Finally, if the load test runs longer than the time of the last pair, the last known load parameter value is kept stable.

Load Test Phases

The execution of a test scenario during a load test can be divided into different phases: the initial delay, the warm-up period, the measurement period, and the shutdown period.

The initial delay is only required if you do not want this test scenario to run right from the beginning of the load test. This is useful if the test scenario depends on the results created by another test scenario. The initial delay is optional.

In order to minimize discrepancies that could be caused by applications and other systems starting up and not yet operating at an optimal level, a warm-up period can be defined which is the time given before any measurements are taken. The warm-up period is optional. Please keep in mind that this will create a period of time where you do not have any insights into your test. We suggest to omit a warm-up period and instead modify the report later by specifying a time filter. This ensures that you do not miss any important information.

The test will be measured during the measurement period. As the name already suggests, this is the only time period where measurements are taken. The measurement period is a required setting.

To ensure that a test scenario runs to completion even if the measurement period is over, a shutdown period can be set in which the users will continue to run (without taking measurements, though) and try to orderly finish their current iteration. This is useful in case things need to be cleaned up at the end of the test scenario. Once the last iteration is finished, the user stops automatically. If, however, the last iteration is still not finished at the end of the shutdown period, the user is forcibly terminated. The shutdown period is optional, but note that if no shutdown period is defined, the user is aborted right after the measurement period.

Typically, the ramp-up period (as defined above as part of the load profile) is put into the warm-up period to ensure that the system under test is working at an optimal level before any measurements are taken. However, this up to you. It might as well be interesting to measure the system performance during the ramp-up phase. In this case, a warm-up period must not be defined.

The following figure shows the several phases in relation to the total test time.

Load Test Phases

Load Test Environment

Typically, a distributed load generation environment is needed to generate enough load. This usually requires a cluster of test machines. XLT has to be installed on each of these machines.

Load generation environment

• Master controller: The master controller is the brain of the load test environment. It deploys the test suite to all load machines, distributes the load evenly, and starts/stops the load test. There can only be one master controller in the test cluster.
• Agent controller: Since the master controller does not have direct access to the remote load machines, it needs a counterpart on these machines which is the agent controller. The agent controller acts on behalf of the master controller.
• Agent: The agent is the component that actually executes the test suite against the system under test. The agent is started and stopped by the agent controller.

Load Test Environment Configuration

Before you can start the load test, some configuration work needs to be done. The configuration of XLT load generation environment itself is discussed in the next section, followed by an explanation of the configuration of your test suite.

The property files mentioned below are used to configure the main components of the XLT load generation runtime:

• <XLT>/config/agentcontroller.properties - Agent Controller Configuration
• <XLT>/config/mastercontroller.properties - Master Controller Configuration

Agent Controller Configuration

Inside agent controller configuration file you can define the following properties.

Port Number

Port number, the agent controller is listening on. Default is 8500. You can pick any free port number, but make sure that the corresponding master controller entry matches that number. Also ensure that the firewall rules in place allow unrestricted communication. The used protocol is HTTPS. If you would like to run more than one agent controller per machine, make sure that all controllers use different port numbers.

com.xceptance.xlt.agentcontroller.port = <portnumber>

Key Store Credentials

The credentials, your key store is encrypted with. You only have to change this if your Java key store password has been changed from the default.

com.xceptance.xlt.agentcontroller.keystore.password = <password>
com.xceptance.xlt.agentcontroller.keystore.key.password = <password> 

Agent Controller Logging

The following properties are used to configure the agent controller logging facility. These settings only affect the agent controller output and do not change the logging of your test code. Most of the time, a change here is not required.

log4j.rootLogger = info, console, file
log4j.appender.console = org.apache.log4j.ConsoleAppender
log4j.appender.console.layout = org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern = [%d{HH:mm:ss,SSS}] %-5p [%t] - %m%n

For more information about log4j settings also see Apache Log4j API Docs.

Master Controller Configuration

The second configuration file contains the properties of the master controller.

Test Suite Location

To determine which test suite should be used for the load test, you have to specify the location of the test suite, either as an absolute path or relative to your XLT installation. It will be uploaded to the agent controllers from there.

com.xceptance.xlt.mastercontroller.testSuitePath = <location>

For example:

com.xceptance.xlt.mastercontroller.testSuitePath = samples/testsuite-pebble

When running on and from Windows, make sure that you use the correct encoding for backslashes, because property file format uses back-slashes to quote other special characters. So you have to quote the backslash with an additional backslash to ensure its original meaning, e.g. c:\\test\\mysuite.

Update Interval

Defines how often the master controller prints the status of the currently running load test to the console.

com.xceptance.xlt.mastercontroller.ui.status.updateInterval = <time in seconds>

Status Display

Whether to display detailed status information for each simulated test user or not. If set to false, status information will be aggregated into one line per user type. If you have a lot of test users running, it can be helpful to set this to false, otherwise you might become overwhelmed by the amount of information presented. This property will not change the data collection, but only the final data presentation. This is a display property only.

com.xceptance.xlt.mastercontroller.ui.status.detailedList = <true/false>

Agent Controllers

This property lists the locations of the agent controllers, the master controller should use.

com.xceptance.xlt.mastercontroller.agentcontrollers.<id>.url = <url> 
com.xceptance.xlt.mastercontroller.agentcontrollers.<id>.weight = <weight> 

You can use any name for the <id> part of the property. We recommend to use name and number combinations, such as ac1 for the first agent controller or blade01-02 for the second agent controller on the first blade. Make sure that the agent controller IDs differ from each other, otherwise a later entry in the file will overwrite the previous one.

In order to use load machines of different power together in a load cluster, you may specify a “weight” for each agent controller (defaults to 1 if not set). This value influences the automatic distribution of virtual users across the load machines. A machine with a weight of 3 will get 3 times the load of a machine with a weight of 1.

com.xceptance.xlt.mastercontroller.agentcontrollers.ac1.url = https://localhost:8500
com.xceptance.xlt.mastercontroller.agentcontrollers.ac1.weight = 1
com.xceptance.xlt.mastercontroller.agentcontrollers.ac2.url = https://localhost:8501
com.xceptance.xlt.mastercontroller.agentcontrollers.ac2.weight = 3

Master Controller Logging

You can set a different logging behavior for the master controller. This can help to solve problems and also provides information in case of support inquiries.

log4j.rootLogger = debug, file
log4j.appender.console = org.apache.log4j.ConsoleAppender
log4j.appender.console.layout = org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern = [%d{HH:mm:ss,SSS}] %-5p [%t] - %m%n

Test Suite Configuration

The test suite itself is configured independently of the master controller. All properties are read from the <testsuite>/config directory. All files and most important properties are explained in detail in the Test Suite and Framework Configuration section. This section will discuss only those settings that are relevant to load testing.

Default Configuration – default.properties

Result directory location

Specifies the directory location where you want to store load test results. Normally, it is not necessary to change this.

com.xceptance.xlt.result-dir = <directory path>

Error Behavior

Specifies the framework behavior in case of an error – whether or not the framework should abort a transaction if any of the following errors occurs:

• While loading a page – If an HTTP error occurred while loading a page.
• Page resource unavailable – If an HTTP error occurred while loading a resource that is embedded in a page.
• Java script error – If a JavaScript error occurred.
• Agent termination in case of server errors – maximum number of errors allowed before an agent terminates. It helps to stop unobserved long running test cases automatically in case of severe error conditions, such as unavailability of the system under test. The number of errors specified here is the error count per running agent controller.

com.xceptance.xlt.stopTestOnHttpErrors.page = <true/false>
com.xceptance.xlt.stopTestOnHttpErrors.embedded = <true/false>
com.xceptance.xlt.stopTestOnJavaScriptErrors = <true/false>
com.xceptance.xlt.maxErrors = <number of errors per agent controller>

Think Times

To specify the think time between two subsequent actions or transactions, use the following properties. If a random think time is needed, set the appropriate deviation to a value greater than 0. It specifies the maximum deviation from think time in milliseconds. The respective value is added to or subtracted from the think time using a pseudo-random, uniform distribution.

com.xceptance.xlt.thinktime.action = <time in [ms]>
com.xceptance.xlt.thinktime.action.deviation = <time in [ms]>
com.xceptance.xlt.thinktime.transaction = <time in [ms]>
com.xceptance.xlt.thinktime.transaction.deviation = <time in [ms]>

For example, the think time configuration might look like this:

com.xceptance.xlt.thinktime.action = 100
com.xceptance.xlt.thinktime.action.deviation = 50
com.xceptance.xlt.thinktime.transaction = 0
com.xceptance.xlt.thinktime.transaction.deviation = 0

This will set the action think times between 50 and 150ms and no transaction think time at all.

The deviation has to be smaller than the specified base think time.

Test Project Configuration – project.properties

To configure your test project, you’ll have to edit the file named project.properties.

Test Properties File

XLT permits to prepare and use multiple test.properties files for easy maintenance of test setups. This makes switching between test setups easier and avoids configuration errors. This property does not allow the use of a path-specific file name. The test definition files reside in the same directory as the project.properties file.

com.xceptance.xlt.testPropertiesFile = <filename>.properties

Test Class Mapping

The test class mapping specifies which test IDs should be used by XLT and, more specifically, which test ID uses which test case implementation. That’s why you have to specify the fully-qualified class names of your tests here. Please note that you can map the same class to multiple load test names if needed. This is extremely useful when you want to run the same test case in different configurations.

com.xceptance.xlt.loadtests.<name>.class = <fully qualified class name>

For example, a test class mapping might look like this:

com.xceptance.xlt.loadtests.TVisitor.class = com.xceptance.xlt.samples.tests.TVisitor
com.xceptance.xlt.loadtests.TJSVisitor.class = com.xceptance.xlt.samples.tests.TJSVisitor

Test Class Specific Settings

Project-wide settings that are test case specific but not test run specific can be defined as well, using the following syntax:

<fully-qualified class name>.<property-name> = <value>

For example:

com.xceptance.xlt.samples.tests.blog-url = http://localhost:8080/pebble/
com.xceptance.xlt.samples.tests.TAuthor.username = username
com.xceptance.xlt.samples.tests.TAuthor.password = password
com.xceptance.xlt.samples.tests.webdriver.TAuthor.write-count = 2

Load Test Profile Configuration – test.properties

Test-run-specific settings – you can also configure an (optional) property file which contains the settings specific to a certain load test run. You can define more than one test property file, such as test-target-load.properties and test-2x-target-load.properties. This way, many configurations can be defined and prepared in advance and used as needed. You switch between these files by changing the property com.xceptance.xlt.testPropertiesFile in the project.properties file.

Load test profile configurations are done inside your test property file which is named test.properties per default. There you can define the test ID, the number of virtual users and all other load test specific settings, which are to be run in parallel agents using the following syntax:

com.xceptance.xlt.loadtests.<testID>.<setting> = <value>

For <testID> use any proper name. The following table lists all supported values for <setting> where the required settings are displayed in bold face:

A sample load profile configuration is given below:

com.xceptance.xlt.loadtests = TAuthor
com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TAuthor.iterations = 100
com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 3600
com.xceptance.xlt.loadtests.TAuthor.initialDelay = 0
com.xceptance.xlt.loadtests.TAuthor.warmUpPeriod = 30s
com.xceptance.xlt.loadtests.TAuthor.measurementPeriod = 10m 0s

All time period values can be specified in one of the following formats (without the quotes):

• total number of seconds: ‘1234s’ or '1234'
• natural style: ‘0h 12m 0s’, ‘0h 12m’, ‘12m 0s’ or '12m'
• digit style: ‘1:23’, ‘01:23’, ‘0:1:23’, or ‘0:01:23’

If you want to run several test cases simultaneously, specify the test case names as value for the property com.xceptance.xlt.loadtests in form of a space-separated list:

com.xceptance.xlt.loadtests = TAuthor TVisitor TCrawler
com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TVisitor.users = 3
com.xceptance.xlt.loadtests.TCrawler.users = 4 

Sample Configurations

User Count Model With Constant Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 5

Run exactly 5 users right from the beginning.

User Count Model With Ramp-Up Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 50
com.xceptance.xlt.loadtests.TAuthor.rampUpInitialValue = 10
com.xceptance.xlt.loadtests.TAuthor.rampUpPeriod = 5m

Run exactly 50 users, but ramp up the user count from 10 to 50 over a period of 5 minutes.

User Count Model With Variable Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 0/5 1h/50 2h/50 2h/100 3h/20

Runs the TAuthor scenario with a variable number of concurrent users (5 → 50 → 100 → 20).

Arrival Rate Model With Constant Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 100

Runs the TAuthor scenario exactly 100 times per hour using at most 5 concurrent users.

Arrival Rate Model With Ramp-Up Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 5
com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 100
com.xceptance.xlt.loadtests.TAuthor.rampUpInitialValue = 50
com.xceptance.xlt.loadtests.TAuthor.rampUpSteadyPeriod = 1m
com.xceptance.xlt.loadtests.TAuthor.rampUpStepSize = 10

Runs the TAuthor scenario exactly 100 times per hour using at most 5 concurrent users, but start with an arrival rate of 50 per hour and increase it every minute by 10 until the target level of 100 is reached.

Arrival Rate Model With Variable Load Profile

com.xceptance.xlt.loadtests.TAuthor.users = 5 
com.xceptance.xlt.loadtests.TAuthor.arrivalRate = 0/50  1h/100 2h/200 3h/150 

Runs the TAuthor scenario with a variable arrival rate (50 → 100 → 200 → 150) and use at most 5 concurrent users.

Other Settings

In case you want to modify the behavior of the logging facility of the load test agents, the test suite configuration directory contains a file named log4j.properties which can be changed to meet your needs.

To launch the Java Virtual Machine that runs the agent with additional parameters, specify them in a file named jvmargs.cfg.

Run the Load Test

Running the load test consists of two steps:

1. Running the agent controllers,
2. Running the master controller.

Running the Agent Controllers

To start the agent controllers, open a command line window/console and type the following sequence of commands:

cd <XLT>/bin
./agentcontroller.sh

Windows users have to use the appropriate .cmd file which is located in the same directory.

The agent controller will start up and listen on the specified port. The output will look like that:

- Using "C:\Users\AppData\Local\Temp\vfs_cache" as temporary files store.
- Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
- jetty-6.1.19
- Started SslSocketConnector@0.0.0.0:8500

Running the Master Controller

Make sure all agent controllers are running on all respective load test machines before starting the master controller. The master controller cannot be started if the agent controllers are not running. Also check that the test suite has been compiled successfully to avoid errors when uploading the test suite.

The master controller can be started in one of the following modes:

• Interactive mode: Typical sequence of steps to be executed to run a load test,
• Auto mode: Load test is started automatically, without user interaction.
• Embedded mode: For running a load test where master controller and agent controller run inside the same Java Virtual Machine.

Interactive Mode

You can start the master controller in interactive mode with the following command line:

cd <XLT>/bin
./mastercontroller.sh

Windows users have to use the appropriate .cmd file which is located in the same directory.

A screen like this will appear and displays the command line menu:

Xceptance LoadTest 4.2.0
Copyright (c) 2005-2012 Xceptance Software Technologies GmbH. All rights reserved.
Basic License (5 virtual users). This license does not expire.


(u) Upload test suite
(s) Start agents
(a) Abort agents
(r) Show agent status
(d) Download test results
(c) Create load test report
(q) Quit
=>

The following options are offered:

• Upload agent files (u): Choose this option to upload your test suite (code, data, and configuration) to all configured agent controllers. This is necessary at the very beginning and each time you have modified your test suite. XLT will upload only files changed to speed up the testing.

• Start agents (s): All agent controllers will receive a start command to spin off an agent that will execute its configured tests. This effectively starts the load test.

• Abort agents (a): Choose this option to terminate any running load agent immediately.

• Show agent report (r): The current status of the load agents can be monitored by choosing this option. Depending on the configuration, either a short summary (per test case) or a very detailed list (per test user) is shown. In either case, you will get information about:
  • test case name on which you are running the load test,
  • how many users are running,
  • how often a test case has been executed so far,
  • how long it took on average to execute the test case,
  • how many events and errors occurred, and
  • the overall progress.

• Download test results (d): Each load agent writes log files and runtime data files. Choose this option to download this data from all configured agent controllers. After entering d command one more menu appears where you can choose the amount of data to download. Press 1, 2 or 3 here. The files will be saved to a newly created directory at the location specified in default.properties. By default, the result directory is set to <XLT>/results. The name of the new directory is given by the current date and time, for example: 20110501-161718.

• Create report (c): Generates a load test report of the last downloaded test results.

• Quit (q): Shuts down the master controller and closes its connections to the agent controllers. Note that this will not stop any running load test. The load agents continue to execute the load test until they have finished. To regain control, just reconnect to the test cluster by restarting the master controller.

Once you have chosen an option (by pressing the associated key followed by ENTER), the appropriate action is executed. Afterwards, you will return to the menu immediately (unless you have chosen to quit, of course).

A typical usage scenario for a load test is reflected by the order of the master controller menu items and might look like this:

1. Upload the test suite (using (u) shortcut)
2. Start the agents (using (s) shortcut)
3. Check the agent status regularly (using (r) shortcut)
4. Download the test results once the test has finished (using (d) shortcut)
5. Create a report of the downloaded results (using (c) shortcut)
6. Quit the master controller (using (q) shortcut)

Auto Mode

As you have seen in the previous sections, there is a typical sequence of steps to be executed when running a load test. It may quickly become tedious and error-prone to type the necessary keys over and over again. To avoid this repetition, XLT provides another operating mode: the auto mode. In this mode, all the steps mentioned above are executed automatically without any user interaction. To start XLT in this operating mode, use the following command line:

Unix based systems:

cd <XLT>/bin
./mastercontroller.sh -auto

Windows:

cd <XLT>\bin
mastercontroller.cmd -auto

If the test suite files were uploaded and the load agents started successfully, XLT automatically refreshes the agent status regularly. Once the test has finished, the test results are downloaded. Afterwards, XLT quits.

If the command is followed by the option -report , a load test and performance report will be generated automatically after the load test has finished and the results have been downloaded.

cd <XLT>\bin
mastercontroller.cmd -auto -report

See below what the screen shows in auto mode:

Xceptance LoadTest 4.2.0
Copyright (c) 2005-2012 Xceptance Software Technologies GmbH. All rights reserved.
Basic License (5 virtual users). This license does not expire.

Uploading test suite ...
    0% ... 10% ... 20% ... 30% ... 40% ... 50% ... 60% ... 70% ... 80% ... 100% - OK

Starting agents ...
     0% ... 100% - OK

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TAddToCart      Running        10 of     10            0      0,00 s      0,00 s      0:00:00           0        0         0%
TCreateUser     Running        10 of     10            1      0,72 s      0,72 s      0:00:01           0        0         0%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10           72      0,67 s      0,77 s      0:00:06           0        2         5%
TAddToCart      Running        10 of     10           55      0,70 s      1,03 s      0:00:06           0        0         5%
TCreateUser     Running        10 of     10           83      0,95 s      0,67 s      0:00:06           0       17         5%

.
.
.

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Running        10 of     10        1.472      0,69 s      0,66 s      0:01:37          17       65        96%
TAddToCart      Running        10 of     10        1.412      0,66 s      0,68 s      0:01:37           0        0        96%
TCreateUser     Running        10 of     10        1.525      0,91 s      0,63 s      0:01:37           0      316        96%

Test Case       State         Running Users   Iterations   Last Time   Avg. Time   Total Time      Events   Errors   Progress
-------------   --------   ----------------   ----------   ---------   ---------   ----------   ---------   ------   --------
TAddToCart_lw   Finished        0 of     10        1.533      1,16 s      0,65 s      0:01:41          17       65       100%
TAddToCart      Finished        0 of     10        1.476      1,17 s      0,68 s      0:01:40           0        0       100%
TCreateUser     Finished        0 of     10        1.590      0,79 s      0,63 s      0:01:41           0      325       100%

Downloading test results ... (Please be patient, this might take a while)
    0% ... 30% ... 60% ... 100% - OK

To abort the test prematurely, press CTRL-C to terminate the master controller. This terminates all running agents as well and triggers the download of all test results generated so far. This also means, that it is not possible to disconnect the master controller from the test cluster while keeping the load test running.

For long running load tests, it is recommended to run the test without the -auto option, because this allows a disconnect from the test as well as inhibits accidental test termination.

Embedded Mode

Both interactive mode and auto mode, can be combined with the command line option -embedded. This option starts the master controller together with an internal agent controller.

This is useful if you want to run load tests without a distributed load test environment but run just one agent controller together with the master controller on the same machine. There is no need to start an agent controller manually before you run the load test then. This helps to handle automated load tests started from within a build process. This option is also recommended when playing around with pebble demo for training purposes because it simplifies the process for running a load test.

When using the option -embedded, the local agent controller settings will override the set of agent controllers configured in mastercontroller.properies.

Test Results and Reports

Collected Values

When running a load test, the XLT framework automatically collects a lot of information about the transactions, actions, and requests being executed and certain events. Additional custom timers and events can be added programmatically using the XLT API. All this data will later be the source for the XLT load test report.

These values are stored — separately for each test case and each virtual user — in a file named results/<TestCaseName>/<UserNo>/timers.csv. As the name already suggests, the file format is CSV. See the following snippet for an example:

R,PublishArticle.1,1366360224994,25,false,566,48930,200,http://localhost:8080/pebble/manageBlogEntry.secureaction,text/html,0,0,15,8,15,23
R,PublishArticle.7,1366360225027,19,false,456,653,200,http://localhost:8080/pebble/dwr/interface/Pebble.js,text/plain,0,0,2,0,2,2
R,PublishArticle.3,1366360225027,22,false,446,43876,200,http://localhost:8080/pebble/dwr/engine.js,text/javascript,0,0,5,0,5,5
E,Failed to download resource,1366360225027,TAuthor,==[404]== http://localhost:8080/themes/default/images/favicon.ico
A,PublishArticle,1366360224993,76,false
R,ConfirmPublishing.1,1366360225141,62,false,609,134,302,http://localhost:8080/pebble/publishBlogEntry.secureaction,,0,0,60,0,60,60
R,ConfirmPublishing.2,1366360225189,15,false,462,55555,200,http://localhost:8080/pebble/2013/04/19/1366360200018.html,text/html,0,0,9,5,9,14
R,ConfirmPublishing.8,1366360225221,15,false,457,653,200,http://localhost:8080/pebble/dwr/interface/Pebble.js,text/plain,0,0,2,0,2,2
R,ConfirmPublishing.4,1366360225221,16,false,447,43876,200,http://localhost:8080/pebble/dwr/engine.js,text/javascript,0,0,3,0,3,3
A,ConfirmPublishing,1366360225141,108,false
R,Logout.1,1366360225348,19,false,489,200,302,http://localhost:8080/pebble/logout.action?redirectUrl=http://localhost:8080/pebble/,,0,0,3,0,3,3
R,Logout.2,1366360225364,35,false,477,51601,200,http://localhost:8080/pebble/,text/html,0,0,15,8,15,23
R,Logout.8,1366360225395,5,false,471,653,200,http://localhost:8080/pebble/dwr/interface/Pebble.js,text/plain,0,0,1,0,1,1
R,Logout.4,1366360225395,6,false,461,43876,200,http://localhost:8080/pebble/dwr/engine.js,text/javascript,0,0,2,0,2,2
A,Logout,1366360225347,74,false
T,TAuthor,1366360222893,2599,false,

As you can see, the lines can have a different number of columns as they represent different types of information. The following table explains the meaning of each column depending on the data record type:

Note that the file format might be changed or extended in future XLT releases.

XLT Result Browser

When running test cases outside of Script Developer, either in Eclipse, as a load test, or as an Ant build, you have the option to save the page output to disk. The relevant property is com.xceptance.xlt.output2disk. By default, this is set to never. If you want to enable page output to disk you should copy the following lines to dev.properties or test.properties:

## Enables page output to disk. Possible values are:
## - never ..... pages are never logged
## - onError ... pages are logged only if the transaction had errors
## - always .... pages are logged always 
com.xceptance.xlt.output2disk = always   

All saved results can be found in the <testsuite>/results directory. See the following lines for details of the results sub-directory structure:

---+ results
   `---+ [testcase]
       `---+ [virtual-user]
           `---+ output
               `---+ [transaction-ID]
                   |---- css
                   |---- images
                   |---+ pages
                   |   `--- cache
                   `---- responses

In the folders for each test run (results/[testcase]/[virtual-user]/output/[transaction-ID]) you will find an index.html which contains the XLT Result Browser. The result browser offers an integrated navigation to browse the complete page output of the transaction, and allows to every single request in detail. The file last.html in the output folder results/[testcase]/[virtual-user]/output references the result browser for the last executed transaction of this virtual user.

Result browser navigation only allows access to the pages of a transaction if they are directly related to actions. Therefore defining actions properly is very important to make the most effective use of the result browser. For details of how to structure test cases and create actions, see also Basic Concepts and code structuring recommendations.

XLT Result Browser – Page Output

Clicking on one of the action names in the navigation makes the result browser show the respective page. When double-clicking an action name, the navigation expands to list all related requests. The listed requests are color-coded with black for successful responses (HTTP status code 200), red for protocol errors (HTTP status code 404) and grey for redirects (HTTP status code of 301 or 302).

When selecting one of the requests from the navigation, the page content is replaced by detailed information about the request and the related response accessible via four tabs on top of the page. The following information is available:

• Request Information
  • General Information
  • Request Headers
  • Request Parameters
• Request Body (Raw)
• Response Information
  • General Information
  • Response Headers
• Response Content

XLT Result Browser – Request Details

Creating and Evaluating Load Test Reports

As the most important tool for analyzing the results of a load test run, XLT offers three different load test reports. Each of these reports is explained in detail in separate sub-sections:

• Load and Performance Test Report
• Performance Comparison Report
• Performance Trend Report

To create the reports, you have to download all load test results from the agent controllers to the master controller. See the section Run The Load Test for details.

Once you have downloaded the load test results to your local disk, you can create the test reports with the XLT report generator. Enter a command in the console following the shown pattern:

cd <XLT>/bin
./<report-shortname>.(sh/cmd) ../results/<downloaded-results-dir> [options] 

The <downloaded-results-dir> and <report-shortname> have to be replaced with the appropriate values. For example:

./create_report.sh ../results/20110503-152920

This tells the report generator to take the specified results directory as input for the report. By default, the generated report is saved to <XLT>/reports. The report sub-directory is named after the respective results directory.

The report generator supports the following options:

• -o <dir>: an alternative output directory (optional)
• -from <time>: ignore results generated before the given time (optional)
• -to <time>: ignore results generated after the given time (optional)

With the -o option, you can specify an alternative output directory. Please keep in mind, that you have to specify a target directory name including the final directory for your report. When using the -o option, the directory name is not set automatically but the specified directory will be created. For example:

./create_report.sh ../results/20110503-152920 -o D:/Test_Reports/MyLatestReport

If you are only interested in creating a report for a specific time range:

./create_report.sh ../results/20110503-152920 -from 20110503-152600 -to 20110503-152800

Note that <time> has to be specified in the format yyyyMMdd-HHmmss and it has to match the time zone of your local machine. The resulting report will be rendered using your machine’s time zone per default.

All this information is rendered to HTML pages that can be viewed using a standard web browser. Once the report is generated, which may take a while depending on the amount of data gathered during the load test, you will find the file index.html in the root of the appropriate test report directory. Open this file in a web browser to view the report.

Load and Performance Test Report

The Load and Performance Test Report will give you all the information needed for a detailed analysis after a load test run. It contains several sections, each with a table and one or more charts showing graphic development of relevant measurements over time.

Load and Performance Test Report

Load and Performance Test Report – Charts

• Load Profile: This section shows the load profile as configured in the properties files of your test suite.

• Overview: This section shows some general information about the load test, such as start and end time, total duration, and some information about the general test setup. It also displays a hit statistic for HTTP/HTML based load tests.

• Users: The number of active users shows the number of test cases that were underway concurrently. The chart shows ramp-up and shutdown periods, as well as potential problems with agents, which are often visible as sudden drops in the active user count. This section also visualizes the test user setup.

• Network: The network section covers the areas of incoming and outgoing traffic during the load test. Sent Bytes is an estimated number based on the data that was given to the network layer. Cookies for instance are not included. Received Bytes is an accurate number because it is based on the data received and includes http-header information. Depending on the test runtime, the numbers for per hour and per day might be estimations based on a linear projection of the available data.

• Transactions: A transaction is a completed test case. The test case consists of one or more actions. The shown run time of a transaction contains the runtime of all actions within the test case, think times, and the processing time of the test code itself. If the test path of the test case is heavily randomized, the runtime of transactions might vary significantly. The average runtime shows the development of tests over time and especially helps to evaluate the outcome of long running tests.

• Actions: An action is part of a test case and consists of a prevalidation, an execution, and postvalidation. The data shown here is the time spent in the execution routine of an action. Therefore its runtime includes the runtime of a request, e.g. an HTTP operation, and the necessary time to prepare, sent, wait, and receive the data.

• Requests: The request section is the most important statistics section when testing web applications. It directly reflects the loading time of pages or page components. Each row holds the data of one specific request. Its name is defined within the test case as timer name. The Count section of the table shows the total number of executions (Total), the calculated executions per seconds (1/s), minute (1/min), as well as projections or calculations of the executions per hour (1/h) and day (1/d). The Error section shows the total amount (Total) of errors that occurred during loading the page or page component. The error count does not include errors detected during the post-validation of the data received. Typical error situations are HTTP-response codes such as 404 and 505, timeouts, or connection resets. The runtime section of the table shows the median, the arithmetic mean, as well as the minimum and maximum runtime encountered, and the standard deviation of all data within that series. The runtime segmentation sections shows several runtime segments and the number of requests within the segment’s definition. If the runtime of the test case is shorter than a displayed time period, e.g. test runtime was 30 min and the time period is hour, the numbers are a linear projection. That means they show a possible outcome of a longer test run, if load and application behavior would remain the same.

• Custom Timers & Values: The custom timers includes all timers that have been placed individually within the test code. The chart and data description is identical to the request section. If custom samplers have been run during the test, the collected data is shown in the Custom Values sub-section below.

• Errors: This table contains all errors and their stack traces thrown by the test cases. This data helps to identify application errors or test problems.

• HTTP Response Codes: If the test run included web activities or other activities that return an HTTP response code, it can be found here. The table contains the response code, the total number of occurrences, and the percentage in relation to the total number of response codes.

• Events: This table contains all events that have occurred during the load test. Events are used to indicate that the test has encountered a special situation, which is not an error, but is too important to ignore or to write to the log only.

• Agents: This section reports the resource utilization of each user agent in terms of CPU and memory usage. This helps to identify potential resource bottlenecks which might have influenced the load test. Please note, that all data is local to the Java Virtual Machine of the agent and therefore covers only a process view.

• Configuration: The configuration section lists the test configuration as well as the load profile used to run the test. It helps to make the test reproducible and preserves the test settings for later test evaluation.

Create A Load And Performance Test Report

In order to generate a load and performance test report, use the following command:

create_report.(sh/cmd) ../results/<testDataDir> [options]

For example:

./create_report.sh ../results/20110503-160520

As an alternative to the command described above you can also create a load and performance test report with the (c) shortcut from the master controller’s command line menu. This shortcut creates a report of the least recently downloaded results.

Configuring the Report Generator

Linking to Result Browser Directories

If an error occurred during the load test, the corresponding error message and stack trace will be shown in the Errors section of the load test report. If you enabled storing the visited pages to disk, you will also find a directory name as part of the error information. To view the visited pages, use this directory name to locate the corresponding result browser in the results directory of the load test.

You can also make the result browsers directly accessible right from the load test report. This greatly speeds up error analysis as the result browsers are just one click away. You would simply click the directory name next to an error entry to open the respective result browser. However, to make this work you have to ensure that:

1. the results will be provided at the target location, and
2. the results directory will never be renamed oder moved.

Otherwise, the viewers of the report would experience broken links.

In order to let the report generator create links from the load report to the result browsers, set the property com.xceptance.xlt.reportgenerator.linkToResultBrowsers in <XLT>/config/reportgenerator.properties to true.

By default, the report generator calculates the path from the report to the result browsers based on the results directory (given on the report generator’s command line) and the reports directory (either being the default directory or the one explicitly given as command line argument). The computed path will be a relative path if possible and an absolute path otherwise (on Windows, if report and results are on different drives).

Sometimes the relative path approach is not suitable, for example if you send the report to your team members, but not the results. In this case, the results must be made available somewhere on the net. Furthermore, the report generator needs to know about this location to properly generate the links. To this end, you configure a results base URI, for example http://myhost/results. The URI is a base URI as it is common for the results of all your load tests. The report generator automatically appends the name of the results directory (for example 20121106-111751) to this URI when generating the links to the result browsers, so the resulting link might look like this: http://myhost/results/20121106-111751/ac01_00/TSearch/126/output/1352194484275/index.html

By using a base URI you do not need to reconfigure the report generator when generating the report for another load test, unless you choose the publish the results at a totally different location. To configure the base URI, set the property com.xceptance.xlt.reportgenerator.resultsBaseUri in <XLT>/config/reportgenerator.properties to the appropriate value.

Performance Comparison Report

The Performance Comparison Report gives you a quick overview over performance improvements (green color tones) or performance decline (red color tones) between two test runs. The initial test run is labeled baseline. The test run that is compared to the baseline is labeled measurement run.

Every section of the comparison report displays a table with changes in performance and is divided into the following three parts:

• Count: The percentage values show the development of the performance in comparison to the baseline. Positive numbers in the count section mean an improvement of the throughput over the baseline. Negative values indicate a decrease of throughput.

• Errors: An increase in the number of errors is indicated with positive numbers, while a decrease in errors is shown in negative numbers. An infinite sign indicates the occurrence of errors in comparison to an error-free baseline.

• Runtime: Positive values indicate a poorer performance, while negative values show an improvement (smaller runtime values) over the baseline.

When you hover the mouse over the columns of the report table you will see the actual measurement results. This will give you a better impression whether or not the reported percentage change is significant or not.

Performance Comparison Report – Overview

Performance Comparison Report

Performance Comparison Report contains the following sections:

• Overview: The overview section shows some general information about both load tests. This enables you to compare settings, runtime, and profiles. In the later sections, the percentage values show you the development of the performance in comparison to the baseline. Please note that the total columns (total throughput and total errors) might present misleading values if the two load tests used different runtime configurations. All other values are normalized in respect to the runtime and therefore easily comparable. Positive numbers in the count section mean an improvement of the throughput over the baseline. Negative values indicate a decrease of throughput. An increase in the number of errors is indicated with positive numbers, while a decrease in errors is shown in negative numbers. An infinite sign indicates the occurrence of errors in comparison to an error-free baseline. For all runtime numbers, positive values indicate a poorer performance, while negative value show an improvement – smaller runtime values – over the baseline. Added or removed transactions, actions, or requests are indicated. No comparison is provided for these.

• Transactions
  • Count
  • Errors
  • Runtime

• Actions
  • Count
  • Errors
  • Runtime

• Requests
  • Count
  • Errors
  • Runtime

• Custom Timers
  • Count
  • Errors
  • Runtime

Create a Performance Comparison Report

You can generate a performance comparison report only between two existing load and performance test reports. This means you have to create both load and performance test reports first.

After that you can generate a performance comparison report using the following command:

create_diff_report.(sh/cmd) <testReportDir_1> <testReportDir_2> [options]

For example:

./create_diff_report.sh ../reports/20110503-152920 ../reports/20110503-160520

Performance Trend Report

A trend report shows the development of performance over time. Multiple measurements are taken into account and evaluated against each other. An XLT trend report shows you how your system performs over time, how your tuning effort pays out, and how your live environment acts under changing load situation, if used as monitoring.

Two trend report types are available:

• Difference to First Run and
• Difference to Previous Run.

The Difference to the First Run reports the changes compared to your first test run, mostly referred to as baseline. Each table column will show you the difference between your baseline run and the run your are interested in. The quality of your baseline run defines how valuable this report may be. You can also see this as long-term performance trend report.

The Difference to Previous Run visualizes the improvements between two adjacent test runs. This shows you, how your last change or tuning effort payed out in comparison to the previous run. It helps you to see whether or not you are on the right track improving the performance of your application. The report also emphasizes sudden improvements or set-backs. This report can also be seen as a short-term performance trend report.

When you hover the mouse over the columns of the trend report table you will see the actual measurement results. This will give you a better impression whether or not the reported percentage change is significant or not. Please keep in mind, that changes up to 10% are most of the time measurement fluctuation.

Performance Trend Report – Overview

Performance Trend Report

Similar to the other reports the trend report is divided into the following sections, each with the mentioned tables and charts:

• Overview
• Transactions
• Actions
• Requests
• Custom Timers

Create a Performance Trend Report

You can generate a performance trend report over several test reports using the following command:

create_trend_report <testReportDir_1> ... <testReportDir_n> [options]

For example:

./create_trend_report.sh ../reports/20110503-152920 ../reports/20110503-160520 ../reports/20110503-161030

Custom Values

During a load test XLT is logging a large amount of data relevant to the test run. Nevertheless, sometimes it comes in handy to log some additional information about the system under test (SUT) directly during the load test run. For this purpose XLT provides custom values.

Example: An eCommerce application is typically connected to several third-party systems to use external services like credit-worthiness check. The response time of these third-party systems can have a major impact on the SUT’s response to the client request. This application-internal information is not visible to XLT during a load test by default. A typical example for custom values in this context is logging the response time of requests to third party systems. To do so the user must write custom code to access the relevant sources, e.g. via remote connection to the application server. The additional data then can be logged by XLT while load test runtime and is automatically integrated into the load and performance test report.

Sampler

Custom Samplers enable the user to query custom sources and log data (samples) while load test runtime. For that purpose the user must provide a custom sampler class extending com.xceptance.xlt.api.engine.AbstractCustomSampler. The sampler gets configured in the test suite configuration files. Recommended location for the relevant configuration is project.properties.

The provided sampler must override the execute() method that is called after each interval time (see configuration). Furthermore the sampler might override the method initialize() or shutdown() that get called just once for the sampler. While initialize() is called before the first call of execute(), shutdown() is called on shutdown.

The logged custom value is the return value of the execute() method.

The AbstractCustomSampler can store any ‘double’ value. The stored value describes the absolute value at a certain point in time. The corresponding report chart is directly showing the logged value.

Configuration

Samplers get configured by providing the following properties:

com.xceptance.xlt.customSamplers.1.class = com.xceptance.xlt.samples.ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.name = ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.description = This sampler logs a custom value which is just a random number
com.xceptance.xlt.customSamplers.1.interval = 1000
com.xceptance.xlt.customSamplers.1.chart.title = ValueSamplerDemo
com.xceptance.xlt.customSamplers.1.chart.yAxisTitle = Value
#com.xceptance.xlt.customSamplers.1.property.foo = 123
#com.xceptance.xlt.customSamplers.1.property.bar = abc
...
com.xceptance.xlt.customSamplers.9.class = ...
com.xceptance.xlt.customSamplers.9.name = ...
...

• com.xceptance.xlt.customSamplers.n. is the saved key for custom sampler properties. Each sampler configuration block must have a unique number (called n in this example). The numbers don’t need to be in strictly successive order.
• class points to the sampler class (including full package path).
• name is a customizable name of the sampler. This name must get used when instantiating a sample class (it’s recommended to use method getSamplerName()).
• interval defines the period the sampler gets started (unit of time is milliseconds). The value must be positive (including 0). A new sampler gets only started if it is the first execution or the previous sampler has came to end.
• Providing a chart title is optional. By default the sampler name is used. yAxisTitle defines the title of the y-axis for the rendered chart.
• Providing further sampler properties is optional. The properties can be accessed by calling the method getProperties() or getProperty(key) (where key is the string in the configuration between com.xceptance.xlt.customSamplers.n.property. and the equals sign (=). In this example, the keys are foo and bar with values 123 and abc). Sampler property keys must not contain dots or whitespace. Apart from that they are free in name and count.

Example

The following code shows an example of a very simple custom sampler that logs random values.

public class ValueSamplerDemo extends AbstractCustomSampler
{
    public ValueSamplerDemo()
    {
        super();
    }

    @Override
    public void initialize()
    {
        // initialize
    }

    @Override
    public double execute()
    {
        // generate random value based on the configured limits

        // get properties
        final String lowerLimitProp = getProperties().getProperty("generatedValueLowerLimit");
        final String upperLimitProp = getProperties().getProperty("generatedValueUpperLimit");

        // convert to integer
        try
        {
            final int lowerLimit = Integer.valueOf(lowerLimitProp);
            final int upperLimit = Integer.valueOf(upperLimitProp);

            // return the value to be logged
            return XltRandom.nextInt(lowerLimit, upperLimit) + XltRandom.nextDouble();
        }
        catch (final NumberFormatException e)
        {
            // log 0 in case of an exception
            return 0;
        }
    }

    @Override
    public void shutdown()
    {
        // clean up
    }
}

The resulting chart is automatically integrated into the XLT performance and load test report and can be accessed via the report navigation menu item Custom Values.

Custom sampler report

External Data Report

As an alternative to custom values XLT report generator provides the ability to include external data available as files into the report. This can be used if it is not possible to access the external data source directly during load test runtime.

Parser

To read the external files and integrate them into the load test report a Parser class is needed for each type of file or format. XLT provides a set of predefined parsers for CSV files. If you plan to integrate some other file formats or reports you have to write your own parser class extending AbstractLineParser in com.xceptance.xlt.api.report.external.

• SimpleCsvParser: Extracts the data by splitting each line into the comma separated values.
• HeadedCsvParser: Like SimpleCsvParser, but additionally names the values by the column value in the first line.

XLT ships with a demo test suite demo-external-data to show the usage of external data in the load test report. As an Example of advanced parser classes two parsers for handling logs of the command line tool iostat can be found in the source directory of this test suite project <XLT>\samples\demo-external-data\src in package com.xceptance.xlt.report.external.

• IostatCpuParser: Parses the CPU section of a log of command line tool iostat (with parameter -t to log time stamps).
• IostatDeviceParser: Parses the Device section of a log of command line tool iostat (with parameter -t to log time stamps).

Configuration

Location

The configuration file externaldataconfig.xml is expected in the result directory of the respective load test run. If this file can’t get located there, it will get looked up in the master controller’s config directory. See the provided sample file <XLT\samples\demo-external-data\config\externaldataconfig.xml.sample and adapt to your needs after removing the .sample extension. At least adapt the source file and columns/indexes. The configuration file adapted for the demo-external-data project is located at <XLT>\samples\demo-external-data\results\20110621-101041.

Structure

The configuration defines files to parse by a specified parser, the column indexes to mark the relevant data and other settings like headlines, descriptions or colors to enrich the chart/table information for the report.

• file: Define which file gets processed by which parser. If you do not set a file encoding the default encoding UTF-8 is used.
• charts: You might configure one or more charts. The minimum chart configuration requires the chart title and the column indexes of the series. See the following list of available attributes and their default values.
  • Chart defaults:
    • title [mandatory and unique] (chart title)
    • yAxisTitle/yAxisTitle = "Values" (title of the 1st/2nd y-axis (value axis))
    • xAxisTitle = "Time" (title of the x-axis (time axis))
    • Series defaults:
      • valueName [mandatory] (e.g. column index for SimpleCvsParser or column headline for HeadedCsvParser)
      • title = [_valueName]_ (series title)
      • axis = "1" (number of axes (1 or 2))
      • color/averageColor = [a color of internal default color set] (hex based RGB color e.g. “#1a2b3f”)
      • average = [no average] (percentage of data used to build an average value)
• tables: You might configure one or more tables. The minimum table configuration requires the table title and the column indexes of the rows. See the following list of available attributes and their default values.
  • Table defaults:
    • title [mandatory] (table title)
    • type = "minmaxavg" (currently there is only this type supported printing the minimum, maximum and average of read column values)
    • Row defaults:
      • valueName [mandatory] (e.g. column index for SimpleCvsParser or column headline for HeadedCsvParser)
      • title = [_valueName]_ (table title)
      • description = [no description]
      • unit = [no unit] (unit of measurement for parsed data)
• properties: You may provide properties to the report generation. In case you extend a parser or write your own this is a good way to provide some free properties.

Currently there are two properties supported by the framework to convert a human readable date/time to a time stamp (parser.dateFormat.pattern and parser.dateFormat.timeZone). If the time stamp is already a UNIX time stamp, the pattern is not needed. It’s strictly recommended to adapt these properties to your needs. Please see java.text.SimpleDateFormat for more information on date/time patterns.
Another supported property is parser.csv.separator that describes the expected field separator char in CSV files. By default this is a comma (,).

Command Reference

This chapter provides a detailed explanation of the commands available in XLT using the Scripting API or the XLT SD. Before explaining the commands in detail we give some more general information (about the command structure and so on). After that a detailed description for each of the commands follows. For convenience and to have a brief overview some tables are given at the end including a table with an example for each command.

At a first glance it may sound confusing that we explain the commands for these two components (XLT Scripting API and XLT SD) together but omit for example the XLT Action API. That is due to both components supporting nearly exactly the same commands with the same syntax. For further information about these components and other components of XLT (for example the XLT Action API) take a look at the sections dedicated to them. If you are planning to use the XLT Scripting API take note of the additional hints at the end of this chapter.

Command Structure

Commands have a name and may have a target parameter and/or a value parameter, depending on the command in question. Most commands somehow deal with parts of an HTML page, so the target of the command is taken to identify the element on the page. Furthermore a command may contain a delay, for example the number of milliseconds to wait after clicking on an element before continuing the test. A command may also deal with some text that should be matched and which is given as argument.

The exact parameters are given for each command in the command’s description. In general a command has one of the following forms:

1. CommandName() or
2. CommandName(Target) or
3. CommandName(Value) or
4. CommandName(Target, Value)

where the target is used to identify a part of an HTML page and the value is some string or numerical value.

In general the parameters define:

1. which element of an HTML page should be identified and how it should be identified (see Locators below)
2. which text should be matched and how it should be matched (see Text Matching below)
3. other parameters, such as a delay

Note that even with the three parameters types listed above a command at most has two parameters.

A command may interpret its parameters differently and deal with them in a very specific way. For example, the command “storeEval(target, value)” interprets its target as JavaScript expression.

Locators

Almost all commands operate on a certain item, mostly an element on a page, but it can also be a window or a frame. In order to identify, or locate, the item in question,
locator expressions can be specified, which define which item should be located and by which means.

The general syntax of a locator expression is as follows: <strategy>=<value>

The strategy defines how to search for an item, while the value defines which item is to be addressed. For example, to locate an element with id attribute “foo”, we would specify this locator: id=foo

Note that the strategy (including the following equal sign) can be omitted. In this case, a built-in default strategy is used. However, we do not recommend omitting the strategy. It may save you some characters to type, but you would always need to remember what the default strategy for a certain locator type is.

If the evaluation of the locator results in no elements being identified, the command will fail. If, on the other hand, the locator identifies more than one element, the first element found will be used.

Currently, there are four types of locators:

1. element locators: locate elements on an HTML page
2. option locators: locate <option> elements inside a <select> element on an HTML page
3. window locators: locate a browser window
4. frame locators: locate a frame window on an HTML page

Each of them will be explained in detail in the following sub sections.

Element Locators

Elements can be identified with one of the following strategies:

Note that not all strategies are applicable to all elements. For example, the link strategy makes sense only when locating <a> elements.

Option Locators

When dealing with select boxes, we will need two locators, one that identifies the select element, and one that identifies the option element(s) in question. The first locator is a normal element locator. For an option locator, the following strategies can be used:

Note that when using any of the commands addSelection or removeSelection, all options that have been found, not just the first, will be processed if the given option locator identifies more than one option.

Window Locators

Windows can be identified with one of the following strategies:

Frame Locators

Frames can be identified in two ways:

• by locating the frame via special document properties, or
• by locating the corresponding <frame> element on the page.

The latter can be achieved by a regular element locator, so we have to explain the other strategies only:

Text Matching

Many commands check for the presence of some text on the page or wait for some text to appear. Since the text displayed on the page may not always be static, but can contain dynamic parts (such as the user name in “Welcome, Mr. John Smith!”), the provided validation text needs to be somewhat blurry as well (for example “Welcome, Mr. *!”). This is where text matching expressions come into play.

The general syntax of a text matching expression is as follows: <strategy>:<text pattern>

The strategy defines which pattern syntax will be used in the text pattern that follows the colon. For example, if we want to use the famous “?” and “*” wildcard characters in the text pattern, we would specify an expression such as this one: glob:Welcome, Mr. *!

Note that the strategy (including the following colon) can be omitted. In this case, a built-in default strategy is used.

Currently, there are four text matching strategies:

If your text pattern text contains a colon, you have to specify a proper text matching strategy since the colon character is used as separator to parse the text matching strategy and text pattern from the input string.

Be aware that prior to comparing the actual text with the expected text, each sequence of whitespace (spaces, non-breaking spaces, line breaks, or tabs) is replaced by a single space in both the actual text and the expected text. This is done to make comparisons independent of the actual amount of whitespace.

Further note that depending on the command in question the same expression is treated slightly different. Some commands check whether the text (of the located element / page) matches the expression while others check that the text contains a sub sequence (may be the text itself) that matches the expression. For example, with exact=user the text No users found! may or may not match depending on the command.

Commands

In general, commands can be categorized. Some of them interact with the elements of an HTML page (for example clicking on a button), some verify things (for example that the page’s title matches the expected one), some wait for a condition to be fulfilled (for example waiting that a specific element becomes visible), and some store data in variables so that it can be reused later on. Finally, there is a left-over category for miscellaneous commands.

Commands in different categories may nevertheless be somehow related. For example, let us assume that we want to do something with the text that is shown on an HTML page. Then we can use assertText to check whether the text occurs on the current page, but we can also use waitForText for the text in question to show up on the page (it may be inserted by an AJAX call). Furthermore, we have storeText which enables us to store the text of the targeted element in a variable so that it can be accessed later on.

In the following we explain the commands using the categories informally introduced at the beginning of this section. Within each category the commands are explained in alphabetical order. The explanation of each command ends with one example.

Note that for each of the assertXXX and waitForXXX commands there is also a negated counterpart assertNotXXX, and waitForNotXXX respectively. In these cases, we use references to previously explained commands to avoid lengthy explanations to be duplicated.

Commands That Interact With A Page

Here we refer to commands that change the page or trigger events (for example by clicking on a link).

addSelection

Selects options in a multi-selection element. It selects all options that match the command value, so it may select several options! Aborts the test with an error if no option matching the command value is found or the element is no multi-selection element. Also aborts the test with an error if the multi-select element is disabled or does not support multi-selection. Any element locator except the link locator can be used to identify the multi-select item, and any option locator can be used for the value with this command.

Example: adddSelection(id=RadioChannel,index=1)

check

Checks the targeted element. The element has to be an input element, must be of type radio or checkbox and must be enabled. Otherwise the test aborts with an error. Any of the element locators can be used to identify the element.

Example: check(id=beautifulWrapping)

checkAndWait

Does the same as check but waits until page load triggered by checking the targeted input element is complete or until the configured timeout has been reached. If no page load is triggered and/or the timeout is reached the test aborts with an error.

Example: checkAndWait(id=beautifulWrapping)

click

Simply clicks on the targeted element. Be aware that the click command does not wait for a page being loaded, so do not use the click command to click on a link. Instead, use clickAndWait in such a case (otherwise you will probably end up with only a few parts of the new parts being loaded). Aborts the test with an error if there is no element matching the given locator. Any element locator can be used.

Example: click(id=productsLink)

clickAndWait

Does the same as click but waits until page load triggered by clicking the targeted element is complete or the configured timeout has been reached. If no page load is triggered and/or the timeout is reached the test aborts with an error.

Example: clickAndWait(id=productsLink)

doubleClick

Double clicks the targeted element. Accepts any element locator but makes the test to abort with an error if there is no element which can be identified using the locator.

Example: doubleClick(id=confirm)

doubleClickAndWait

Same as doubleClick but waits after the double click for the page load triggered by this action to become complete. If no page load is triggered and/or the configured timeout has been reached the test aborts with an error message.

Example: doubleClickAndWait(id=confirm)

mouseDown

Clicks on the targeted element with the left mouse button but does not release the mouse button. Aborts the test with an error if there is no element which can be identified using the locator. Any element locator can be used.

Example: mouseDown(id=confirmationButton)

mouseOut

Moves the mouse away from the targeted element. Currently the mouse just hovers to the current page’s body and the locator is ignored. Nevertheless it is tried to identify an element using the locator. So the test aborts with an error if there is no element which can be identified using the locator. You can use any element locator and you can not move the mouse outside the current page’s HTML body.

Example: mouseOut(id=confirmationButton)

mouseOver

Hovers the mouse over the targeted element. Aborts the test with an error if there is no element which can be identified using the locator. Any element locator can be used.

Example: mouseOver(id=confirmationButton)

Please notice that CSS hover events are not triggered by mouserover since these are implemented using native browser events.

mouseUp

Releases the left mouse button on the targeted element. Aborts the test with an error if there is no element which can be identified using the locator. Any element locator can be used.

Example: mouseUp(id=confirmationButton)

removeSelection

Deselects options in the targeted multi-selection element. It deselects all options that match the command value, so it may deselect several options! Aborts the test with an error if no option matching the command value is found or the element is no multi-selection element. Also aborts the test with an error if the multi-select element is disabled or does not support multi-selection. Any element locator except the link locator can be used to identify the multi-select element, and any option locator can be used for the value with this command.

Example: removeSelection(id=RadioChannel,index=1)

select

Selects options in the targeted element. Aborts the test with an error if no element can be identified using the given locator or the element is not an HTML select element or the element is disabled. Does nothing to disabled options. If the targeted element is a multi-select element, all other options except the ones identified by the locator are deselected. For the target any element locator can be used, for the command value any option locator can be used.

Example: select(id=RadioChannel,index=1)

selectAndWait

Same as select but waits for the page load triggered by the selection to become complete or the configured timeout has been reached. Aborts the test with an error message if there was no page load triggered and/or the configured timeout has been reached.

Example: selectAndWait(id=RadioChannel,index=1)

submit

Submits the targeted form. Aborts the test with an error if the element is not an HTML form. Any element locator can be used.

Example: submit()

submitAndWait

Same as submit but waits for the page load trigggerd by the form submit to become complete or the configured timeout has been reached. It fails with an error if the targeted form could not be found. It also fails if there was no page load triggered and/or the configured timeout has been reached.

Example: submitAndWait()

type

Types the command value into the targeted element. Aborts the test with an error if no element can be identified using the locator. Any element locator can be used.

Example: type(id=quantity,1)

typeAndWait

Same as type but waits for the page load triggered by typing the given string to become complete. Aborts the test with an error message if there is no page load triggered and/or the configured timeout has been reached.

Example: typeAndWait(id=quantity,1)

uncheck

Deselects the targeted checkbox or radio button. Aborts the test with an error if no element can be identified using the given locator or the element is not an input element or not of type checkbox or radio, or it is disabled. Accepts any element locator.

Example: uncheck(id=birthdayWrapping)

uncheckAndWait

Same as uncheck but waits for the page load triggered by uncheck to become complete. Aborts the test with an error message if there is no page laod triggered and/or the configured timeout has been reached.

Example: uncheckAndWait(id=birthdayWrapping)

Commands That Make An Assertion

In this section we explain all the assertXXX and assertNotXXX commands. In general, all assertXXX commands check that the given condition evaluates to true and all assertNotXXX commands check that the given condition evaluates to false.

assertChecked

Asserts that an element is checked. Aborts the test with an error message if there is no element matching the element locator, the found element is neither a checkbox nor a radio element, or it is not checked.

Example: assertChecked(id=termsAndConditions)

assertElementPresent

Asserts that an element is present on the page. Aborts the test with an error message if there is no element matching the element locator. Any element locator can be used.

Example: assertElementPresent(id=ProductDetails)

assertLoadTime

Asserts that the time needed to load the current page does not exceed the given value. Aborts the test with an error message if page loading lasts too long. Has no target (cause it aims at the current page) but only a mandatory value that represents the number of milliseconds the page load may last at most.

Example: assertLoadTime(15000)

assertNotChecked

Asserts that the given element is not checked. Aborts the test with an error message if there is no element matching the given element locator, the found element is neither a checkbox element nor a radio element, or it is checked.

Example: assertNotChecked(name=sameAsBilling)

assertNotElementPresent

The inversion of assertElementPresent, thus the syntax is the same except the command name.

Example: assertNotElementPresent(id=ProductDetails)

assertNotSelectedId

Asserts that the id of none selected option of the given select element matches the argument pattern. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: assertNotSelectedId(id=shippingMethod,UPS)

assertNotSelectedIndex

Asserts that the index of none selected option of the given select element matches the argument index. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. The indices start with 0.

Example: assertNotSelectedIndex(id=shippingMethod,1)

assertNotSelectedLabel

Asserts that the label of none selected option of the given select element matches the argument pattern. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: assertNotSelectedLabel(id=shippingMethod,UPS)

assertNotSelectedValue

Asserts that the value of none selected option of the given select element matches the argument pattern. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: assertNotSelectedValue(id=shippingMethod,UPS)

assertNotText

Asserts that the command value (=text) does not match the text of the identified element. Any element locator can be used for the command target. Any text matching approach can be used for the command value. Aborts the test with an error message if the element’s text matches the value text or there is no element matching the target expression. Note that hidden elements have an empty string as text!

Example: assertNotText(id=availability,regexp:.*out of stock.*)

assertNotTextPresent

Asserts that there is no text on the page matching the command value (=text). Aborts the test with an error message if the page’s text matches the text. Has no target (cause it aims at the current page). Any text matching approach can be used for the value. This command differs to assertNotText in so far that the text value is matched if the page contains a text snippet that matches the text value, even if you use the exact text matching approach.

Example: assertNotTextPresent(exact:? results returned) (also matches if the text is “No results returned!” cause “o results returned” is exactly matched by the command value)

assertNotTitle

Asserts that the page title does not match the command value. Like for assertNotTextPresent there is no target because this command aims at the current page’s title. Unlike assertNotTextPresent the text must strictly match the command value (which may contain wildcards and such things nevertheless). Any text matching approach can be used. Aborts the test with an error message if the current page’s title matches the command value.

Example: assertNotTitle(exact:? results returned) (does not match if the title is “0 results returned!” due to the exclamation mark)

assertNotValue

Asserts that the value of the element identified by the locator does not match the command value. Any element locator can be used for the target and any text matching approach can be used for the command value.

Example: assertNotValue(id=shippingMethods,UPS)

assertNotVisible

Asserts that the element identified by the locator is currently not displayed. Any element locator can be used.

Example: assertNotVisible(id=shippingMethods)

assertNotXpathCount

Asserts that the number of elements that can be identified using the locator is unequal to the command value. Only a xpath expression can be used for the locator. Aborts the test with an error message if the number of matching elements is equal to the command value.

Example: assertNotXpathCount(xpath=a[.=‘continue’],1)

assertPageSize

Asserts that the size of the current page does not exceed the given number of bytes as specified in the command value. Aborts the test with an error message if the page’s size is greater than the given value. Same as with the other page related commands there is no locator. As value you have to specify an integer number, giving a floating point number will result in an error and test abortion when executing the test.

Example: assertPageSize(1024)

assertSelectedId

The exact inversion of assertNotSelectedId (considering the behavior, not the syntax).

Example: assertSelectedId(id=shippingMethod,UPS)

assertSelectedIndex

Asserts that the index of at least one selected option of the given select element matches the argument index. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value but obviously it only makes sense to give integers as arguments. The indices start with 0.

Example: assertSelectedIndex(id=shippingMethod,1)

assertSelectedLabel

The exact inversion of assertNotSelectedLabel (considering the behavior, not the syntax).

Example: assertSelectedLabel(id=shippingMethod,UPS)

assertSelectedValue

The exact inversion of assertNotSelectedValue (considering the behavior, not the syntax).

Example: assertSelectedValue(id=shippingMethod,UPS)

assertText

This is the exact inversion of the assertNotText command. The syntax remains the same except for the command name.

Example: assertText(id=availability,regexp:.* in stock.*)

assertTextPresent

This is the exact inversion of the assertNotTextPresent command. The syntax remains the same except for the command name.

Example: assertTextPresent(exact:? results returned)

assertTitle

This is the exact inversion of the assertNotTitle command. The syntax remains the same except for the command name.

Example: assertTitle(? results returned)

assertValue

Asserts that the value of element identified by the locator matches the command value. Any element locator can be used for the target and any text matching approach can be used for the command value.

Example: assertValue(id=shippingMethods,UPS)

assertVisible

Asserts that the element identified by the locator is currently displayed. Any element locator can be used.

Example: assertVisible(id=shippingMethods)

assertXpathCount

This is the exact inversion of the assertNotXpathCount command. The syntax remains the same except for the command name.

Example: assertXpathCount(xpath=a[.=‘continue’], 1)

Commands That Wait For A Condition To Become True/False

In this section we explain all the waitForXXX and waitForNotXXX commands. In general, all waitForXXX commands wait for the condition to become true and all waitForNotXXX commands wait for the condition to become false. The waiting time has a default value which can be overridden by using the setTimeout command (see here below).

waitForChecked

Waits for the given element to be checked. Aborts the test with an error message if the timeout is reached and no such element is present which is a checkbox or a radio element, and that is checked.

Example: _waitForChecked(name=shipAsGift)

waitForElementPresent

Waits for the targeted element to be present on the current page. Aborts the test with an error message if the timeout is reached and no such element is present. Accepts any element locator.

Example: waitForElementPresent(id=shippingPanel)

waitForNotChecked

Waits for the given element to be unchecked. Aborts the test with an error message if the timeout is reached and no such element is present which is a checkbox or a radio element, and that is not checked.

Example: _waitForNotChecked(name=newsletter)

waitForNotElementPresent

The exact inversion of waitForElementPresent (considering the behavior, not the syntax).

Example: waitForNotElementPresent(id=minicart)

waitForNotSelectedId

Waits for none selected option of the given select element matching the argument pattern as id. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: waitForNotSelectedId(id=shippingMethod,UPS)

waitForNotSelectedIndex

Waits for none selected option of the given select element matching the argument index. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. The indices start with 0.

Example: waitForNotSelectedIndex(id=shippingMethod,1)

waitForNotSelectedLabel

Waits for none selected option of the given select element matching the argument pattern as label. Any element locator can be used for the target. If no element is found, the found element is not a HTML select or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: waitForNotSelectedLabel(id=shippingMethod,UPS)

waitForNotSelectedValue

Waits for none selected option of the given select element matching the argument pattern as value. Any element locator can be used for the target. If no element is found, the found element is not an HTML select element or has no options the test is aborted with an error. Any text matching approach can be used for the command value.

Example: waitForNotSelectedValue(id=shippingMethod,UPS)

waitForNotText

The exact inversion of waitForText (considering the behavior, not the syntax).

Example: waitForNotText(id=current,regexp::.* rainy.*)

waitForNotTextPresent

The exact inversion of waitForTextPresent (considering the behavior, not the syntax).

Example: waitForNotTextPresent(weather forecast)

waitForNotTitle

The exact inversion of waitForTitle (considering the behavior, not the syntax).

Example: waitForNotTitle(exact:Product Details Page)

waitForNotValue

Waits for the targeted element to have its value not matching the command value. Aborts the test with an error message if the timeout is reached and no element is found for the given element locator or the value of the found element does match the command value.

Example: waitForNotValue(name=surname,Doe)

waitForNotVisible

Waits for the targeted element to become unvisible. Aborts the test with an error message if the timeout is reached and no element is found the given element locator or the found element is visible.

Example: waitForNotVisible(id=DialogOverlay)

waitForNotXpathCount

The exact inversion of waitForXpathCount (considering the behavior, not the syntax).

Example: waitForNotXpathCount(id('lines'),5)

waitForPageToLoad

Waits for the page to load. Aborts the test with an error message if the timeout is reached and page loading has not been completed. Has neither a target nor a command value.

Example: waitForPageToLoad()

waitForPopUp

Waits for the first pop up to be loaded completely. If no pop up is loaded before the default timeout, the test aborts with an error.

Example: waitForPopUp()

waitForPopUp

Same as the previous waitForPopUp but instead uses the given value as timeout.

Example: waitForPopUp(,5000)

waitForPopUp

Same as the no parameter version of waitForPopUp but instead waits for the targeted pop up. Any window locator can be used.

Example: waitForPopUp(name=popUpConfirmation)

waitForPopUp

Same as the version of waitForPopUp that only has a target but also has a value that is interpreted as timeout to use. Any window locator can be used.

Example: waitForPopUp(name=popUpConfirmation,5000)

waitForSelectedId

The exact inversion of waitForNotSelectedId (considering the behavior, not the syntax).

Example: waitForSelectedId(id=shippingMethod,UPS)

waitForSelectedIndex

Waits for the index of at least one selected option of the given select element matching the argument index. Any element locator can be used for the target. If no element is found, the found element is not an HTML select element or has no options the test is aborted with an error. Any text matching approach can be used for the command value but obviously it only makes sense to give integers as arguments. The indices start with 0.

Example: waitForSelectedIndex(id=shippingMethod,1)

waitForSelectedLabel

The exact inversion of waitForNotSelectedLabel (considering the behavior, not the syntax).

Example: waitForSelectedLabel(id=shippingMethod,UPS)

waitForSelectedValue

The exact inversion of waitForNotSelectedValue (considering the behavior, not the syntax).

Example: waitForSelectedValue(id=shippingMethod,UPS)

waitForText

Waits for the targeted element to have its text matching the command value (text matching is done the same way as in assertNotText). Aborts the test with an error message if the timeout is reached and the text does not match the command value. Accepts any element locator and any text matching approach.

Example: waitForText(id=registerButton,Register)

waitForTextPresent

Waits for the current page’s text to have its text matching the command value. Aborts the test with an error message if the timeout is reached and the text does not match the command value. Accepts any text matching approach. Like with assertTextPresent it is sufficient for the page to contain a text snippet such that the snippet matches the command value.

Example: waitForTextPresent(exact:Registration completed)

waitForTitle

Waits for the page’s title matching the command value. Aborts the test with an error message if the timeout is reached and the title does not match the command value. Has no target and accepts any text matching approach in the command value.

Example: waitForTitle(exact:Registration completed)

waitForValue

Waits for the targeted element to have its value matching the command value. Aborts the test with an error message if the timeout is reached and no element is found for the given element locator or the value of the found element does not match the command value.

Example: waitForValue(phone,regexp:\d+)

waitForVisible

Waits for the targeted element to become visible. Aborts the test with an error message if the timeout is reached and no element is found or the found element is visible.

Example: waitForVisible(id=minicart_items)

waitForXpathCount

Waits for the page to have exactly the number of elements matched by the command value. Aborts the test with an error message if the timeout is reached and the number of elements that can be identified using the locator does not match the command value. Accepts only XPath expressions as locators.

Example: waitForXPathCount(//*[contains(., ‘offer’)],3)

Commands That Store Data

In this section we explain the store commands. These commands are used to store some data in variables such that it can be reused later on. For example, you may click on a link pointing to the details of a specific product in some online store. To check that the details page for the correct product is shown you need the product name from the previous page. If you define a variable productName with a store command you can use it later on by typing ${productName}.

Note that for all store commands the variable name is given as the command’s value. Hence for the simple store command the variable value, and for the storeEval command the expression to evaluate, is given as command’s target.

store

Stores the text given as command’s target in the variable which has the command value as name. This can be used to define a value once and use it at several places. Furthermore, if you use random numbers, you may need a possibility to access the randomly generated value again for checks (calling for a random again will only give the same value by coincidence).

Example: store(MICHAEL,foo)

storeEval

Has no real target but instead interprets it as JavaScript which is executed. Uses the command value as name of the variable in which to store the result of the JavaScript execution. Note that this will only have any effect if JavaScript is enabled during the test run!

Example: storeEval(SOME_JAVASCRIPT,foo)

storeText

Stores the text of the targeted element in the variable which has the command value as name. Any element locator can be used.

Example: storeText(id=cartBalance,foo)

storeValue

Stores the value (in case of a <textarea> the contained text) of the element identified by the given locator in the variable which has the command value as name.

Example: storeValue(id=cartBalance,foo)

storeXpathCount

Stores the number of elements identified by using the locator in the variable with the command value as name. Interprets the whole target as XPath which should be used to identify elements.

Example: storeXpathCount(id(‘emptyDiv’), foo)

Miscellaneous Commands

close

Closes the browser window, so aborts the test (if the browser has tabs remaining the test just pauses but it does not make any sense to continue the test). This is likely to be changed such that the close command may only be called as the last statement in a test. Has neither a target nor a command value.

createCookie

Creates a cookie. This command is exceptional such that it neither has a “normal” target nor a “normal” command value. Instead it takes two arguments whereas the first one has to be in the form key=_value_ and the second one can be max_age=an integer or path=a path or an arbitrary other expression. If the second argument has the form max_age=an integer the cookie is stored with an expiration of the given integer value as seconds in the future. If the second argument has the form path=a path, the cookie will be visible to a path (default is /) only. If the second argument has another form, it will be ignored.

Example: createCookie(stokkeMovieShown=true,max_age=123456789000)

deleteAllVisibleCookies

Deletes all cookies for the current domain. It has neither a target nor a value.

Example: deleteAllVisibleCookies()

deleteCookie

Deletes the cookie having the first argument as name. Similar to createCookie this instruction has two arguments. Currently the second argument is ignored, the first one has to be a valid cookie name.

Example: deleteCookie(stokkeMovie,isIgnored)

open

Opens the URL given as command value. It has no target. The URL can be relative or absolute.

If you specify a relative URL, it will be resolved using the currently active base URL which is usually the test case’s base URL.

Example: open(article.php?story=2012020309143182) (where the base URL is http://www.groklaw.net/)

pause

Causes the test to pause the given amount of time in milliseconds. This command hass no target. The time to pause has to be given as an integer number as command value.

Example: pause(5000)

selectFrame

Switches to the targeted frame. Accepts any frame locator. Furthermore, you can simply use the frame’s name or ID. The relative approaches will never cause the test to abort with an error. The other approaches do so if no frame is identified using them.

Example: selectFrame(index=1)

selectWindow

Similar to the selectWindow command below but has no parameter. Instead it searches for the window that has no name or no title which is often the top level window. Aborts the test with an error if there is no such window.

Example: selectWindow()

selectWindow

Switches to the targeted window. Accepts any window locator. Aborts the test with an error if no window is found using the given window locator.

Example: selectWindow(title=customerAccount)

setTimeout

Sets the timeout used by the ...AndWait and waitFor... commands. The new timeout (in milliseconds) has to be given as an integer number as command value.

Example: setTimeout(30000)

Command Short Reference

While the previous section aims at providing a thorough understanding of the commands and how they work, this section provides a short reference for commands. For this purpose we give some tables which give the commands in more formal way to achieve a concise presentation where the following abbreviations/terms are used:

• “EL” means any element locator accordingly for “OL” (any option locator), “WL” (any window locator) and “FL” (any frame locator). If there are further restrictions, for example, just some of the element locators are possible, we give the allowed locators by their identifier as “name”, “link” and so on.
• “Text matching” means any of the text matching approaches and accordingly to the locators “glob”, “exact” and “regexp” are used if only a subset of them is possible.
• “Integer” means an arbitrary integer in the range 0 to 2147483647, “JS expression” means a valid JavaScript expression, “XPath expression” means a valid XPath expression (do not mix it up with the XPath element locator!).
• “Variable” means an arbitrary variable name. A valid variable name is a non-empty string matching the regular expression: [A-Za-z]([A-Za-z0-9_])*
• “Arbitrary” means an arbitrary text (for example, so,e text that should be typed into a form). While it may sound confusing, “Arbitrary” does not mean that there aren’t any restrictions. So, for example, a web application may require a password to have a minimum length or the text is interpreted as a path on a file system.

Interacting Commands

Assert Commands

Wait Commands

Store Commands

Miscellaneous Commands

Using the XLT Scripting API

Coming thus far, we have already seen some examples for the commands in action:

• xpath=//a[contains(@href,‘1423033.html’)]
• assertTextPresent(exact:Any Questions?)
• clickAndWait(link=Samsung)
• pause(5000)

However, this notation neither matches the Script Developer, nor the Scripting API. The Script Developer provides a GUI with fields in which the parts of the commands can be entered and fields that are disabled since they are invalid for the chosen command. So depending on the command in question you may not be able to enter a target or a value.

Using the Scripting API you are writing Java code where strings have to start and end with double quotes, meta characters have to be escaped and statements must end with a semicolon. However, we ignored all this and chose a notation close to Java as we think of this as the best readable compact notation that is intuitively understandable without giving lengthy formal definitions.

For example, consider the command: assertTextPresent(regexp:“\\home”)

This command checks if the text "\home" is present on the current page. When using this in Java, you have to make three things:

1. You have to escape characters appropriately. In this case, it means that you have to escape the double quote AND the backslash characters appropriately.
2. The whole argument string has to be enclosed into double quotes.
3. The Java statement has to end with a ‘;’.

Doing all this you end up with (added characters are red): assertTextPresent("regexp:\"\\\\home\"");

Furthermore, in the graphical user interface of the XLT Script Developer you won’t see any opening or closing parentheses for commands. In the Scripting API, however you’ll have to use parentheses since commands are actually calls to Java methods.

Since scripts are stored as XML files, each command has an appropriate XML representation where the name attribute specifies the type of command, the target attribute the command’s target and the value attribute the command’s value. If the command doesn’t have a target or value, the appropriate XML attribute is omitted.
The XML representation of the command above would look like this:

<command name="assertTextPresent" value="regexp:&quot;\\home&quot;"/> 

Note that the double have been escaped in XML.

Appendix

Script Developer Keyboard Shortcuts

Terminology

• Action: An Action represents a self-contained logical step in a test case. Typically, actions are re-usable building blocks which can be used across multiple test cases. Actions themselves may issue one or more requests.
• Agent: An Agent simulates a group of virtual users, which repeatedly execute certain test cases against the system under test.
• Agent Controller: An Agent Controller controls one or more agents.
• Master Controller: The Master Controller controls all agent controllers and gets the results, the logs and the pages generated by the agents from the storage of the agent controller machines.
• Property File: Property Files are text files that contain Java properties. A Java property represents a mapping from a symbol (property name) to a value (property value).
• Request: Typically, a Request represents one call to a (remote) server. It does not matter what protocol is used to contact the server.
• System under Test: The System Under Test is the application being tested. In the XLT demo, for example, this is the Pebble blog application.
• Test Case: A Test Case is the program that models a transaction.
• Transaction: A Transaction represents one execution of a certain test case. Typically, a transaction uses multiple actions to model the test scenario.
• User: A User is one incarnation of a client that interacts with a server. During a load test, multiple users will operate in parallel to simulate a multitude of human or technical users interacting with the target system at the same time. Typically, users are configured to execute a certain test case repeatedly.
• Virtual User: See User.

Acknowledgments

• This product includes software developed by Andy Clark.
• This product includes software developed by Simon Brown.
• This product includes software developed by The Apache Software Foundation.
• This product includes software developed by Caucho Technology.
• This product includes software developed by Gargoyle Software Inc..
• This product includes software developed by Harald Kirschner.
• This product includes software developed by The jQuery Team under MIT license.
• This product includes software developed by Alex Gorbatchev.
• This webpage features jQuery LightBox developed by Warren Krewenki published under BSD license.

Copyright © 2013 by Xceptance Software Technologies. All rights reserved.