Once a Testcase has been created, the next step is to execute it. The execution is started by selecting a Testcase, TestGroup, or TestSuite from the hierarchy view, and then using the "Start Run" menu option. An individual Testcase may be executed at a time, or a single TestGroup (meaning that all Testcases contained within the group will be executed sequentially), or an entire TestSuite (meaning that all Testcases contained within the suite will be executed sequentially). The execution of one or more Testcases creates a new Run.

When a new Run is created, the user will first be asked to supply a name for the run, and a location on disk where the result files will be stored. The name and the location must be unique among all Runs that the workspace holds, although the application will supply default values for both of these.

The Runs tree will gradually be filled in as each Testcase executes. The Runs tree will display the testcases in the same hierarchical arrangement as they appear in the TestSuite. An icon adjacent to the Testcase will indicate its status. The Testcases will also be displayed in a colour that corresponds to their status.

If the user selects the root of the Run on the hierarchy view, an editor will display the progress of the run (number of Testcases executed and the pass/fail statistics of each). This progress information can be viewed either as a table, or as a pie chart, with colour-coding used to indicate the status of each testcase. A Cancel button and menu option are also provided to abort the execution.

When the execution of a Testcase has completed, it will be shown in the Run hierarchy view. If the user selects a Testcase from this view, an editor will be shown that will display the following information:

The name of the Testcase, and the location of the results files will be shown. The results files are any files that were created or modified by the Testcase execution. A View button is provided to open a file manager application on the results files.

An execution log file is also stored in the results location. This is the log of all execution procedures for the Testcase. Logging information that is relevant to the execution is written to this file. A View button is provided to open a text file viewer on this log file.

There is also a verification log file, which records information from the verification procedures. For example, file differences found when comparing against benchmark results are stored in the verification log file. A View button is provided to open a text file viewer on this log file.

The execution status of the test is shown. This will be one of the following values:

Completed The Testcase executed successfully. Note that this does not imply that the Testcase has passed or failed. It merely says that the execution procedures have completed without problems.

Failed The Testcase has failed to complete. In this case, the verification procedures will not even be run. This implies that there was a problem in configuring the execution procedures. It is possible that there is a required property that has not been set, or has been set to an incorrect value. The execution log file will contain information as to why the failure has occurred.

Cancelled The user cancelled the execution

Unknown An unknown problem has occurred with the Testcase. Again, the execution log file should be used to determine the nature of the problem

The verification status of the test is also shown. This will be set to one of the following values:

Successful All of the verification procedures have concluded that the Testcase has passed.

Unsuccessful At least one of the verification procedures has concluded that the Testcase has failed. The verification log file will indicate the nature of the failure.

No Benchmark This Testcase does not yet have any benchmark results that are required to perform the verification. For example, a verification procedure may be used to compare the contents of a result file with the "benchmark" copy of that file. If no benchmark copy exists, then the verification procedure does not know whether or not the test was successful

Expected Fail All of the verification procedures have concluded that the Testcase has passed. However, this Testcase has been marked with the "Expected Fail" flag. This is used to indicate that the benchmark results for this Testcase are known to be incorrect, but the problem is not planned to be fixed until a later date. So the Testcase execution has produced the expected result, but the result is known to be wrong.

Expected Fail Differences At least one of the verification procedures has concluded that the Testcase has failed. The verification log file will indicate the nature of the failure. However, this Testcase has been marked with the "Expected Fail" flag, as described above. So this means that either a) this test did not complete correctly, or b) the original problem with the Testcase (which caused it to be marked as Expected Fail) has now been fixed, and that the application is now producing the correct results.

A button is provided to make the results from this execution into the benchmark result for this Testcase. The user is asked to confirm this operation, because it will destroy any existing benchmark files for the Testcase. This operation should be performed only when the user has confirmed that the results files generated are "correct".

When generating benchmark results, the user also has the option of setting the results to be Expected Fail. This is a standard technique in regression testing, when it is known that the application under test does not produce the correct results, but it is not planned to fix the problem until a later date. This enables the automatic pass/fail verification of a Testcase, but with a reminder to the user that the Testcase has not really passed, and needs to be fixed at some point. The user has the option of setting the flag at this point, when the benchmark files are being created. Alternatively, the flag can be set at any time using the Testcase editor panel.

Finally, the start and end time of the Testcase execution is also shown on the editor panel.