Exploratory Testing

Dennis Joele is working for TriOpSys as test designer and has, as such, set up and performed different tests for de Royal Navy Hydrographic Service (Dutch: Dienst der Hydrografie van de Koninklijke Marine), based upon structured testing. In this article, he will describe a practical example in which Exploratory Testing was selected. Since both manners have been used within the same project, it is possible to draw a comparison. The results of this comparison led to a customer asking: ‘Why is this not a customary process?’ This question will also be answered.

Usually, we use structured testing. When using structured testing, a couple of conditions have to be met. An adequate documented test-base and sufficient preparation time for the specification of the tests have to be available. If one of these conditions is not met, a different test technique is available. This technique is known as Exploratory Testing.

The technique behind Exploratory Testing

According to one of the founders of the technology, Cem Kaner, Exploratory Testing can be defined as: ‘Any sort of test in which the tester prepares the test design during the test, and the information that is obtained during testing is used to test new and improved test cases’.

This implies that a number of steps have to be performed: a part of the system has to be explored (exploring), a consideration of what should be tested has to be made (test design) and the intended test has to be performed (test execution).

There are a couple of disadvantages: the repeatability of the test may be difficult, the chosen test technique during the test design is not visible, there is no test coverage and the direction of the testing is not known in advance. To resolve these issues, James Bach has upgraded the technique with a number of elements, which has resulted in ‘session based Exploratory Testing’: a test charter, a session, notes and a debriefing.

Test charter

A test charter describes the tasks and goals that have to be implemented, as shown below:

  • What (scope)
  • What not (not within scope)
  • Why (questions to be answered)
  • How (brainstorm)
  • Expected Problems
  • Reference (diagrams/models)

The test charter may also include the test techniques to be used and the test coverage to be expected. The test charter provides a general record of the secondary conditions.

Session

A session takes place within a certain period of time, for example 2 hours in which the system is tested using the predefined test charter. Such a session is often performed by two people, for example a tester and a domain expert. The tester ensures that the (implicit) testing techniques are used and the domain expert completes the test cases from his/her system and domain knowledge.

Notes and debriefing

In order to reconstruct the manner in which the session has gone, during the session notes will be made. These include both the test design that has been created during the test and the test results. Findings will be noted as well. This improves the repeatability of the tests. After the session, the notes and findings will be reviewed with person who requested the test charter. Subsequently, it will be reviewed if the questions of the test charter have been answered.

When to implement

In certain circumstances, it is possible to use Exploratory Testing. Some of these circumstances are listed below:

  • There is insufficient documented test base available
  • There is insufficient time available to specify the tests
  • An addition to/diversification of formal testing techniques is desirable
  • There are testers available with sufficient test knowledge

For example, the technique may also be used to figure out quickly how a particular system works, to investigate the quality of a system quickly or to examine a specific result in more details.

When not to implement

If one of the aforementioned conditions I met, it does not automatically mean that Exploratory Testing may be applied directly. Of course, there are situations in which the technique may not be suitable, such as:

  • The client requests demonstrability of the tests performed
  • Testing has to be repeated by others or with a test tool
  • Critical functionality that could cause damage if not functioning
  • If tests need to be specified more extensively
  • If tests lack direct feedback, such as batch functionality

Exploratory Testing at the Royal Navy Hydrographic Service

The project in which Exploratory Testing has been applied concerns the construction of a new civil and military production system for creating nautical charts on paper or electronically. The system consists of several components, which are partly purchased externally and partly developed internally. These components are integrated into one unit where Workflow Management is the overarching layer. This is to ensure the management of the execution of the business processes. In total, five business processes were automated successively. Structured tests had already been executed for the first business process.

Exploratory Testing has been introduced as addition to / diversification of the formal technique used, specifically the Process Cycle Test. This test technique focuses on blocking out the variations in the process. Variations within the process are not blocked and when using Exploratory Testing this is possible.

Exploratory Testing may only be used if testers are available with sufficient knowledge. Usually, this would be enough. However, the hydrographic field is highly specialized, and the knowledge regarding the subject matter was limited. To solve this, an end user was involved in the test execution.

These two circumstances were the basic assumptions upon which was decided to use this technique.

Results

In a time period of two hours, a total of 11 results have been found. A total of 52 findings were made prior to the Exploratory Testing of the relevant sub-system. These findings emerged during the Performance Functional Acceptance Test (FAT) and User Acceptance Test (UAT). Both tests took approximately two days; one person executed the FAT and three people were included for the UAT.

Effectively, Exploratory Testing thus provided a Findings Frequency of 11 findings / 4 man-hours = 2.75 findings / man-hour. This calculation is accurate because the entire time was spent effectively on the testing itself. If the time spend on compiling the test charter is included, the Finding Frequency becomes 11 findings / 4.5 man-hour = 2.44 findings / man-hour.

For the previous period, in which structural testing was used, a calculation can be made as well, although this may be less accurate. The Findings Frequency becomes 52 findings / 60 man hours = 0.87 findings / man hour. A factor of 0.75 is used to calculate the effectively spent time. In this calculation, the time required to compile the Software Test Description (STD) and the reviewing of this has not been included. If this is set on 2 Mondays, the Findings Frequency reduces to 0.68 findings / man-hour.

Translation: Structured Testing. [red] Number of findings per spend man-hour (excluding effort for documentation). [purple] Number of findings per spend man-hour (including effort for documentation).

Dividing both Findings Frequencies results in a factor of 2.75/0.87=3.16 (excluding effort for documentation) and 2.44/0.68=3.59 (including effort for documentation). Exploratory Testing appears to be more effective than the previous tests. This can be stated because the deviations in both calculations should be greater than the factor, which is not likely. The second calculation shows the effect of the time required for the preparation of documentation clearly. This results in a higher decrease of the Effective Findings Frequency of Structured Testing than for Exploratory Testing.

This calculation only includes the number of findings, and not the origins of those findings. Of the results from the FAT and UAT, eight were classified as ‘Blocking’ or ‘Major’, versus three findings in the Ex[ploratory Testing session. This leads to the following Finding Frequency: 3 findings / 4.5 man-hour = 0.67 findings / man-hour and 8 findings / 76 man-hours = 0.11 findings / man-hour. However, it should be mentioned that two of the findings were caused by regression. If these are excluded from the calculation, Exploratory Testing is still twice as effective. Regardless, if both methods have to be equally effective, structured tests should yield more than 16.8 times as many results in this particular case. This is not very likely in this situation.

Translation: Structured Testing. [red] Spend man-hours excluding effort for documentation. [purple] Spend man-hours including effort for documentation.

Conclusion

When the results of Exploratory Testing and Structured Testing are compared, it can be stated that for Exploratory Testing the number of findings per spent man-hour is three times higher. This observation is based upon 76 spend man-hours for structured testing versus 4.5 hours spend for Exploratory Testing.

As a result, Exploratory Testing only uses 1/3 of the spend man-hours to achieve the same number of findings. However, several sessions are required. Because of this, Exploratory Testing is more effective than structured testing.

After reporting the results of the session, a simple but effective question arose from the direct stakeholders, the head of the Technology and Development Department. ‘Great e-mail, why aren’t we using Exploratory Testing normally?’.

Of course, Exploratory Testing may not be suitable in all situations. This may be the case when high-risk functionality has to be tested, or if standards require the demonstratiblity of the tests performed. However, if the conditions for Exploratory Testing are met, this can be a very effective manner of testing.