On every test automation project we group the regression tests into a logically structured testset. A well structured testset is more easily maintained and integrated with a Continuous Integration server like Jenkins.
What dimensions are useful to group tests?
Here are a few dimensions that we find useful for grouping our tests.
First of all, it makes sense of course to structure along a functional breakdown. As far as tests cover a certain singular function or module, the test structure should reflect this modular structure of your SUT.
Another grouping can be along non-functional concerns that cross-cut your system. These might be integration points with third-party systems, places where email is sent, multi-linguality, batch processing, authentication/authorisation etc.
The dimension that we find the most useful to group tests along, is the usage context. This is determined by characteristics like running lead time, data dependencies, resource utilization and the extent to which the tests changes the application data.
XML2Selenium has the possibility to tag test which we use to make groupings along the functional and non-functional dimensions. We use the third dimension, the usage context, as our primary divider for the tests.
Divide by usage context
We find the following distinction useful:
Cycle 0 These tests are sanity tests that are typically run after every patch or mere system reboots. They check if all subsystems are up and running. The tests does not modify any data, and don’t take a long time to run.
Cycle 1 These tests are run less often but do a more thorough check of the main usage scenario’s. They take longer to run and depend on certain preset data. Still, they don’t use too many system resources and keep data manipulation limited.
Cycle 2 These are more sophisticated tests that take a long time to finish and have a heavy dependency on data. Typically these are reports and asynchronous data manipulation. Because these rests require a data preparation and can potentially diminish the system’s performance, these tests are typically run less often and need careful preparation and planning.