We all know the value of software testing. But in the current era of the software development industry, we can see that the fast-paced software development environments can create time and cost constraints that make it difficult to thoroughly test an application prior to a release. If defects slip undetected, into the production environment, the result can be customer dissatisfaction and increased maintenance costs.
In this situation, test automation comes into the picture. It allows your team to execute more tests in less time, increasing coverage and freeing human testers to do more high-level testing.
Automation is especially beneficial for test cases that are executed repeatedly, such as those for cross-browser and cross-device compatibility and those that are part of a full or partial regression suite.
So, let us see automation testing approaches with their pros and cons to see how it will be suitable for relevant scenarios.
1. Record & Playback
This is the most basic automation testing approach, with limited flexibility and long term utility. Record & Playback functionality allows testers to interactively record user actions and replay them back any number of times, comparing actual results to those expected. These techniques use only hard-coded data, the utility is very dependent on the automation tool used. Record & Playback can be used by non-coders if customization is kept at a minimum level and the verification points are graphical.
Usage
Record & Playback is very useful for basic automation projects of short duration like automated data entry or exploratory testing that feature re-record from scratch, have short test flows and automated data entry.
Pros
Record & Playback is an ideal way to get into testing automation. This approach has several advantages including low development cost, relatively simple design and ease of deployment.
Cons
The basic nature of Record & Playback comes with a functional price. This technique has limited performance (e.g.: minimal test flow coverage), robustness (e.g.: can break on build) and flexibility. Ongoing maintenance costs can be high especially when manual intervention is required.
2. Record & Playback Enhanced
This framework provides more functionality, supports parameterized data and enables more use cases than the basic Record & Playback approach. However, its value is highly dependent on the automation tool employed and the testing and programming skills of the team. A good return on investment can be obtained within a single project.
Usage
Like standard Record & Playback, this approach is useful for testing projects with modest automation requirements featuring: short term tests with short flow test cases and a limited number of checkpoints and relatively stable environments.
Pros
Record & Playback Enhanced has some advantages including low-cost development and deployment (i.e. limited programming needed).
Cons
Like its relative Record & Playback, this approach has some drawbacks including modest performance (e.g., limited test flow coverage) moderate usability and flexibility with no multi-environment support, high maintenance costs due to manual intervention (for analysis and validation) and the need to address checkpoints.
3. Data-Driven Frameworks
Data-Driven techniques involve the creation of test scripts to run together with their related data sets within a framework. The test environment settings and controls are not hard-coded. A Data-Driven framework is ideal when the workflow is repetitive but the data input is large – meaning different combinations of data are required to successfully test the application. A skilled team is needed to implement this approach.
Usage
This automation framework effectively satisfies more advanced testing needs like testing single applications in multiple environments with big data sets, a limited number of checkpoints and stable test cases. Batch run tests are possible with the extra development effort. Decent support is provided for distributed testing teams.
Pros
Data-Driven frameworks have several benefits including very good usability and reusability of test scripts, solid test flow coverage and reproducible test results.
Cons
There are watch-outs in using these methods. Automation performance is based largely on the automation skills of the person(s) undertaking the implementation. These techniques also require regular maintenance (increasing costs) and manual intervention upon failure.
4. Keyword-Driven Frameworks
A Keyword-Driven framework is an advanced type of automated testing. Its methodology divides the test development process into two distinct stages a design & development stage and an execution stage. A Keyword-Driven approach is ideal for many projects, environments and data sets. However, it also requires advanced coding skills plus an upfront investment in development time and tools to maximize value. The design of the framework can play a major role in determining the return on investment.
Usage
This testing method is suitable for many automation testing projects across multiple applications, platforms and environments, featuring large data sets, short & straight test cases and a limited set of checkpoints. Keyword-Driven frameworks are also ideal for batch run tests.
Pros
Keyword-Driven frameworks are a very good approach to leveraging automation. They have many advantages including very good script usability and re-usability as well as test flow coverage and the ability to reproduce test results. Moreover, test development does not require sophisticated programming skills.
Cons
Higher performance does come with a cost including a significant upfront investment for the design and implementation of the framework. Keyword limitations restrict the complexity of test cases. Employees will require knowledge of meta-languages. These techniques also require regular maintenance (increasing costs) and manual intervention upon failure.
5. Hybrid Framework
Hybrid frameworks are the most complicated types of automated techniques available. For example, they feature script-less test creation, the ability to import/export data from various sources, a variety of checkpoints and the power to integrate external objects. They can generally accommodate grammar and spelling as well as information input. However, the upfront investment, specialized skills prerequisite and ongoing maintenance costs can limit its long-term value and applicability for simple projects.
Usage
Hybrid Keyword Frameworks are designed to satisfy an organization’s most wide-ranging automation requirements covering multiple applications, platforms and environments. They are suitable for testing situations with large and changeable data sets and data transitioning cases. These frameworks typically provide strong support for distributed testing teams.
Pros
Hybrid Keyword Frameworks are among the most comprehensive and flexible functional testing frameworks available. They feature high usability, re-usability and test flow coverage and offer built-in consistency and severity validation.
Cons
There are some challenges with this approach including the need for significant upfront investment in development and skills for design and implementation. Ongoing maintenance costs could also be noteworthy, potentially limited ROI. Finally, these frameworks may also be an overkill for many testing projects.
The above summary offers a general rule of thumb on which approach to use. What method you conclusively choose will depend on your budget, timing, skills and environment as well as your project’s testing goals. Fundamentally, test automation is a software development activity where your team will need solid design and programming practices plus quality tools. A low-risk way of getting into test automation is by building a smoke test that will quickly run for each new build. This strategy allows you to learn automation as you go and to better understand your organization’s capabilities as well as having an automation testing approach in the company.