Syntactive Blog

Qualities of a top tier test framework

Introduction

So you want to write a test framework to automate your testing effort? That's an excellent idea but what should you keep in mind? When building a test framework it is important to pay attention to serveral key elements to avoid having to refactor your code constantly and get into maintenance hell. Test automation, unlike classical development frameworks, are more reactive to the elements that need to be tested and the changes inside the applications under test. You're often a step behind and will need to quickly and efficiently add requirements as they are needed. It's therefor crucial that you keep a few things in mind, so lets dive in with our first point below.

Easy to use

A good framework needs to be easy to use by various (test)engineers within your organization. Typically the skill levels can vary greatly so it must adhere to anyone who needs to use it. Below are a few concrete points about ease-of-use to keep in mind when building the framework :

• Lightweight, avoid unneeded dependencies wherever you can. Less functionality means less things that can break.
• Easy to install/deploy on fresh environments. Pay attention to cross platform compatibility and containerization. This will help other engineers getting started and also allow you to scale up testhosts when needed.
• Modularity. Keep your scripts minimalistc and only import necessary components at runtime
• Reduce user input/local configuration(s) for running tests
• Shared secrets and sensitive data should be managed in a secure environment and (remotely) accessible by scripts
• Provide meaningful logging output in a clear (formatted) way. Debugging should be made as straightforward as possible.
• Shared committed Configuration files for common configurations (profiles)

A highly complex test framework that is too difficult to use by your peers, or even yourself in the future, is bound to encounter a lot of resistance and lose its edge overtime.

Maintainability, extendability and reusability

Like most software projects, a test framework is alive and needs to constantly evolve to follow the product under test. When new features are added to be tested, you might need to redesign or extend a part of your test framework. Or when another team within your organization would like to adopt your framework, they might need additional functionality. Here are the main points to keep in mind when sharing your framework across a broader team or organization.

• Standardized code and project structure
• Multi layered logic, to separate testing code from product interaction logic. Testcases should be language and environment agnostic.
• Separate your test data to allow data-driven testing
• Follow proper versioning strategies by using GIT rules, hooks, merge requests and branching
• Integration-friendly, various reporting options to report to distributable File or Test Management System (TMS)
• Support external test scripts (e.g. BDD, gauge cases defined in xRay)
• Allow mocking components during test development

By following proper coding practices and keeping your structure clear, you create a strong foundation for the testing framework that will allow resuability out of the box. Do also not be afraid to throw away 'dead code' when it is not applicable anymore. This reduces the clutter inside your framework and keeps it easier to maintain.

Stability

We've brought up the subject of stability briefly in the previous sections, but it's important to emphais on it. An unstable test automation framework is going to cause you more harm than good. You don't want to be spinning your wheels debugging and fixing testcases every release to the point it takes more time than manual testing. Typically testcases become more robust as they run more often and regularly, for example in a Ci/Cd pipeline. You need to be able to comfortably rely on your tests when a pressing release is knocking at your door. Here are a few tips to increase your test stability :

• Run your automated tests regularly, preferably with different, but compatible, data
• Preliminary checks/validations can be executed before performing heavy test operations
• Attempt to recover from recoverable errors, but log them for a later review
• Fail fast if unrecoverable error occurs and provide clear error message indicating failure(s)
• Make tests/suites independent of each other. They should run in any order

At the end of the day you write test automation to speed up the testing process and to reduce the repritive nature of manual testing. In the end the tests can only be as stable as the product under test itself. If the environments are unstable and unreliable, your will have a hard time to create meaningful test automation.

Performance

Whilst this might not be the most critical aspect of test automation, performance and efficiency is an often overlooked part of test automation. Faster feedback during a testing cycle means faster confidence in the product or more time to resolve discovered issues. Everytime a test is run, the time it takes to complete is cumulative. You want to avoid testing becoming the bottleneck of the release process as much as possible by having quick and reliable testcases.

• Avoid executing unnecessary steps (e.g. repeating heavy preconditions on multiple runs)
• Avoid (hardcoded) implicit waits (like time.sleep(xx)) inside your testscripts. Using polling with a max wait time instead
• Keep paralelization in mind from the start, avoid dependencies on global variables amd resources
• Run your tests on performant machines, wether they are physical hardware devices or virtual machines/conainers

Automation performance is often overlooked but plays a curcial role, not only during test execution, but also during test development and debugging. That's why it's important to optimize your tests to keep the time and effort costs low.