Behaviour Tests as a Product

Test Automation for Long Running Release Cycles

Follows from Water-scrum-fall

While build-once/deploy-many software releasing is well established, test automation is commonly executed directly from source, sometimes with branches to cover variations. As an alternative, I encourage packaging (versioning) test automation. As a reusable product, tests can be re-run from your artefact registry without exposure to the transient nature of source control.

alt text alt text

Package Management

A stand alone package registry can be used as described above, or if using an integrated tooling offering, the registry included can be used.

alt text alt text

Stability Tests

With the versioned automated test product, the package can not only be consumed within delivery pipelines, it can also be used for scheduled tests, for example nightly tests. While new features and corresponding tests are progressing through the delivery lifecycle, the nightly tests can still consume a known version with confidence.

alt text alt text

Implementation Patterns with CDAF

One of the key features of CDAF is that packages are abstracted from the implementation ecosystem, e.g. NodeJS, .NET, Python, etc. and a single, self-extracting script is the output of the CI process. It is this which is published in the registry, and executed by reference to perform automated tests.

Note: if your automated test product is a container image, the release package script will provide a wrapper to execute the container based test and capture the output.

Incorporation with a Release Pipeline

Test Product(s) can be included as part of the component manifests in the following release disciplines.

  • Declarative Release for implementation examples, which incorporate intermediary tools such Ansible Tower, Puppet Enterprise and Terraform Cloud
  • Release Trains for implementation examples, which incorporate Octopus Deploy and Azure DevOps Release feature