Tests as a Product

Declarative Deployment

For context see Behaviour Tests as a Product. The development of test automation is autonomous and is not tightly coupled with the solution development. When the solution is ready to execute a set of tests, the versioned package is obtained and executed.

Environment Definitions

Configuration Management within the source allows for test execution by simply invoking the test package for the desired environment, e.g. ./package.ps1 PROD, or targeted testing ./package.ps1 PROD_WEB

context  target      testSuite       featureSet  resources                                 secrets
local    PROD_CLASS  SpecFlow.Specs  data        supabase                                  creds_prd
local    PROD_WEB_1  Selenium.Specs  login       https://example.com/Calculator.html       vault_prd
local    PROD_WEB_2  Selenium.Specs  dashboard   https://example.com/Calculator.html

local    TEST_CLASS  SpecFlow.Specs  data        supabase                                  creds_tst
local    PROD_WEB_1  Selenium.Specs  login       https://test.example.com/Calculator.html  vault_tst
local    PROD_WEB_2  Selenium.Specs  dashboard   https://test.example.com/Calculator.html

The delivery phase of the pipeline pushes the package to a registry, with a semantic version. It is these versions that the solution delivery process can consume to provide predictable test results.

graph LR

  subgraph Component A
    Rbuild["Build"] -->
    Rtest["Test"] -->
    Rpublish["Publish</br>1.0.1"]
  end
  subgraph Component B
    Pbuild["Build"] -->
    Ptest["Test"] -->
    Ppublish["Publish</br>0.5.9"]
  end
  subgraph Test Product
    Sbuild["Build"] -->
    Stest["Test"] -->
    Spublish["Publish</br>1.2.0"]
  end

  subgraph Release
    TEST:::release
    PROD:::release
  end

  store[(Registry)]
  
  Rpublish --> store
  Ppublish --> store
  Spublish --> store
  store --> TEST
  TEST --> PROD

classDef release fill:lightgreen

In this example, current production tests are at version 1.1.0. While the next release of 1.2.0 is in progressing through the development lifecycle, the scheduled tests can still consume version 1.1.0 with confidence. The scheduled tests may have a reduced scope, i.e. web only, this is where the ability to select a subset is valuable, e.g. ./package.ps1 PROD_WEB.