Entry Points

CDAF Entry Scripts

CDAF providers 4 entry scripts for different purposes.

• ci : Build and Package only, i.e. Continuous Integration, mandatory argument is BUILDNUMBER
• cd : Release, i.e. Continuous Delivery or Deploy (depending on gating or not), mandatory argument is ENVIRONMENT
• cdEmulate : Executes ci and then cd, generates BUILDNUMBER if not supplied and uses configurable ENVIRONMENT
• entry : Executes ci and then cd, generates BUILDNUMBER if not supplied and uses configurable ENVIRONMENT(s)

The CI entry point (ci.bat/ci.sh) will perform the build and package process. Recommended configuration is to produce a self extracting deployable artefact (release.ps1/release.sh).

sequenceDiagram
  ci ->>+ buildPackage: BUILDNUMBER
  buildPackage ->>- ci: release

CD Emulation

The CD Emulation (cdEmulate.bat/cdEmulate.sh) is a simple wrapper which performs the CI process, and then executes the CD process, using the configured or default environment.

sequenceDiagram
  cdEmulate ->>+ buildPackage: BUILDNUMBER
  buildPackage ->>- cdEmulate: release
  cdEmulate ->>+ release: ENVIRONMENT
  release ->>- cdEmulate: deployment results

Entry

The entry wrapper (entry.bat/entry.sh), similar to cdEmulate, performs the CI process and CD process. The CD process however supports additional, optional, configuration for releases based on branch name (substrings).

sequenceDiagram
  cdEmulate ->>+ buildPackage: BUILDNUMBER
  buildPackage ->>- cdEmulate: release
  loop for each matching branch name
    cdEmulate ->>+ release: ENVIRONMENT
    release ->>- cdEmulate: deployment results
  end

Task Processing

Execution Engine

The CDAF execution engine allows the DevOps Engineer to focus on the primary objective, and not have to cater for logging, exception and error handling. Within this engine are a set of operations for common problems, further allowing a focus on results and rewriting/copying scripts. See Execution Engine documentation.

Note: The following is details a subset of CDAF capabilities, describing a common usage, For a detailed breakdown of each component, see the CDAF Reference Guide.

sequenceDiagram
  autonumber
  participant entry point
  participant buildPackage
  participant buildProjects
  participant execute
  participant package

  entry point ->>+ buildPackage: BUILDNUMBER
    Note right of buildPackage: "Build" Process Begins
    buildPackage ->> buildPackage: Property Translation (properties.cm)
    loop for each Directory containering build.tsk
      Note right of buildPackage: "Project" is a historical name <br/> from Eclipse & Visual Studio
      buildPackage ->>+ buildProjects: Project Name
      buildProjects ->>+ Transform: Load Properties
      Transform ->>- buildProjects: propertyList
      buildProjects ->>+ execute: build.tsk
      loop for each Line in build.tsk
        Note right of execute: Build commands, such as <br/> MSBuild, NPM, etc.
        execute ->> execute: log, execute and manage errors
      end
      execute ->>- buildProjects: build artefacts
      buildProjects ->>- buildPackage: build complete
    end

    Note right of buildPackage: "Package" Process Begins
    buildPackage ->>+ package: proejctName
    package ->>+ Transform: Load Properties
    Transform ->>- package: propertyList
    package ->>- buildPackage: project complete
    buildPackage ->>+ packageLocal: prepare release
      packageLocal ->> packageLocal: Gather CDAF SCripts <br/> and deploy properties
      packageLocal ->> packageLocal: Gather user defined artefacts
    packageLocal ->>- buildPackage: artefacts
    buildPackage ->> buildPackage: generate self-extract release
  buildPackage ->>- entry point: release.ps1 or release.sh

Configuration Mapping

Configuration Mapping

The local and remote configuration will trigger a task execution based on each unique declaration of context and target, using the corresponding default task tasksRunLocal.tsk and tasksRunLocal.tsk.

context  target  deployHost
remote   UAT     vm.example.com
local    UAT     vm.example.com
remote   PROD    vm.example.com
local    PROD    vm.example.com

Customer tasks can be defined for directories customLocal and customRemote respectively, or custom if shared.

context    target  deployTaskOverride
local      DOCKER  docker-compose-test.tsk
remote     UAT     on-premise-deploy.tsk
remote     PROD    on-premise-deploy.tsk
container  PUSH    publish-production-ertefact.tsk

Note that container override tasks are made available in the customRemote directory.

Execution Engine

Execution Engine

To alleviate the burden of argument passing, exception handling and logging, the execution engine has been provided. The execution engine will essentially execute the native interpretive language (PowerShell or bash), line by line, but each execution will be tested for exceptions (trivial in bash, significantly more complex in PowerShell).

Where is it used

In all places using .tsk files, i.e. build, package, wrap and deploy. The following operations are available to all tasks, however, some are more applicable to specific processes, see Build, Local and Remote task execution for more details of how these can be used.

Operations

The following operations are provided to simplify common tasks.

Notes on EXCREM use, the properties are similar to those used for remote tasks, where the minimum required is the host, if other properties are not used, must be set to NOT_SUPPLIED, i.e.

deployHost=localhost
remUser=NOT_SUPPLIED
remCred=NOT_SUPPLIED
remThb=NOT_SUPPLIED

Windows only

The following operations are only available in PowerShell version

Common Variables

These are automatically set at both build and deploy time.

Build-time Variables

These are automatically set at execution start-up

See also Environment and Global Variables.

Deploy-time Variables

Environment Variables

Environment Variables

Due to inconsistencies between Windows and Linux handling of environment variables, these have been divided between environment variables that are set before calling an entry script to alter the behaviour of CDAF, and environment variables that are set within bash scripts to make them globally available.

Control Variables

The following environment variables are available to control the behaviour of CDAF

Global Variables

These variables are available to child scripts, custom scripts and task execution engine, see also Build-time Variables.

See also Build-time Variables.

Image Registry

These override Solution Properties properties. They are used to push the image created by dockerBuild to pull a base image from a private registry.

These override Solution Properties properties. They are used to push the image created by imageBuild to push to a private registry.

These override Solution Properties properties. They are used to push the image created by dockerPush to push to a private registry.

Solution Properties

Solution Properties

CDAF.solution : file to identify a directory as the automation solution directory where the key configuration files are placed. This file is used as the bases of the manifest.txt file while is included in the resulting CI artefact package.

See solution/CDAF.solution in CDAF automation directory.

Solution Properties

Environment Variable Substitution

The following properties can be used in place of environment variables

Image Registry

These properties are used to push the image created by dockerBuild to pull a base image from a private registry. These can be overriden by Environment Variables.

These properties are used to push the image created by imageBuild to push to a private registry. These can be overriden by Environment Variables.

These properties are used to push the image created by dockerPush to push to a private registry. These can be overriden by Environment Variables.

Git Clean-up Properties

To clean-up Git branches and docker images, the following properties are used.

Extended Processes

Extended Processes

Optional Build Processes

By placing these files in your solution root, the processes will execute as described

Optional Package Processes

By placing these files in your solution root, the processes will execute as described

Feature Branch Execution

Feature Branch Execution

Place feature-branch.properties in your SOLUTIONROOT to allow dynamic delivery execution, based on Git Branch name. This capability is limited to entry.sh/entry.bat/entry.ps1, which are Git aware, and the recommended loose coupling entry scripts for CDAF.

# Separate environments for features and bugs
feature=DEV1
bugfix=DEV2

# Hotfixes deploy to all environments
hotfix=DEV1
hotfix=DEV2

See CDAF Samples for complete implementations in Linux and Windows.

Sensitive Data Strategies

Sensitive Data Strategies

Loose Coupling

A key approach to support the principle of automation execution in a local desktop context, is the use of environment variables. It’s important to remember that environment variables do not necessarily need to be persisted, i.e. stored unencrypted on disk, it’s the global availability of the variable that makes it an environment variable.

context  target  databaseFQDN        dBpassword
local    TEST    db1.nonprod.local   $DB1_PASSWORD
local    UAT     db2.nonprod.local   $DB2_PASSWORD
local    PROD    cluster.prod.local  $PRD_PASSWORD

Variable Expansion

Variables can be referenced in preoperties files (see [Configuration Management][mydoc_basics_configuration_management]) or CDAF.solution file and then expanded at deploy time into variables or files using ASSIGN, PROPLD or DETOKN in the execution engine.

Encrypted Files

This approach is to allow secrets in a file to be stored in source control. Encryption key for Windows is an EAS key, while for Linux it’s a GPG key. This approach is used when there are a large number of secrets to cater for, and therefore only the key needs to be managed as a secret.

In early generations of secret management, the secrets would be stored as persistent environment variables, however all modern toolsets provide an encrypted store which can load secrets as environment variables.

See the DECRYP & DETOKN operations in the execution engine for guidance on usage.

Cloud Storage Integration

Toolset providers who also supply public cloud provide integration to their secret storage offerings, while these can be convenient, this does couple your automation to toolset and makes the execution of locally challenging.

Legacy Features

Legacy Features

Remote Deployment

Machine to machine deployments are increasingly uncommon, as local agents/runners are readily available, making on-premise deployments from the build server an infrequent use case. While there is no plan to deprecate this capability, it’s complexity makes local testing i.e. shift-left complicated, especially in windows. For CDAF configuration, see Remote Tasks.

Windows Remote PowerShell

This approach uses the local host for both target (CD) and build (CI) execution. Provision the host with both roles

.\automation\provisioning\mkdir.ps1 C:\deploy
.\automation\provisioning\CredSSP.ps1 server

.\automation\provisioning\trustedHosts.ps1 *
.\automation\provisioning\CredSSP.ps1 client

Linux SSH

Generate PKI key and public certificate, and perform a loop-back connection to local host to place the public certificate in the authorised hosts configuration.

.\automation\provisioning\agent.sh deployer@localhost

Symetric Encryption

With the implementation of 12-Factor applications, secret management in files is less common, and the storage of encrypted files in source control for subsequent decryption is now uncommon. While this capability is not planned for deprecation, it is recommended to use sensitive data strategies instead.

Variable Validation

Variable Validation

There are 5 rules available, two for plain text and three for secrets. When validating a secret against a known MD5 value, either a literal or variable can be supplied. See VARCHK in the execution engine operations

# Plain text values
OPT_ARG                                # Optional plain text
terraform_version=required             # Required plain text

# Secret values
TERRAFORM_TOKEN=optional               # Optional secret
TERRAFORM_TOKEN=secret                 # Required secret
TERRAFORM_TOKEN=$TERRAFORM_TOKEN_MASK  # Required secret verified against supplied SHA-256 value

containerBuild

containerBuild

Some CI/CD pipeline toolsets support native capability (GitLab, BitBucket) to execute with a container. In other some cases, (CircleCI, Travis) all pipeline activity can only be executed within containers.

For toolsets which do not support this functionality, but do allow for self-hosted agents or where a self-hosted agent is preferred/mandated i.e. execution within a private network, the CDAF container helpers can provide consistency for construction, execution and housekeeping.

Even with a toolset uses containers, if they support docker-in-docker, the CDAF container helpers can still be utilised.

containerBuild

containerBuild

Master of Build Success

The containerBuild option allows the execution of the build process from within a container. Unlike toolsets which reference a image that is used to create the build container, CDAF uses a Dockerfile, for the following advantages:

• Build Prerequisites can be defined in code, without being limited to available published images
• Once constructed the image image cache provides improved performance, without having to use a image registry
• Working directory and user home directory are volume mounted, to allow caching of build dependencies, e.g. Maven, node_modules

Container Build Configuration

To execute the build within a container, add the containerBuild definition and containerImage to CDAF.solution. Note: complete definitions are provided in the GitHub samples for Windows and Linux.

The following samples have the default process commented out, and can be used to define a custom process.

Windows

containerImage=cdaf/windows
# containerBuild=& ${AUTOMATIONROOT}/processor/containerBuild.ps1 $SOLUTION $BUILDNUMBER $REVISION $ACTION

Linux

containerImage=cdaf/linux
# containerBuild=$AUTOMATIONROOT/processor/containerBuild.sh $SOLUTION $BUILDNUMBER $REVISION $ACTION

Build Time Variables

To supply variables to the build process, prefix with CDAF_CB_ (see CDAF Environment Variables) and the variables will be mapped into the build container.

See GitHub samples for Windows and Linux for dockerfile and additional properties.

imageBuild

imageBuild

This helper script supports the creation of docker images, and conditionally, the pushing of that image to a registry.

Container Build Configuration

To execute, define the buildImage definition. Note: complete definitions are provided in the GitHub samples for Windows and Linux.

The following samples have the default process commented out, and can be used to define a custom process.

Windows

buildImage=cdaf/windows
# imageBuild=& "$AUTOMATIONROOT/remote/imageBuild.ps1" ${SOLUTION}_${REVISION} ${BUILDNUMBER}

Linux

buildImage=cdaf/linux
# imageBuild="$AUTOMATIONROOT/remote/imageBuild.sh" ${SOLUTION}_${REVISION} ${BUILDNUMBER}

Immutable Deploy in Construction

If a custom docker file is not supplied, the default dockerfile will execute the IMMUTABLE release in the image construction process.

Registry Push

To include a push to a registry, add the following to CDAF.solution for DockerHub

CDAF_REGISTRY_URL=DOCKER-HUB
CDAF_REGISTRY_TAG=repo/${SOLUTION}:$BUILDNUMBER
CDAF_REGISTRY_USER=pat
CDAF_REGISTRY_TOKEN=${ACCESS_TOKEN}

Or for another registry provider or a self-hosted registry

CDAF_REGISTRY_URL=myregistry.io/repo
CDAF_REGISTRY_TAG=${CDAF_REGISTRY_URL}/${SOLUTION}:$BUILDNUMBER
CDAF_REGISTRY_USER=pat
CDAF_REGISTRY_TOKEN=${ACCESS_TOKEN}

Custom Image & Process

For samples of more complex usage see the GitHub samples for Windows and Linux dockerfile and additional properties.

Stand-alone Usage

The imageBuild script is used for docker-compose stand-up/tear-down process, for complete samples see Windows and Linux.

The following parameters must be passed

• Unique ID : normally "${SOLUTION}_${REVISION}"
• Sequential Number : normally "${BUILDNUMBER}"
• Base Image : any valid image, e.g. cdaf/linux or cdaf/windows

Optional parameters

• Context : if supplied, only the named directory is built, if ommited, all directories are built.

containerDeploy

containerDeploy

Like containerBuild, containerDeploy provides both image build and container task execution. The common use for container deploy where a command line interface is required.

Master of Deployment Success

The containerDeploy option allows the execution of the deploy process from within a container. Unlike toolsets which reference a image that is used to create the deploy container, CDAF uses a Dockerfile, for the following advantages:

• Deploy Prerequisites can be defined in code, without being limited to available published images
• Once constructed the image image cache provides improved performance, without having to use a image registry

Container Deploy Configuration

To execute the deploy within a container, add the containerDeploy definition and runtimeImage (if not supplied, containerImage will be used) to CDAF.solution. Note: complete definitions are provided in the GitHub samples for Windows and Linux.

The following samples have the default process commented out, and can be used to define a custom process.

Windows

runtimeImage=cdaf/windows
# containerDeploy=& ${WORK_DIR_DEFAULT}/containerDeploy.ps1 "${TARGET}" "${RELEASE}" "${SOLUTION}" "${BUILDNUMBER}" "${REVISION}" -imageDir cli

Linux

containerImage=cdaf/linux
# containerDeploy=${WORK_DIR_DEFAULT}/containerDeploy.sh "${TARGET}" "${RELEASE}" "${SOLUTION}" "${BUILDNUMBER}" "${REVISION}" cli

Deploy Time Variables

To supply variables to the build process, prefix with CDAF_CD_ (see CDAF Environment Variables) and the variables will be mapped into the build container.

See GitHub samples for Windows and Linux for dockerfile and additional properties.

Custom Image

The default directory used for container deploy is containerDeploy, if this is not found, the default Dockerfile is used, with the default runtime files. If you have your own Dockerfile in containerDeploy, or a custom directory specified in CDAF.solution containerDeploy property, then that will be used.

Runtime Files

The release.sh file is included in the default image, however, if using a default image, this needs to be explicitly defined in CDAF.solution runtimeFiles property. This can be a space separated list of files.

runtimeFiles=$WORKSPACE_ROOT/release.sh

Runtime Retain

To skip image clean-up, set CDAF.solution runtimeRetain property.

runtimeRetain=yes

dockerPush

dockerPush

Using the same logic after imageBuild, this utility script provides simple login and push logic.

The script can be called passing arguments

./dockerPush.ps1 $TARGET_TAG cdaf/${SOLUTION} "${artifactPrefix}.${BUILDNUMBER} latest" $DOCKERHUB_TOKEN cdaf

This example uses an environment variable (complete list follows) to set the URL. The registry in this example does not require authentication.

export CDAF_PUSH_REGISTRY_URL=hub.private.registry
./dockerPush.sh ${SOLUTION}_master_target:${BUILDNUMBER} ${SOLUTION} ${BUILDNUMBER}

Available environment variables