For the Drupalize.Me site we have a functional/integration test suite that's built on the Nightwatch.js test runner. We use this to automate testing of critical paths like "Can a user purchase a membership?" as well as various edge-case features that we have a tendency to forget exist -- like "What happens when the owner of a group account adds a user to their membership and that user already has a Drupalize.Me account with an active subscription?"
For the last few months, running these tests has been a manual process that either Blake or I would do our on our localhost before doing a release. We used to run these automatically using a combination of Jenkins and Pantheon's MultiDev, but when we switched to using Tugboat instead of MultiDev to build previews for pull-requests, that integration fell to the wayside and eventually we just turned it off because it was failing more often than it was working.
Aside: The Drupalize.Me site has existed since 2010, and has gone through numerous rounds of accumulating and then paying off technical debt. We once used SVN for version control. Our test suite has gone from non-existent, to Behat, then Casper, then back to Behat, and then to Nightwatch. Our continuous integration (CI) relies primarily on duct tape and bubble gum. It's both the curse, and the joy, of working on a code base for such a long time.
I recently decided it was time to get these tests running automatically again. Could I do so using GitHub actions? I have a bunch of experience with other CI tools, but this was my first time really diving into either of these in their current form. Here's what I ended up with.
- We use Tugboat.qa to build preview environments for every pull-request. These are a clone of the site with changes from the pull-request applied. This gives us a URL that we can use to run our tests against.
- We use GitHub Actions to spin up a robot that'll execute our tests suite against the URL provided by Tugboat and report back to the pull request.
Setting up Tugboat to build a preview for every pull request
We use a fairly cookie-cutter Tugboat configuration for building preview environments that Blake set up and I mostly just looked at and thought to myself, "Hey, this actually looks pretty straightforward!" The setup:
- Has an Apache/PHP service with Terminus and Drush installed, and a MySQL service
- Pulls a copy of the database from Pantheon as needed
- Reverts features, updates the database, and clears the cache each time a pull request is updated
- Most importantly, it has a web-accessible URL for each pull request
Here's what our .tugboat/config.yml looks like with a few unrelated things removed to keep it shorter:
services:
php:
# Use PHP 7.2 with Apache
image: tugboatqa/php:7.2-apache
default: true
# Wait until the mysql service is done building
depends: mysql
commands:
# Commands that set up the basic preview infrastructure
init:
# Install prerequisite packages
- apt-get update
- apt-get install -y default-mysql-client
# Install opcache and enable mod-rewrite
- docker-php-ext-install opcache
- a2enmod headers rewrite
# Install drush 8.*
- composer --no-ansi global require drush/drush:8.*
- ln -sf ~/.composer/vendor/bin/drush /usr/local/bin/drush
# Install the latest version of terminus
- wget -O /tmp/installer.phar https://raw.githubusercontent.com/pantheon-systems/terminus-installer/master/builds/installer.phar
- php /tmp/installer.phar install
# Link the document root to the expected path.
- ln -snf "${TUGBOAT_ROOT}/web" "${DOCROOT}"
# Authenticate to terminus. Note this command uses a Tugboat environment
# variable named PANTHEON_MACHINE_TOKEN
- terminus auth:login --machine-token=${PANTHEON_MACHINE_TOKEN}
# Commands that import files, databases, or other assets. When an
# existing preview is refreshed, the build workflow starts here,
# skipping the init step, because the results of that step will
# already be present.
update:
# Use the tugboat-specific Drupal settings
- cp "${TUGBOAT_ROOT}/.tugboat/settings.local.php" "${DOCROOT}/sites/default/"
- cp "${TUGBOAT_ROOT}/docroot/sites/default/default.settings_overrides.inc" "${DOCROOT}/sites/default/settings_overrides.inc"
# Generate a unique hash_salt to secure the site
- echo "\$settings['hash_salt'] = '$(openssl rand -hex 32)';" >> "${DOCROOT}/sites/default/settings.local.php"
# Import and sanitize a database backup from Pantheon
- terminus backup:get ${PANTHEON_SOURCE_SITE}.${PANTHEON_SOURCE_ENVIRONMENT} --to=/tmp/database.sql.gz --element=db
- drush -r "${DOCROOT}" sql-drop -y
- zcat /tmp/database.sql.gz | drush -r "${DOCROOT}" sql-cli
- rm /tmp/database.sql.gz
# Configure stage_file_proxy module.
- drush -r "${DOCROOT}" updb -y
- drush -r "${DOCROOT}" fra --force -y
- drush -r "${DOCROOT}" cc all
- drush -r "${DOCROOT}" pm-download stage_file_proxy
- drush -r "${DOCROOT}" pm-enable --yes stage_file_proxy
- drush -r "${DOCROOT}" variable-set stage_file_proxy_origin "https://drupalize.me"
# Commands that build the site. This is where you would add things
# like feature reverts or any other drush commands required to
# set up or configure the site. When a preview is built from a
# base preview, the build workflow starts here, skipping the init
# and update steps, because the results of those are inherited
# from the base preview.
build:
- drush -r "${DOCROOT}" cc all
- drush -r "${DOCROOT}" updb -y
- drush -r "${DOCROOT}" fra --force -y
- drush -r "${DOCROOT}" scr private/scripts/quicksilver/recurly_dummy_accounts.php
# Clean up temp files used during the build
- rm -rf /tmp/* /var/tmp/*
# What to call the service hosting MySQL. This name also acts as the
# hostname to access the service by from the php service.
mysql:
image: tugboatqa/mysql:5
In order to get Tugboat to ping GitHub whenever a preview becomes ready for use, make sure you enable the Set Pull Request Deployment Status feature in Tugboat's Repository Settings.
Run tests with GitHub Actions
Over in GitHub Actions, we want to run our tests and add a status message to the relevant commit. To do this we need to know when the Tugboat preview is done building and ready to start testing, and then spin up a Node.js image, install all our Nightwatch.js dependencies, and then run our test suite.
We use the following .github/workflows/nightwatch.yml configuration to do that:
name: Nightwatch tests
on: deployment_status
jobs:
run-tests:
# Only run after a successful Tugboat deployment.
if: github.event.deployment_status.state == 'success'
name: Run Nightwatch tests against Tugboat
runs-on: ubuntu-latest
steps:
# Set an initial commit status message to indicate that the tests are
# running.
- name: set pending status
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
debug: true
script: |
return github.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha: context.sha,
state: 'pending',
context: 'Nightwatch.js tests',
description: 'Running tests',
target_url: "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
});
- uses: actions/checkout@v1
- uses: actions/setup-node@v1
with:
node-version: '12'
# This is required because the environment_url param that Tugboat uses
# to tell us where the preview is located isn't supported unless you
# specify the custom Accept header when getting the deployment_status,
# and GitHub actions doesn't do that by default. So instead we have to
# load the status object manually and get the data we need.
# https://developer.github.com/changes/2016-04-06-deployment-and-deployment-status-enhancements/
- name: get deployment status
id: get-status-env
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
result-encoding: string
script: |
const result = await github.repos.getDeploymentStatus({
owner: context.repo.owner,
repo: context.repo.repo,
deployment_id: context.payload.deployment.id,
status_id: context.payload.deployment_status.id,
headers: {
'Accept': 'application/vnd.github.ant-man-preview+json'
},
});
console.log(result);
return result.data.environment_url;
- name: echo tugboat preview url
run: |
echo ${{ steps.get-status-env.outputs.result }}
# The first time you hit a Tugboat URL it can take a while to load, so
# we visit it once here to prime it. Otherwise the very first test
# will often timeout.
curl ${{ steps.get-status-env.outputs.result }}
- name: run npm install
working-directory: tests/nightwatch
run: npm ci
- name: run nightwatch tests
# Even if the tests fail, we want the job to keep running so we can set the
# commit status and save any artifacts.
continue-on-error: true
working-directory: tests/nightwatch
env:
TUGBOAT_DEPLOY_ENVIRONMENT_URL: ${{ steps.get-status-env.outputs.result }}
run: npm run test
# Update the commit status with a fail or success.
- name: tests pass - set status
if: ${{ success() }}
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
return github.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha: context.sha,
state: "success",
context: 'Nightwatch.js tests',
target_url: "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
});
- name: job failed - set status
if: ${{ failure() }} || ${{ cancelled() }}
uses: actions/github-script@v3
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
return github.repos.createCommitStatus({
owner: context.repo.owner,
repo: context.repo.repo,
sha: context.sha,
state: "error",
context: 'Nightwatch.js tests',
target_url: "https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}"
});
# If the tests fail we take a screenshot of the failed step, and then
# those get uploaded as artifacts with the result of this workflow.
- name: archive testing artifacts
uses: actions/upload-artifact@v2
with:
name: screenshots
path: tests/nightwatch/screenshots
if-no-files-found: ignore
The one maybe abnormal thing I had to do to get this working is use the actions/github-script@v3 action to manually query the GitHub API for information about the Tugboat deployment. There's a good chance that there is a better way to do this -- so if you know what it is, please let me know.
The reason is that Tugboat sets the public URL of a preview in the deployment.environment_url property. But, this property is currently hidden behind a feature flag in the API. It isn't present in the deployment object that your GitHub workflow receives. So in order to get the URL that we want to run tests against, I make a query to the GitHub API with the Accept: application/vnd.github.ant-man-preview+json header. There are other actions you can use to update the status of a commit that are a little cleaner syntax, but this workflow is already using actions/github-script@v3 so for consistency I used that to set a commit status as well.
This Debugging with tmate action was super helpful when debugging the GitHub Workflow. It allows you to open a terminal connection to the instance where your workflow is executing and poke around.
Our nightwatch.config.js looks like the following. Note the use of the Tugboat URL we retrieved and set as an environment variable in the workflow above, process.env.TUGBOAT_DEPLOY_ENVIRONMENT_URL. Also note the configuration that enables taking a screenshot whenever a test fails.
module.exports = {
"src_folders": [
"tests"// Where you are storing your Nightwatch tests
],
"output_folder": "./reports", // reports (test outcome) output by nightwatch
"custom_commands_path": "./custom-commands",
"webdriver": {
"server_path": "node_modules/.bin/chromedriver",
"cli_args": [
"--verbose"
],
"port": 9515,
"timeout_options": {
"timeout": 60000,
"retry_attempts": 3
},
"start_process": true
},
"test_settings": {
"default": {
'launch_url': 'http://dme.ddev.site',
"default_path_prefix": "",
"persist_globals": true,
"desiredCapabilities" : {
"browserName" : "chrome",
"javascriptEnabled": true,
"acceptSslCerts" : true,
"chromeOptions" : {
// Remove --headless if you want to watch the browser execute these
// tests in real time.
"args" : ["--no-sandbox", "--headless"]
}
},
"screenshots": {
"enabled": false, // if you want to keep screenshots
"path": './screenshots' // save screenshots here
},
"globals": {
"waitForConditionTimeout": 20000 // sometimes internet is slow so wait.
}
},
// Run tests using GitHub actions against Tugboat.
"test" : {
"launch_url" : process.env.TUGBOAT_DEPLOY_ENVIRONMENT_URL,
// Take screenshots when something fails.
"screenshots": {
"enabled": true,
"path": './screenshots',
"on_failure": true,
"on_error": true
}
}
}
};
Finally, to tie it all together, the GitHub workflow runs npm run test which maps to this command:
./node_modules/.bin/nightwatch --config nightwatch.config.js --env test --skiptags solr
That launches the test runner and starts executing the test suite. Ta-da!
Is this even the right way?
While working on this I've found myself struggling to figure out the best approach to all this. And while this works, I'm still not convinced it's the best way.
Here's the problem: I can't run the tests until Tugboat has finished building the preview -- so I need to somehow know when that's done.
For this approach I get around this by enabling deployment_status notifications in Tugboat, listening for them in my GitHub workflow using on: deployment_status, and then executing the test suite when I get a "success" notification. One downside of this approach is that in the GitHub UI the "Checks" tab for the PR will always be blank. In order for a workflow to log its results to the Checks tab, it needs to be triggered via a push or pull_request event. I can still set a commit status, which in turn will allow for a green check or red x on the pull request, but navigating to view the results is less awesome.
This approach allows for a pretty vanilla Tugboat setup.
It seems like an alternative would be to disable Tugboat's option to automatically build a preview for a PR. Instead, we'd use a GitHub workflow with an on: [push, pull_request] configuration that uses the Tugboat CLI to ask Tugboat to build a preview, wait for the URL, and then run the tests. This would allow for better integration with the GitHub UI, but require more scripting to take care of a lot of things that Tugboat already handles. I would need to not only build the preview via the CLI, but also update it and delete it at the appropriate times.
I do think that much of the Tugboat scripting here would be pretty generic, and I could probably write the workflow to manage Tugboat previews via GitHub Actions once and mostly copy/paste it in the future.
Yet another approach would be to not use GitHub Actions at all, and instead run the tests via Tugboat. Then use the GitHub Checks API to report back to GitHub about the status of a PR and log the results into the "Checks" tab. However, this looks like a lot of code and would probably be better if it could be included into Tugboat in a more generic way. Something like a "run my tests" command, and a way to parse the standard jUnit output, and log the results to GitHub, or maybe just bypass the Checks UI all together and instead have Tugboat provide a UI for viewing test results.
I might explore these other options further in the future. But for now... it's working, so don't touch it! Like I said earlier -- it's all duct tape and bubble gum.
Recap
It took a while to figure this all out, and to debug the various issues on remote machines, but in the end, I'm happy with where things ended up. More than anything, I love having robots run the tests for me once again.