Triggering integration tests after deployment

[Steven Wirges]

Hi,

i want to trigger a deployment plan for our staging environment after a successful build. This works with the trigger option "after successful build plan". But after the deployment, i want to trigger some integration/acceptance tests. What is the best practice, solution for that?

Copy my deploy tasks to a build stage is really ugly i think. There is a solution for deployments and the project environments, so why not using them?

[rpocklin]

This is a core piece of missing functionality.  Deployment plans should just be an extension of build plans and have all the same options and features.  Only workarounds are manual webhooks or create a deployment plan as a build plan (but that has other drawbacks).

Why the hell isn't this feature supported?

[Deleted user]

This is why we moved to Jenkins. Bamboo has just become too bloated and cumbersome over the years and doesnt have the flexibility to support continuous delivery. With Jenkins you can do arbitrary workflows, and theres a ton of 3rd party plugins available for doing whatever you need. We're also looking at GoCD which was designed from the ground up to do continuous delivery, but is a little beta-ish right now.

[ernestm]

We just started trying to use deployment projects in Bamboo (we had been doing deployments jammed into build plans) and this is a bit of a showstopper for us too. 

[Steffen Opel _Utoolity_]

Atlassian has just released the Bamboo After Deployment Trigger add-on, which allows to run your integration/acceptance tests as a regular build plan by adding a trigger to run plan when deployment project was deployed.

It is currently release as an 'Atlassian Labs' add-on and will hopefully be integrated into Bamboo down the road, thereby addressing the resp. improvement request:

• https://jira.atlassian.com/browse/BAM-14878
  

[seepatcode]

Our approach was to add a REST call to the second build project. This works BUT if you want to then deploy to a second environment after that build, it becomes murky.

Example, in the Final tasks of your initial deploy, create a script task such as :

wget --tries=10 --retry-connrefused --timeout=15 --waitretry=10 http://<yourdeployedapp>/healthcheck --delete-after
curl --user someuser:somepassword -X POST -d "JOB1&ExecuteAllStages" http://<your bamboo server>:8085/rest/api/latest/queue/<Your Bamboo AT Build Job>

"What is this doing?"

Line 1 is hitting some healthcheck end point in your app to confirm that the application has been fully stood up. You would have to implement some sort of endpoint if you don't already have one.

Line 2 is utilizing bamboo's REST api to kick of your Acceptance Test build.

---UPDATE---

We actually stopped taking this approach. NOW our first deployment is not a deployment at all, it's a build stage. By having it be part of the build, it totally cleaned up our build process and the dumb rest calls we were doing. Here's a small view into one of our apps called 'Filter' which is depended on by 'Timer' and 'Recorder'. We have actually two deployments in our build stages now. One for a first environment that we confirm we didn't break contracts with dependent apps AND one for a full acceptance test suite against all of our apps:

Screen Shot 2015-09-02 at 9.07.16 AM.png

[Tomas Vala]

I am currently hitting the exactly same problem. The situation is plain frustrating.

The first thing I want to do after successful deployment is to run a batch of tests in an automated fashion. You'd expect this to next logical step if not for all then for many users. Bad luck, Build Plans can't be triggered by Deployment Plans. Ugh, what was Atlassian thinking?

It's really terrible idea to glue tests or other types of post jobs to Deployment Plan. It's just poor concept. I had actually tried that (due to lack of other options) but ultimately trashed the idea. Deployment Plans should remain exclusive to guess what - deployment task.

Three main reasons:

1. Deployment Plans can't be segmented into Stages as ordinary Builds Plans can. Ouch. That means all steps (deployment and tests) have to be stuffed into one giant sequence. Now imagine some low priority test step failing for negligible reason - in result whole deployment is declared as FAILED regardless of actual failure severity (remember, no stages, no resolution). Fail notifications are sent despite the fact deployment alone finished successfully. This is not only misleading but downright false. Trust me, you do not want to face such messy situation.
2. Tests may be relatively time consuming or even hang (I am thinking of automated UI tests now) and those should not delay notifications of successful deployment. Never.
3. With every new Deployment Environment added, you'd need to duplicate testing steps, commands, scripts. That would be silly and inpractical to maintain.

What is proper and intended way of running post deployment jobs in Bamboo 5.4.2 then? I wish I could answer that. At this point I am seriously considering workaround like this:

1. By the end of successful Deployment Plan collect some essential information (like meta-data, variables, server url, application version, whatever is important for step 2 and commit that in form of property file(s) to purpose built VCS repository.
2. Create a new Build Job triggerable by VCS repository change from previous point. Fetch meta-data, configure and run tests.
3. Profit. Overly complicated and smelly but could work, well somewhat tolerably.

At the same time I keep wondering why some better solution is not provided out of box (or is it, just hiding from us?) as you'd expect from tool like Bamboo that is supposed to make tasks like this simple and straightforward.

Cheers,

Tomas

[Warren Cheetham]

Another approach is to commit a change to source control in a post-deployment task and then have your tests build monitor that repo.

For example:

Deployment Project

1. Package up build
2. Push to environment
3. Checkout git repo x
4. Update repo x
5. Commit to repo x

Build Project (Integration tests)

1. Triggers on changes to repo x

Note repo x could be a dedicated git branch with a single version file. (you need to avoid a circular dependancy)

Stages 4 & 5 could be this:

Stage 4 (script task - powershell)

$FilePath = "Artifacts/Web.config"
[ xml ] $xml = Get-Content -Path $FilePath
$node = $xml.SelectSingleNode('//appSettings/add[@key="version"]');
$node.SetAttribute("value", "${bamboo.deploy.version}");
$xml.Save($FilePath)

Stage 5 (script task)

git config --global user.email "system@email.com"
git config --global user.name "systemname"
git remote set-url origin "https://xxxx:${bamboo.bitbucket_password}@bitbucket.org/xxxx/repox.git"
git add Web.config
git commit -m "Updated version number"
git push origin release/trigger
exit /b 0

[Bhagyashree]

Cucumber for Bamboo plugin (available in Atlassian marketplace) allows you add Cucumber Parser tasks in Build as well as Deployment plans.

you can run your integration test with maven and add Cucumber task to parse the tests. It also gives options to integrate reaults with JIRA and provides multiple options to fail or pass the deployment depending on tests results.

https://marketplace.atlassian.com/plugins/com.mdb.plugins.cucumberforbamboo/server/overview

[Blaz Zabukovec]

We also have the same problem as described here. After a successful deployment we would like to trigger a plan that runs our integration tests. The problem was 'solved' by having and extra task in deployment project which does a dummy commit to repository. We have a file called deployment.log which is updated on every deployment and committed to repository. The commit then triggers the plan which runs integration tests.

[Erik Yeger Scudeler]

"Nice" solution. Will try to do the same. I was thinking and having the same problem as you guys. Let's try to find some issue on the atlassian backlog or create one and vote to have this feature ASAP.

[Pritam]

It is a nice solution..Will try to implement but it would be better if Atlassian gave us this features as an options of CD..

[Jeremy Roelfs]

I found this: How to Trigger Build After Successful Deploy

Which led me to this: Bamboo After Deployment Trigger that states:

"Trigger your smoke tests plan on deployment complete"

 I haven't tried it yet.

[Vikas Borse]

You can use Cucumber For Bamboo plugin. This plugin adds Cucumber parsers in Build as well ad Deployment plans. It provides options to pass or fail build if any test is failed or based on failure threshold or can just print test results in Log without affecting deployment status.

The plugin also adds new UI elements to display deployment tests, Cucumbers results and pie chart of tests.

https://marketplace.atlassian.com/plugins/com.mdb.plugins.cucumberforbamboo/server/overview

[Mike Cendana]

Here's another approach to this problem...

• Configure your deployment project to trigger a deployment upon completion of a build plan stage, lets call this stage Stage A.
• In the next stage of the same build plan, call it Stage B, the first task is a Script task that essentially does the following:
  • Uses the Bamboo REST API to query the comments service for the current build.
  • Look for a comment indicating that the deployment is complete, something like "Deployment to Environment X Complete."
  • If comment is found, finish task.
  • If comment is not found, wait for some amount of time, and try again.
  • If after some timeout period has expired and still have not found the comment, then fail task.
• In the deployment project, in the environment configuration, after all of the deployment related tasks are complete, add a task that does the following:
  • Uses the Bamboo REST API to add a comment "Deployment to Environment X Complete." to the current build.

And so this way you can keep the thread of execution in the same single build plan, and also leverage deployment projects to manage the deployments.  I have not actually tried this out yet, but it seems feasible.

[Mike Cendana]

One other note... the feature that lets you create a deployment trigger based on the completion of a build plan stage is only available in 5.10.

[ernestm]

I have a slightly more optimal REST workaround.  Also, am running in Bamboo Cloud so had to fiddle with getting the API working, most of the docs out there say to use weird ports or that "the API isn't available in Cloud," which must have been true at some point but no longer is. Rather than triggering a new plan, I have a CD plan with the following stages:

1. stage1 (triggered off repo change) - does the build
2. (deployment project 1 triggers on it being complete) - deploy to staging
3. stage 2 (manual) - runs integration tests
4. (deployment project 2 triggers on it being complete) - deploy to production
5. stage 3 (manual) - runs smoke tests

The very last step in each deployment project checks the service to make sure it's up, and if so, runs

curl -D- -u USERNAME:PASSWORD -p -H "Content-Type: application/json" -d '?stage+2' -f -X PUT "https://COMPANY.atlassian.net/builds/rest/api/latest/queue/${bamboo_buildResultKey}"

(or '?stage+3' in the second deployment project, of course - note the + to URL-encode the space)

The trick here is that the release knows what build it's using the results of, stored in bamboo_buildResultKey. This starts the same build back up at "stage+2" (same as just going in and manually clicking it) - this has the benefit of you don't need to pass a bunch of variables - it is part of the same build so it remembers all the variables, what head revision you were at, the release number, etc. (the second deployment correctly says it's deploying the same release to the production environment). See https://docs.atlassian.com/bamboo/REST/5.8.0/#d2e653 for the specific call documentation (yes, it's using a PUT, not a POST). 

That call fails if anything is even slightly weird (previous stage failed, a stage in that plan is running, etc.) which is probably for the best in this case, you'd rather it fail when it should run and just need a click to kick into gear than to kick off a stage when something may be hinky. 

Also be careful about string escaping, you are trying to insert variables but also it's likely you have special shell characters in your password, URL, stage names, etc.

[ernestm]

I think this is the feature improvement to vote on for this issue: https://jira.atlassian.com/browse/BAM-14878

[pscheit]

I solved the "issue" by using the deployment projects only for deployment.

Transfering files and running integration tests are part of the build plan and not the deployment. This works well for me. E.g. deployment to staging is in build plan but deployment to production is an deployment plan

[pscheit]

i used an inline script task to accomplish that:

release=${bamboo.deploy.release}
reftarget=${bamboo.www_target}/versions/$release
pwd=${reftarget//\//%2F}
code=$(curl --user ${bamboo.restuser}:${bamboo.restpassword} -X POST \
  -H "Accept: application/json" \
  --silent --output /dev/stderr --write-out "%{http_code}" \
  http://bamboo.ps-webforge.net/rest/api/latest/queue/TIPTOI-QA?bamboo.variable.release=$release\&bamboo.variable.pwd=$pwd
)  
if test $code -ne 200; then
    echo "curl request does not return 200"
    exit 1
fi

You have to store a user / password combination as variables (restuser/restpassword). It posts to the  REST api of bamboo itself and adds the release and pwd variables to the custom build.

This works for me, but there are still missing a lot of features for deploys.

(Please note my poor mans URL encoding in line 3 .. there are better ways to do this)

[Dewet Diener]

Thanks, that was a useful starting point. Instead of using --write-out to capture the return code, I am using -f to have curl fail explicitly on errors, removing the need to silence and capture the code. It is more verbose when things go wrong since the logs contain more info then.

[Deleted user]

So it seems that bamboo does not support continuous delivery, since this is a basic requirement for CD.

I dont understand why the deployment plan is a totally different beast from a build plan.  Why is this?  It seems like the deployment stuff was developed by a different company then poorly integrated into bamboo.  The configuration method and UI is completely different, and even more cumbersome than the build plans.

We too are looking into using other CD build tools, such as jenkins.

[Ted Sheibar]

We, too, are running into this problem, and it is forcing us to evaluate other CI tools, such as Go by Thoughtworks. Not being able to chain various builds/deployments together to create a pipeline is a significant oversight. It is our hope Atlassian has something planned to address this in the near future.

[zaid alzobaidi]

I have a similar issue, that is basically after the deployment complete, I want to run some external tests and depending on these result I will either decide the deployment is approved or not. Is there a way to acheive that ? By writing a "new deployment task" or not?

I aprreciate if someone has some input here.

[Gretchen]

The problem is that the deployment project depends on a build-plan successful completion not a deployment-plan. So you can't trigger one deployment based on the success of a different one. You also can't build based on a successful deployment and that kind of limits your options. You could create the integration tests as part of your deployment plan. That should give you the result you want which is to run integration tests based on successful deployment. It just wouldn't be a separate plan.

This will work because a successful plan will continue, and a failed plan will stop at the point where it fails. Thus a deployment that was unsuccessful will not continue to the next task (the integration tests).

It's probably not what you'd like to see (separation) but it will run the tests after deployment and if successful (assuming your tests don't fail) give you a green deployment. The down side is that the deployment will go red and fail (even if the "deployment" part was successful) if the integration tests fail.

Don't know what else you could do about that except handle the exception using a If fail print (tests failed) return 0 (which will cause your plan to ignore the failure and go green with the integration test failures). Basically it's a pick your poison choice.