Running integration tests across multiple agents

[chphilli]

We have a Bamboo plan setup for running our basic build & unit tests. Now we would like to extend that to also run our integration tests. Our problem is that our integration environment relies on services running on both Windows and Linux machines.

I don't think we can just setup a deployment environment – one of the reasons we're evaluating Bamboo is to take advantage of it's elastic scaling which doesn't seem to work with deployment environments ( as far as I can tell ).

How can we setup a plan that uses two elastic Bamboo agents ( one from a Windows image, one from a Linux image ), applies necessary build artifacts to each, and then runs tests against them?

[Steffen Opel _Utoolity_ · Accepted solution]

Update

Given that the integration tests depend on both a Linux and a Windows image communicating with each other, you might need to increase the level of elasticity and isolation for the test environments:

While elastic agents can be used to simulate an isolated integration test environment (if one ensures to use an agent only once), it isn't really geared towards the goal of Dev/prod parity, because you will always encounter agent instance specific environment conditions and constraints.

Infrastructure as Code

Instead, I would facilitate the elasticity of the cloud and deploy the required Infrastructure as Code, i.e. only use the agent to create a new integration environment dedicated to the test(s) at hand. That is, your build agent would start a single integration environment comprised of a Linux and a Windows system, deploy your code on each, and run the integration tests against that temporary environment (this might require refactoring the tests so that they can be run and/or instrumented remotely).

There are several platforms and tools available to achieve this and I'll name two:

• AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion. CloudFormation is AWS' own solution, which naturally comes with excellent coverage of the entire AWS portfolio, thereby including the option to create both a Linux and a Windows instance within a single CloudFormation stack.
• HashiCorp's Terraform likewise provides a common configuration to launch infrastructure — from physical and virtual servers to email and DNS providers. It is a new provider agnostic tool conceptually similar to CloudFormation, thereby a good choice if you do not want to be constrained to AWS (which is also supported by Terraform of course).

Both tools can be instrumented from the command line (AWS CLI / Terraform CLI), thus used from Bamboo Cloud as well.

• Regardless, I would be remiss if I wouldn't mention Utoolity's (commercial) Tasks for AWS add-on, which (amongst other features) allows to create/update/delete/validate an AWS CloudFormation stack, thereby enabling the very scenario you seem to require - similar to Atlassian's Requirement for Deployments add-on mentioned in my initial answer, it is unfortunately only available for Bamboo Server due to Bamboo Cloud not yet being supported by Atlassian's new add-on framework Connect.

---

Initial Answer

This can be achieved with capabilities and requirements in principle (see below for some major caveats regarding your scenario though), as outlined in How do capabilities work with requirements?:

A requirement is specified in a job or a task. A requirement specifies a capability that an agent must have for it to build that job or task. A job inherits all of the requirements specified in its tasks.

Accordingly, you would need to separate your integration tests by operating system, so that each OS is isolated in dedicated jobs, which will allow Bamboo to schedule those jobs on an agent that matches the job's resp. OS requirements, scaling up the required agents elastically as usual (e.g. one agent type for Linux and another one for Windows jobs).

Unfortunately, as you already hinted upon, the ability to add requirements to deployment environments is not officially supported yet, and thus not available in Bamboo Cloud:

• https://jira.atlassian.com/browse/BAM-13499

However, Atlassian meanwhile provides this feature for Bamboo Server in the Requirement for Deployments Labs add-on at least, and incidentally the first highlight matches your use case:

Atlassian's latest update suggests that they are well aware of the need to officially support and ship it with Bamboo (and thereby Bamboo Cloud), but apparently the team [won't] be able to look into full support before the end of 2015.

So if this is a crucial requirement, your only option at this point might be to migrate to Bamboo Server ...

• Either way, you might want to watch/vote/comment on BAM-13499 to further increase awareness for this obvious need (which is amongst the top ten most voted features already though).

[chphilli]

Thanks! Unfortunately, I can't just split up my jobs with capabilities -- the integration tests are against services on the Windows hosts that rely on services running on the Linux hosts. e.g. our Windows-hosted log processor relies on some Linux only services to function properly.

[Steffen Opel _Utoolity_]

@Chris Phillips - thanks for the clarification. I have updated my answer with a different approach how to achieve this - while it is naturally subjective, it comprises an increasingly common pattern enabled by the very elasticity of the cloud that seemingly sparked your interest in Bamboo as well.

[chphilli]

@Steffen Opel [Utoolity] Thanks for the detailed answer. This is basically how we've been thinking about approaching the problem as well. I'm sure I'll be referencing your answer as we continue to work on this problem.