Elastic Images Terminate

Galen Pyle October 12, 2012

Hi, I'm trying to create custom build server instances from Bamboo elastic images. I'm installing .NET 4.5 and a Visual Studio Shell on the drive. But the elastic images are terminating suddenly. I'll be logged on and moving forward with building the build server, and it just shuts down and terminates. Please see the below log from Bamboo. Why are the Bamboo elastic images terminating unexpectedly??

Oct 12, 2012 1:53:10 PM Requested that new elastic instance be created for configuration: Windows .NET 4.5 Build Server
Oct 12, 2012 1:53:10 PM Elastic instance [i-80f9dbfd] transitioned from STARTING to IDENTIFIED.
Oct 12, 2012 1:53:52 PM Elastic instance [i-80f9dbfd] transitioned from IDENTIFIED to RUNNING.
Oct 12, 2012 2:02:35 PM An elastic agent is loading on instance: [i-80f9dbfd]
Oct 12, 2012 2:04:01 PM Elastic Agent "Elastic Agent on i-80f9dbfd" started on instance i-80f9dbfd
Oct 12, 2012 3:04:12 PM An elastic agent is loading on instance: [i-80f9dbfd]
Oct 12, 2012 3:09:17 PM Elastic Agent "Elastic Agent on i-80f9dbfd" stopped on instance i-80f9dbfd
Oct 12, 2012 3:09:51 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" started on instance i-80f9dbfd
Oct 12, 2012 3:14:36 PM Elastic instance [i-80f9dbfd] transitioned from RUNNING to SHUTTING_DOWN.
Oct 12, 2012 3:14:57 PM Elastic instance [i-80f9dbfd] transitioned from SHUTTING_DOWN to TERMINATED.
Oct 12, 2012 3:14:57 PM Detected that the elastic instance [i-80f9dbfd] has been terminated.
Oct 12, 2012 3:14:57 PM Elastic Agent "Elastic Agent on i-80f9dbfd (2)" stopped on instance i-80f9dbfd

3 answers

1 accepted

2 votes
Answer accepted
Galen Pyle October 18, 2012

I've spoken with Atlassian technical support and the problem is caused by the automatic elastic instance management configuration setting. It is the best practice to disable this feature. There is a Bamboo agent that runs on each Bamboo build server instance. If this agent stops and the automatic elastic instance management configuration setting is enabled, then the server auto-terminates. The agent may stop if you restart the Bamboo build server instance.

Jeff Hansen January 19, 2015

I have disabled everything that could possibly auto-terminate my elastic images, but they are still terminating on their own (like once a month - fairly infrequently but still annoying). Any other ideas to ensure that the image doesn't shut down? Is Atlasssian shutting down their EC2 cloud group or something like that periodically?

0 votes
derek williams August 17, 2015

Hi,

I didn’t see a specific answer here, it’s an old thread so, I suspect it was resolved.  We recently encountered the same symptoms, just leaving my experience for future visitors.

Bamboo would launch elastic instance, the instance would terminate about 3-4 minutes later, the agent never moved out of pending.

Cause:

  • Our AMI was setup to create 2 volumes when new instances are launched.  

  • When the instance was terminated, the second volume was being left behind.  I suspect this has something to do with our AMI profile permissions.  Eventually, AWS wouldn’t provision anymore volumes because, we had too many created.  (got up to around 100 orphaned volumes before we noticed)

Result:

  • Bamboo would attempt to create new elastic instances

  • AWS would start the instance, then terminate it because the volume couldn’t be created

  • Inside Bamboo it appeared the agent was just randomly going offline


For us, we just adjust the AMI to only create 1 volume and used the asw console to cleanup the volumes.

 

cheers

 

140401 kindia May 1, 2019

Hi ,

 

I am also observing similar error. Bamboo elastic agents get terminated while building plans.

any workaround and solution?

0 votes
Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 12, 2012

I suspect that the agent is having problems starting up, shuts down, gets restarted (that why you see the agent marked with (2) ). Eventually Bamboo decides the agent won't come up and kills the instance.

Are you stopping/starting/restarting the instance in the meantime?

If not, what does the agent log say (you can find it in the agent home directory)?

Galen Pyle October 14, 2012

Thanks for your response! Please see the attached Elastic Agent Usage History. Each of the Windows Bamboo build server instances have auto-terminated. I created the three most recent elastic agents as a test. I never remoted into these three instances, but they each terminated at 7:48 PM on Oct 14. I do not stop/start the instances after their initial creation. I did restart some of these build servers after I installed .NET 4.5. We use AWS on a daily basis and we only have this problem with the Bamboo elastic agents. This is a serious impediment to our progress. Thanks so much for your time & consideration.

Galen Pyle October 14, 2012

Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Galen Pyle October 14, 2012

Elastic Agent Usage History

Listed below are all elastic agents that have run in the past. This does not include any currently running elastic agents, you can view these on the Elastic Instances page

Agent

Configuration

Start Time

Shutdown Time

Uptime

Successful Jobs

Operations

Elastic Agent on i-24f4d959

Windows .NET 4.5 Build Server

12 Oct 2012, 4:09:16 PM

14 Oct 2012, 7:48:07 PM

2 days, 3 hours, 38 minutes

No builds

View | Delete

Elastic Agent on i-26f4d95b

Windows .NET 4.5 Build Server

12 Oct 2012, 4:06:53 PM

14 Oct 2012, 7:48:07 PM

2 days, 3 hours, 41 minutes

No builds

View | Delete

Elastic Agent on i-42f4d93f

Windows .NET 4.5 Build Server

12 Oct 2012, 4:06:39 PM

14 Oct 2012, 7:48:07 PM

2 days, 3 hours, 41 minutes

No builds

View | Delete

Elastic Agent on i-80f9dbfd (2)

Windows .NET 4.5 Build Server

12 Oct 2012, 3:09:51 PM

12 Oct 2012, 3:14:57 PM

5 minutes

0 / 1

View | Delete

Elastic Agent on i-80f9dbfd

Windows .NET 4.5 Build Server

12 Oct 2012, 2:04:01 PM

12 Oct 2012, 3:09:17 PM

1 hour, 5 minutes

No builds

View | Delete

Elastic Agent on i-84496bf9 (2)

Windows .NET 4.5 Build Server

12 Oct 2012, 1:31:58 PM

12 Oct 2012, 1:45:48 PM

13 minutes

0 / 1

View | Delete

Elastic Agent on i-84496bf9

Windows .NET 4.5 Build Server

12 Oct 2012, 1:12:49 PM

12 Oct 2012, 1:40:17 PM

27 minutes

0 / 1

View | Delete

Elastic Agent on i-aa652fd7

Default Image EBS x86_64 (windows)

12 Oct 2012, 12:38:13 PM

12 Oct 2012, 12:51:46 PM

13 minutes

No builds

View | Delete

Windows .NET 4.5 Build Server

Default Image EBS x86_64 (windows)

10 Oct 2012, 11:17:28 AM

12 Oct 2012, 12:46:17 PM

2 days, 1 hour, 28 minutes

8 / 28

View | Delete

Galen Pyle October 14, 2012

Elastic Agent History › Elastic Agent on i-26f4d95b (elastic)

Elastic agent on instance i-26f4d95b, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-26f4d95b (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 4:06:53 PM

Last shutdown timeOct 14, 2012 7:48:07 PM

Up time2 days, 3 hours, 41 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Elastic Agent History › Elastic Agent on i-80f9dbfd (elastic)

Elastic agent on instance i-80f9dbfd, configuration: Windows .NET 4.5 Build Server

You can view historical information for this elastic agent in this page. This agent is offline and will never come back alive.

Agent Details

0%0 / 0 Successful

Elastic instance Instance i-80f9dbfd (Windows .NET 4.5 Build Server image configuration)

The elastic instance this agent is/was running on.

Last startup timeOct 12, 2012 2:04:01 PM

Last shutdown timeOct 12, 2012 3:09:17 PM

Up time1 hour, 5 minutes

Build History

The agent was building for 0% of the up time.

There are no builds to display.

Przemek Bruski
Atlassian Team
Atlassian Team members are employees working across the company in a wide variety of roles.
October 14, 2012

Do you have Elastic Instance Management enabled in your Elastic Bamboo configuration?

Galen Pyle October 15, 2012

Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!

Galen Pyle October 15, 2012

Elastic Bamboo is configured with our Amazon Web Services Account. Automatic Elastic Instance Management is currently DISABLED in my Elastic Bamboo configuration. Could this be the cause of the problem? What is the optimal elastic management configuration?

Thanks for your reply!

-----------------------------------------------------------------------------------------

Spot Instances Configuration

Support for spot instances is disabled.

Elastic Bamboo AWS Settings

AWS account identifier files are not automatically uploaded to new elastic instances.

Automatic Elastic Instance Management

EC2 instances must be manually created and terminated as needed.

Suggest an answer

Log in or Sign up to answer
TAGS
AUG Leaders

Atlassian Community Events