Elastic Agent Stuck Pending - Starting Tunnel Server

[adamgregory]

Custom AMI based on Ubuntu-14.04.
Followed steps laid out here:
https://confluence.atlassian.com/display/BAMBOO/Creating+a+custom+elastic+image+-+Linux
To create custom AMI with Bamboo Prerequisites.

When the Elastic Instance is launched, the Elastic Agent remains in the pending state until the instance is eventually terminated and this process repeats.

The following is shown in bamboo-elastic-agent.out:

2015-05-27 14:27:47,841 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-e1/5.9-OD-02-054/78631190e13b992941890f2b0e4c6afd5fc5950d/ to /opt/bamboo-elastic-agent
2015-05-27 14:27:48,438 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-e1/5.9-OD-02-054/78631190e13b992941890f2b0e4c6afd5fc5950d/ to /opt/bamboo-elastic-agent
2015-05-27 14:27:48,439 INFO [main] [S3Utils] Fetching the list of remote objects...
2015-05-27 14:27:52,594 INFO [main] [S3Utils] Found 654 files in s3://bamboo-agent-release-us-e1/5.9-OD-02-054/78631190e13b992941890f2b0e4c6afd5fc5950d/
2015-05-27 14:27:52,710 INFO [main] [S3Utils] Found 691 files in /opt/bamboo-elastic-agent
2015-05-27 14:27:52,714 INFO [main] [S3Utils] Generating the list of files to fetch from S3...
2015-05-27 14:27:52,748 INFO [main] [S3Utils] Generating the list of files to remove...
2015-05-27 14:27:52,752 INFO [main] [S3Utils] Removing 0 files from /opt/bamboo-elastic-agent
2015-05-27 14:27:52,760 INFO [main] [S3Utils] Fetching 0 files to /opt/bamboo-elastic-agent
2015-05-27 14:27:52,760 INFO [main] [S3Utils] Fetched 0 MB from S3
2015-05-27 14:27:52,761 INFO [main] [ElasticAgentInstaller] Starting [java, -server, -Xms32m, -Xmx256m, -XX:MaxPermSize=128m, -XX:+HeapDumpOnOutOfMemoryError, -cp, /opt/bamboo-elastic-agent/boot/atlassian-aws-1.0.96.jar:/opt/bamboo-elastic-agent/boot/jcl-over-slf4j-1.7.7.jar:/opt/bamboo-elastic-agent/boot/jackson-annotations-2.3.0.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-shared-5.9-OD-02-054.jar:/opt/bamboo-elastic-agent/boot/fugue-2.1.0.jar:/opt/bamboo-elastic-agent/boot/bcprov-jdk15on-1.51.jar:/opt/bamboo-elastic-agent/boot/commons-codec-1.8.jar:/opt/bamboo-elastic-agent/boot/jackson-core-2.3.2.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-kms-1.9.16.jar:/opt/bamboo-elastic-agent/boot/httpclient-4.3.6.jar:/opt/bamboo-elastic-agent/boot/slf4j-log4j12-1.7.7.jar:/opt/bamboo-elastic-agent/boot/atlassian-annotations-0.9.jar:/opt/bamboo-elastic-agent/boot/joda-time-2.3.jar:/opt/bamboo-elastic-agent/boot/bcpkix-jdk15on-1.51.jar:/opt/bamboo-elastic-agent/boot/guava-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/slf4j-api-1.7.7.jar:/opt/bamboo-elastic-agent/boot/guava-bridge-11.0.2-atlassian-01.jar:/opt/bamboo-elastic-agent/boot/gson-2.2.2-atlassian-1.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-ec2-1.9.16.jar:/opt/bamboo-elastic-agent/boot/commons-lang-2.6.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-s3-1.9.16.jar:/opt/bamboo-elastic-agent/boot/httpcore-4.3.3.jar:/opt/bamboo-elastic-agent/boot/jsr305-1.3.9.jar:/opt/bamboo-elastic-agent/boot/aws-java-sdk-core-1.9.16.jar:/opt/bamboo-elastic-agent/boot/log4j-1.2.17.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-elastic-5.9-OD-02-054.jar:/opt/bamboo-elastic-agent/boot/atlassian-tunnel-1.7.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-agent-bootstrap-5.9-OD-02-054.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-api-agent-bootstrap-5.9-OD-02-054.jar:/opt/bamboo-elastic-agent/boot/atlassian-bamboo-core-agent-bootstrap-5.9-OD-02-054.jar:/opt/bamboo-elastic-agent/boot/annotations-13.0.jar:/opt/bamboo-elastic-agent/boot/jackson-databind-2.3.2.jar:, -Dbamboo.agent.installDir=/opt/bamboo-elastic-agent, com.atlassian.bamboo.agent.elastic.client.ElasticAgentBootstrap]
2015-05-27 14:27:56,774 INFO [main] [ElasticAgentBootstrap] Starting Agent Bootstrap using Java 1.7.0_79 from Oracle Corporation
2015-05-27 14:27:57,783 INFO [main] [ElasticAgentBootstrap] Using tunnelling. Setting virtual host name to https://rpsmarf.atlassian.net/builds/agentServer/
2015-05-27 14:27:57,784 INFO [main] [ElasticAgentBootstrap] Using tunnelling for HTTP(S). Registering 'httpt' and 'httpst' protocols.
2015-05-27 14:27:57,803 INFO [main] [ElasticAgentBootstrap] HTTP(S) tunnel: enabled
2015-05-27 14:27:57,803 INFO [main] [ElasticAgentBootstrap] JMS tunnel: enabled
2015-05-27 14:27:57,830 INFO [main] [ElasticAgentBootstrap] Starting tunnel server, waiting for 2 connections.

This issue also arose when completing step 5 shown here when preparing the custom AMI:
https://confluence.atlassian.com/display/BAMBOO/Creating+a+custom+elastic+image+-+Linux

Since the instance was initially launched from Bamboo, I combined step 5 and 6 by executing the command

su -c /opt/bamboo-elastic-agent/bin/bamboo-elastic-agent - bamboo

The command seems to get stuck at the same point - Starting tunnel server, waiting for 2 connections.

[Jaime S]

Hi, Adam.

I found a bug report and another question that may have some helpful information about that.

Kind regards,
Jaime Kirch da Silveira
Atlassian Cloud Support

[Alex Soto]

You will need to make the iptables rules 'permanent' by setting the config in /etc/iptables Don't forget you will need to bake another AMI. Personally I've stayed away from creating custom AMIs and just using the provided customization steps (https://confluence.atlassian.com/display/BAMBOO/Configuring+elastic+instances+to+use+the+EBS), but it does slow the boot up process of the agent. Good luck

[adamgregory]

I have tried to connect to the tunnel port (26224) using telnet and nmap and it appears that access to this port is being filtered by the firewall. I can manually open the firewall for TCP connections on port 26224 by by sshing to my running instance and executing: sudo iptables -I INPUT -p tcp --dport 26224 --syn -j ACCEPT After this, the elastic instance becomes idle and builds are attempted. Why is the tunnel port not open when the instance is started? The instance is part of the security group elasticbamboo in my aws console and port 26224 is open to all incoming tcp connections.