bamboo elastic agent cannot download agent from s3

[Dennis Roberts]

When a new elastic instance comes up it fails to start the elastic agent.  I see the following error in the bamboo-elastic-agent.out log:

Starting Elastic Bamboo Agent...
2015-04-30 18:45:59,557 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w2/5.8.1/6f8427be5a3fc34c1b7e639051e74693e39b80c8/ to /opt/bamboo-elastic-agent
2015-04-30 18:46:00,145 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w2/5.8.1/6f8427be5a3fc34c1b7e639051e74693e39b80c8/ to /opt/bamboo-elastic-agent
2015-04-30 18:46:00,146 INFO [main] [AmazonClients] Detecting bucket location for [bamboo-agent-release-us-w2]
2015-04-30 18:46:00,567 WARN [main] [AmazonClients] Unable to get bucket location for [bamboo-agent-release-us-w2], using default. Error: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 9478876E58479335), S3 Extended Request ID: 5sK+xcl/v+OolC+b6MYst94yy4iGfSUXE9KJe7VR1A63Po6D4mIYMr/oI1wsTheabtS0H5fDHt0=
2015-04-30 18:46:00,567 INFO [main] [S3Utils] Fetching the list of remote objects...
Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 475379D3A66EB952), S3 Extended Request ID: OOzpqvF/41Ob5Pt8NAisYKOMGIfetQdZ8vik8gXcOdrAuRjIptGc27dnzLRhFdrQWBWb1UYG58o=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3714)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:637)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)
at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:354)
at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:162)
at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.install(ElasticAgentInstaller.java:68)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:155)

We have a license for 5 remote agents. We manually terminated 4 of them this morning because of another error.  Is it possible the server thinks those are still running because they were terminated outside of bamboo?

Please advise!  We are totally blocked.

[Sachin Verma]

I am also facing the almost same issue. Even after updating the s3 accessKey with new one its still showing the same error.

 

2016-09-28 07:37:54,259 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w1/5.12.4-D20160704T134120/*********/ to /opt/bamboo-elastic-agent

2016-09-28 07:37:57,164 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w1/5.12.4-D20160704T134120/****** to /opt/bamboo-elastic-agent

2016-09-28 07:37:57,165 INFO [main] [S3Utils] Fetching the list of remote objects...

Exception in thread "main" com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: **********), S3 Extended Request ID: *********/************** at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1077)

 at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:725)

 at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)

 at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3714)

 at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3664)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:637)

 at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:620)

 at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:353)

 at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:161)

 at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)

 at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

[Albert Wu]

EDIT: This appears to occur after we enabled VPC S3 Endpoint:

https://aws.amazon.com/blogs/aws/new-vpc-endpoint-for-amazon-s3/

We have reverted back and looking into what caused the issue.

Hello, this appears to be happening again:

2015-05-12 11:14:24,548 INFO [main] [S3Sync] Syncing from: bamboo-agent-release-us-w2/5.7.2/d7f1fd87dd2cb29fd61c86f85686c4ee09bc87a5/ to /opt/bamboo-elastic-agent
2015-05-12 11:14:25,068 INFO [main] [S3Utils] Syncing s3://bamboo-agent-release-us-w2/5.7.2/d7f1fd87dd2cb29fd61c86f85686c4ee09bc87a5/ to /opt/bamboo-elastic-agent
2015-05-12 11:14:25,068 INFO [main] [S3Utils] Fetching the list of remote objects...
Exception in thread "main" AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: 4238B864CDFBA824, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: N4FpdmmUYgqPfsYHNaieaUtRDWYIAEaoIOe/eFg04D6J02FJ7b4M1Pg0nrb3giMNovuh7e9nxmE=
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:644)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:190)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2974)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2945)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:478)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:462)
at com.atlassian.aws.s3.S3Utils.getObjectNamesAndHashes(S3Utils.java:331)
at com.atlassian.aws.s3.S3Utils.sync(S3Utils.java:159)
at com.atlassian.bamboo.agent.elastic.S3Sync.sync(S3Sync.java:72)
at com.atlassian.bamboo.agent.elastic.installer.ElasticAgentInstaller.main(ElasticAgentInstaller.java:66)

[Swatantra Yadav]

I also had the same problem. It worked after I replaced my AWS key with brand new key.

[Dennis Roberts]

That shouldn't have mattered. The creds that bamboo is using to get that agent are their own not yours.

[Swatantra Yadav]

True that what it looks like from the logs because it is trying to access Bamboo's own s3 bucket. Not sure why but changing AWS key has fixed the issue for me for now.

[Dennis Roberts]

It is working now! Thank you.  Can you share what the problem was and what I should do if it happens again?

[Przemek Bruski]

Can you try now? It should be fixed.