Pipelines: Pulling docker images from a Amazon ECR repository

[christoph_liekens NA]

For a build pipeline I would like to start from a docker image stored in a Amazon EC2 Container Registry (ECR) repository. Pulling this image would normally require me to do a 'aws ecr get-login' first, but I fail to see how this would work with a in a bitbucket-pipelines.yml file. How do I get this working,? Since the pull is the first step of the build pipeline, I don't think I can execute a script?

Thx

[davina · Accepted solution]

There is now an easier way to do this!

See Images hosted by the AWS ECR in the docs

[Will]

Late answer but might help those googler's

You can do the below and it will log you in using your AWS access & secret keys in one line.

eval $(aws ecr get-login --no-include-email --region eu-west-2)

[Steffen Opel _Utoolity_]

As already mentioned by Jeroen (+1), the required Amazon ECR credentials are valid for 12 hours only, see Registry Authentication for details:

[...] you must authenticate your Docker client another way so that Amazon ECR knows who is requesting to push or pull an image. If you are using the Docker CLI, then use the docker login command to authenticate to an Amazon ECR registry with an authorization token that is provided by Amazon ECR and is valid for 12 hours. The GetAuthorizationToken API operation provides a base64-encoded authorization token that contains a user name (AWS) and a password that you can decode and use in a docker login command. 

Workaround for Bamboo

We have been facing a similar limitation when using the built-in Bamboo Docker task to push images to an ECR repository for subsequent use in the Amazon ECS tasks of our Tasks for AWS (Bamboo) add-on. In order to work around it, we have implemented an Amazon ECR Credentials Variables task, which provides the following variables for use with the Docker task (see How to push a Docker image to a repository in your Amazon ECR registry with the Bamboo Docker task for a step-by-step guide):

${bamboo.custom.aws.ecr.proxyEndpointDomain}
${bamboo.custom.aws.ecr.username}
${bamboo.custom.aws.ecr.password}

Partial workaround for Bitbucket Pipelines

Conceptually you should be able to implement a similar approach with Bitbucket Pipelines based on section Images hosted outside Docker Hub within Use Docker images as build environments in Bitbucket Pipelines, i.e. you must provide the registry details, for example:

image:
  name: $ECRPROXYENDPOINTDOMAIN/namespace/java:8u66
  username: $USERNAME
  password: $PASSWORD
  email: $EMAIL

That is, you simply provide the required credentials as Environment variables in Bitbucket Pipelines (note that $EMAIL can be arbitrary as it is ignored by Amazon ECR).

The final piece for rendering this a halfway decent workaround would be an API to update those environment variables programmatically from a scheduled external task (e.g. via a AWS Lambda function triggered by a scheduled CloudWatch event).

Unfortunately, and much to my surprise, Bitbucket Pipelines doesn't seem to offer any API yet, despite one of its main value propositions rightfully being 'configuration as code'. Thorough 'configuration as code' would also need to cover all aspects of Bitbucket Pipelines itself though, most importantly those environment variables for service discovery and credentials management etc. - a resp. feature request seems to be in order ...

[Jeroen De Raedt]

Thanks for this extensive answer @Steffen Opel [Utoolity]! There is indeed a REST endpoint available to update a variable (so you can use it to update the ECR credentials): https://developer.atlassian.com/bitbucket/api/2/reference/resource/teams/%7Busername%7D/pipelines_config/variables/%7Bvariable_uuid%7D 

[Jeroen De Raedt]

It looks like Amazon ECR currently only allows you to generate Docker credentials that are valid for 12 hours: http://docs.aws.amazon.com/cli/latest/reference/ecr/get-authorization-token.html

Generating these for your build should work, however I suppose it is quite inconvenient to regenerate them all of the time. 

[Steffen Opel _Utoolity_]

I've just outlined a potential workaround, only to stumble over the apparent lack of an actual Bitbucket Pipelines API - presumably this is just not officially documented yet (the GUI needs to manage environment variables too after all)? In other words, is a documented and supported Bitbucket Pipelines API in the pipeline already?