Community
Products
Bitbucket
Questions
Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?

Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?

I have a serverless app that uses Poetry for package management.

I am using bitbucketpipelines/serverless-deploy:1.5.0 to deploy my serverless app. When I try to deploy, I get this error:

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)

Why is serverless-python-requirements part of the Docker Image, which supports Poetry projects, but poetry is not installed in the image?

I've tried installing poetry in a pre execution script, and I still get the same error. Can the serverless-deploy pipe's Dockerfile include the installation of Poetry?

Thanks

1 answer

0 votes

@Sam Mahr

Thank you for your question!

The pipe's base docker image contains only base dependencies to keep the image thin.

There are many variety of dependencies all around the users projects.

If you need extra dependencies according to your project, pipe supports preexecution script.

Additionally, check if the version of requirements and runtime corresponds to platform dependency version in Dockerfile.

Also, could you provide your bitbucket-pipelines configuration, serverless configuration and preexecution script.

Best regards,
Oleksandr Kyrdan

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Thanks, @Oleksandr Kyrdan. Here's the files

apt-get update
apt-get install -y git

# START SSH setup
INJECTED_SSH_CONFIG_DIR="/opt/atlassian/pipelines/agent/ssh"
# The default ssh key with open perms readable by alt uids
IDENTITY_FILE="${INJECTED_SSH_CONFIG_DIR}/id_rsa_tmp"
# The default known_hosts file
KNOWN_HOSTS_FILE="${INJECTED_SSH_CONFIG_DIR}/known_hosts"

mkdir -p ~/.ssh || debug "adding ssh keys to existing ~/.ssh"
touch ~/.ssh/authorized_keys

# If given, use SSH_KEY, otherwise check if the default is configured and use it
if [ "${SSH_KEY}" != "" ]; then
   debug "Using passed SSH_KEY"
   (umask  077 ; echo ${SSH_KEY} | base64 -d > ~/.ssh/pipelines_id)
elif [ ! -f ${IDENTITY_FILE} ]; then
   error "No default SSH key configured in Pipelines."
   exit 1
else
   debug "Using default ssh key"
   cp ${IDENTITY_FILE} ~/.ssh/pipelines_id
fi

if [ ! -f ${KNOWN_HOSTS_FILE} ]; then
    error "No SSH known_hosts configured in Pipelines."
    exit 2
fi

cat ${KNOWN_HOSTS_FILE} >> ~/.ssh/known_hosts
if [ -f ~/.ssh/config ]; then
    debug "Appending to existing ~/.ssh/config file"
fi
echo "IdentityFile ~/.ssh/pipelines_id" >> ~/.ssh/config
chmod -R go-rwx ~/.ssh/
# END SSH setup

curl -sSL https://install.python-poetry.org | python3 -
export PATH="$HOME/.local/bin:$PATH"
poetry --version
poetry config http-basic.artifact aws "$TOKEN"

npm install --save -g serverless-step-functions

pre execution script

service: ServerlessAppTemplate
frameworkVersion: ^3.38.0
package:
  individually: true
  patterns:
    - '!config/**'
    - '!lambda_functions/**'
    - '!*.json'
    - '!*.md'
    - '!*.txt'
    - '!.gitignore'
    - '!pytest.ini'
    - '!tests/**'
    - '!*.yml'
    - '!*.yaml'
    - '!Makefile'
    - '!poetry.lock'
    - '!pyproject.toml'

provider:
  name: aws
  runtime: python3.10
  httpApi:
    cors: true
  stage: ${opt:stage, "dev"}
  region: ${opt:region, "us-east-2"}
  deploymentBucket: 
    name: [redacted]-${self:provider.region}
  deploymentPrefix: Serverless
  environment:
    REGION: ${self:provider.region}
    STAGE: ${self:provider.stage}
    POWERTOOLS_SERVICE_NAME: template
    LOG_LEVEL: INFO

functions: ${file(config/functions.yml)}

resources: ${file(config/resources.yml)}

stepFunctions: ${file(config/state_machines.yml)}

plugins:
  - serverless-python-requirements
  - serverless-step-functions

custom:
  powertoolsLayer: arn:aws:lambda:${opt:region}:017000801446:layer:AWSLambdaPowertoolsPythonV2:67
  pythonRequirements:
    layer: true
    slim: true
    strip: false
    usePoetry: true

Serverless config with explicit usePoetry

image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
      path: .venv
  steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
        caches:
          - pip
          - poetry-deps
        image: atlassian/pipelines-awscli
        script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
          - export AWS_DEFAULT_REGION=us-east-2
          - TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain [redacted] --query authorizationToken --output text)
          - echo "export TOKEN=$TOKEN" >> set_env.sh
        artifacts:
          - set_env.sh
    - step: &audit-step
        name: Audit Packages
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry self add poetry-audit-plugin
          - poetry audit
    - step: &test-step
        name: Test
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run python3 -m pytest --cov=./ --cov-report xml
    - step: &lint-step
        name: Lint code
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run ruff check

    - step: &deploy-functions-step
        name: Deploy Functions
        caches:
          - docker
          - node
          - pip
        script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
            variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
              EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
              PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Dev, region us-east-2
          deployment: dev-us-east-2
          script:
            - source set_env.sh
            - <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables
  tags:
    'v*':
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          <<: *deploy-functions-step
          name: Deploy to stage Prod, region us-east-2
          deployment: prod-us-east-2
          trigger: manual
          script:
            - source set_env.sh
            - <<: *deploy-functions-script
              variables:
                <<: *deploy-functions-variables

pipelines file

image: python:3.10

definitions:
  caches:
    poetry-deps:
      key:
        files:
          - poetry.lock
      path: .venv
  steps:
    - step: &get-token-step
        name: Get CodeArtifact Token
        caches:
          - pip
          - poetry-deps
        image: atlassian/pipelines-awscli
        script:
          - export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
          - export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY
          - export AWS_DEFAULT_REGION=us-east-2
          - TOKEN=$(aws codeartifact get-authorization-token --region us-east-2 --domain patronpoint-artifact --query authorizationToken --output text)
          - echo "export TOKEN=$TOKEN" >> set_env.sh
        artifacts:
          - set_env.sh
    - step: &audit-step
        name: Audit Packages
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry self add poetry-audit-plugin
          - poetry audit
    - step: &test-step
        name: Test
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run python3 -m pytest --cov=./ --cov-report xml
          # Install deepsource CLI
          # - curl https://deepsource.io/cli | sh
          # From the root directory, run the report coverage command
          # upload coverage report to deepsource
          # - ./bin/deepsource report --analyzer test-coverage --key python --value-file ./coverage.xml
    - step: &lint-step
        name: Lint code
        caches:
          - docker
          - pip
          - poetry-deps
        script:
          - source set_env.sh
          - pipe: atlassian/poetry-cli-setup:0.2.0
          - POETRY_VIRTUALENVS_IN_PROJECT=true ./setup-poetry.sh
          - poetry config http-basic.artifact aws $TOKEN
          - poetry install
          - poetry run ruff check

    - step: &deploy-functions-step
        name: Deploy Functions
        caches:
          - docker
          - node
          - pip
        script:
          - &deploy-functions-script
            pipe: atlassian/serverless-deploy:1.5.0
            variables: &deploy-functions-variables
              AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
              AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
              EXTRA_ARGS: "-s $STAGE --region $REGION"
              DEBUG: "true"
              TOKEN: $TOKEN
              PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

pipelines:
  default:
    - step: *get-token-step
    - parallel:
        - step: *audit-step
        - step: *test-step
        - step: *lint-step

  branches:
    main:
      - step: *get-token-step
      - parallel:
          - step: *audit-step
          - step: *test-step
          - step: *lint-step
      - step:
          name: Deploy to stage Dev, region us-east-2
          caches:
            - docker
            - node
            - pip
          deployment: dev-us-east-2
          script:
            - source set_env.sh
            - pipe: atlassian/serverless-deploy:1.5.0
              variables:
                AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
                AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
                EXTRA_ARGS: "-s $STAGE --region $REGION"
                DEBUG: "true"
                TOKEN: $TOKEN
                PRE_EXECUTION_SCRIPT: './pipelines_pre_execution.sh'

Same issue with this pipelines file as well

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
× Stack ServerlessAppTemplate-dev failed to deploy (0s)
Environment: linux, node 16.20.2, framework 3.38.0, plugin 7.2.0, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn poetry ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:285:19)
at onErrorNT (node:internal/child_process:485:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
✖ Deployment failed :(

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

@Oleksandr Kyrdan Following up

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Sam Mahr

Could you try the new dev version of the pipe with Poetry in the pipe's base docker image before the release and share feedback:

- pipe: docker://bitbucketpipelines/serverless-deploy:1.5.0.163-dev

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Retrieving Poetry metadata
The latest version (1.8.3) is already installed.
Poetry (version 1.8.3)
eyJ2ZXIiOjEsImlzdSI6MTcxNTgyMjM2MiwiZW5jIjoiQTEyOEdDTSIsInRhZyI6IjQ1bzVrZ2J4b3BleXNjdU04TW9Qa0EiLCJleHAiOjE3MTU4NjU1NjIsImFsZyI6IkExMjhHQ01LVyIsIml2IjoiSXVveEsxT1dpREtyNW1qRiJ9.6uRiS-LVWp3B2eNfDi31LA.DT-uf1zhaXMHhYFJ.iB0zmDgkc0ykdf3bjA9iQAJL_5Xj9C2_Td1Oppes2xF9Krp6iTK9ffoYf5oqduOC8QtMMpET_4svNkXvLnfoi2iTBJW_W76j2rsWiEw-nPueM0-ff_J0ptZ4nc3DOJGYqfW6vtsfnlBkL3MfyyqO4ts7k01IFx7SURxHP8eTGXHFlRATtAwTgWNbwv4W4igCzirxYYNqcWUfcRkQthZrRKypOd1jMJ20_wJLpkwD5wPZl9BTWgWHZxTy9xXdHLXjldWK24CLe2X62NcpKTHcWol23BrTrecD4a8EqvAmjxURyaLrbsVI_NVQmrSX26z58WKpQ2x3DcVdDKvwUC5HcmhOUrD5-AF-4c-CaUNO7qVurjzCvmcNFCL5n0usSGdmILHR0864sf7E_INYaozyt8-BvMsyCHI154teaDCYUIoSNa7bMQRWwnBZLddpQijyGT5IYUmb2i6gVyH3EmV34dgVqksI9yDngrI2Ea5Lopasb-WBYliPK3srOZF4r7xTaS_vG-SMjAXTI1X_o9-WjhyYK4k1SrMrXMu3z_fFhcVeFI_pbDXBE13UaL7swr9RwxAjuS7AEtOnFuxG1PnsonrcH508sfnXJEuI0-gx8nOOfKrtkY6ZG5ed2oV4maMhGhX2inIaBwPF6L5etz4Nk0t2NEqJJ-AVo_gnGup6vIu8JhRVk1HYBjoHjTPKW-6eOlyluQ_l6ElwoAdJm3H3RoEGn8mbTH4r9ol8HsBW5KLJg0D5ccwBfR3EQ40r3EmVmp0V3bJGlIRYgTU_H1hGI6gxqqKAZHtNzY6ttTd-bJzQCtPttMW1bwCb5LH68LuDKtSKQCZvr4X_aAcrButnk4svVD2nCIo31hP9qSVEBo_KXsMgaIYuXDiX10cz7zGaJhTY4Wi32_Rcd_5jNYXSgukqKio.befgFQ-ZgjR0mZZnWdkYVQ
added 570 packages in 21s
113 packages are looking for funding
run `npm fund` for details

INFO: Deploying your serverless application...
DEBUG: serverless deploy -c serverless.yml -s dev --region us-east-2 --verbose

Deploying ServerlessAppTemplate to stage dev (us-east-2)
Generating requirements.txt from "pyproject.toml"
Parsed requirements.txt from pyproject.toml in /opt/atlassian/pipelines/agent/build/.serverless/requirements.txt
Installing requirements from "/root/.cache/serverless-python-requirements/1f0dd62c2bede82c87a91b6c441f0d0940af59d12684e386817a7ff2757d10a6_x86_64_slspyc/requirements.txt"
Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
× Stack ServerlessAppTemplate-dev failed to deploy (1s)
Environment: linux, node 20.13.1, framework 3.38.0, plugin 7.2.3, SDK 4.5.1
Credentials: Local, environment variables
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn python3.10 ENOENT
at ChildProcess._handle.onexit (node:internal/child_process:286:19)
at onErrorNT (node:internal/child_process:484:16)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
✖ Deployment failed :(

Hmm, it looks like there's still an issue.

The pre execution script tries to install poetry, but it's already installed in this docker image, but still getting this issue: "Error: spawn python3.10 ENOENT"

@Oleksandr Kyrdan

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

@Sam Mahr please, pay attention that pipe's docker image based on python:3.11-slim. Looks like you try to spawn python3.10 instead.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Ah, I did not look at the image. I've changed everything to 3.11 and now it is working and deployed! @Oleksandr Kyrdan

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Like • Oleksandr Kyrdan likes this

Nice! Thanks for your feedback! 👍

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Sam Mahr

A new version of serverless-deploy pipe is released:

- pipe: atlassian/serverless-deploy:2.0.0

changes:

2.0.0

major: Breaking changes. Refactored pipe to support better auth logic and major providers AWS, GCP, AZURE.
minor: Add OIDC support for AWS provider. Add support for AWS_OIDC_ROLE_ARN variable.
minor: Add Poetry to the pipe's base docker image as an alternative tool for dependency management and packaging in Python.
minor: Improved support for Azure cloud. Add serverless-azure-functions to the default environment.
minor: Update base NodeJS version to nodejs20.
minor: Update pipe's base docker image to python:3.11-slim.
patch: Internal maintenance: bump project's dependencies to the latest.

Best regards,
Oleksandr Kyrdan

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Suggest an answer

Was this helpful?

Thanks!

Bitbucket

DEPLOYMENT TYPE

CLOUD

PERMISSIONS LEVEL

Site Admin

Missed Team ’24? Catch up on announcements here.

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?

1 answer

Suggest an answer

Was this helpful?

Thanks!

DEPLOYMENT TYPE

PERMISSIONS LEVEL

TAGS

Atlassian Community Events

Missed Team ’24? Catch up on announcements here.

Ask a question

Start a discussion

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Why does serverless-deploy pipe use serverless-python-requirements without poetry installed?

1 answer

Suggest an answer

Was this helpful?

Thanks!

DEPLOYMENT TYPE

PERMISSIONS LEVEL

TAGS

Atlassian Community Events