Runners in "UNREGISTERED" state

Hello,

I have deployed the application in GCP k8s, and I can view the objects created:

And I can view sometimes a new runner is added to my Bitbucket workspace:

The problem is that the runner is alwais in the "UNREGISTERED" state. I think the Controller and the Controller Cleaner are creating and destroying the runners, but the runners are not working properly.

I can also view the secrets created:

I have used the templates published here:

https://bitbucket.org/bitbucketpipelines/runners-autoscaler/src/master/config/

These are the Controller logs (most recent on top):

An these are the runner logs (most recent on top):

Finally, the pipeline never starts:

This is the pipeline definition:

___

# This is an example Starter pipeline configuration

# Use a skeleton to build, test and deploy using manual and parallel steps

# -----

# You can specify a custom docker image from Docker Hub as your build environment.

image: atlassian/default-image:3

pipelines:

default:

- parallel:

- step:

runs-on:

- 'grp1'

- 'self.hosted'

- 'linux'

script:

- echo "Your build and test goes here..."

- ./some_script.sh

- step:

script:

- echo "Your linting goes here..."

- step:

script:

- echo "Your security scan goes here..."

# The following deployment steps will be executed for each pipeline run. To configure your steps and conditionally deploy see https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/

- step:

deployment: staging

script:

- echo "Your deployment to staging script goes here..."

- step:

deployment: production

trigger: 'manual'

script:

- echo "Your deployment to production script goes here..."

___

Thank you very much for your help.

2 answers

Suggest an answer

0 votes

@Aurelio Pérez Noriega update.
I can confirm that the bug exists with runners always in UNREGISTERED state.

In some cases when green k8s jobs linked to non-existing runners (example when runners are manually deleted); and red k8s jobs linked to runners that UNREGISTERED and can't be in ONLINE state because k8s jobs are not ready.

As a workaround to solve this you could manually delete k8s green jobs, where non-existing runners linked (in logs they are always trying to change state to ONLINE with no success)

Regards, Igor

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Igor Stoyanov

I always start the tests with zero runners in the workspace:

I always start the tests deleting all created objects and namespaces in k8s.
I never delete manually any runner during the tests.
I never delete manually any k8s job during the tests.

My purpose is to have an script to install all objects in k8s with no manual interaction, and to have the k8s jobs running OK in the Bitbucket Workspace. I think the workaround is not valid for my team.

Maybe you are close to have a solution to fix the problem. Can you give me an ETA to have a fixed version?

Thank you very much for your help.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

@Aurelio Pérez Noriega please contact our support team.

Regards, Igor.

Alternatively, i could suggest you to check:
- your k8s jobs are running (green status);

- in your k8s jobs there are logs presented with "waiting for runner ONLINE state"

- also try manually create runner via Runners page (without using this tool), and see if runner become in ONLINE state.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Igor Stoyanov ,

I have tested 2.2.3 version and it works OK.

Thank you very much :)

Regards!

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Like • Igor Stoyanov likes this

0 votes

Hi @Aurelio Pérez Noriega .
So your runners in Bitbucket API never became in ONLINE state?
You should wait some minutes (1-2) after runner created to change state from UNREGISTERED to ONLINE.

Also provide us your runners-autoscaler-config configMap.

Keep in mind if you set to low value for

runner_cool_down_period

variable, cleaner will try to delete your runner before runner change the state to ONLINE.

Also for testing you can always start autoscaler without cleaner deployment, and create cleaner deployment later when your runners will be in ONLINE state.

Also the Controller logs are the most useful logs for us, please provide more information related to controller logs.

For testing you can lower value of

runner_api_polling_interval

to in example 60 (seconds) to check the status of your configuration more frequently.

Regards, Igor

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Hi @Igor Stoyanov

Correct: the runners in Bitbucket API never became in ONLINE state. I have waited more than 6 minutes, and the state never changes.

This is the runners-autoscaler-config configMap that I have tried at first:

___

apiVersion: v1

kind: ConfigMap

metadata:

data:

runners_config.yaml: |

constants:

default_sleep_time_runner_setup: 10 # seconds. Time between runners creation.

default_sleep_time_runner_delete: 5 # seconds. Time between runners deletion.

runner_api_polling_interval: 600 # seconds. Time between requests to Bitbucket API.

runner_cool_down_period: 300 # seconds. Time reserved for runner to set up.

groups:

- name: "GCP k8s" # Name of the Runner displayed in the Bitbucket Runner UI.

workspace: "testure" # TODO - Replace the workspace guid - if repository is Not specified, it creates workspace runners.

labels: # runner will be created with the following labels

- "grp1"

namespace: "runners" # namespace where runners are going to be created

strategy: "percentageRunnersIdle" # in the future more strategies will be supported

parameters:

min: 1 # recommended minimum 1 must be in UI because fails, when new build is starting

max: 10 # max number of runners

scaleUpThreshold: 0.5 # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up

scaleDownThreshold: 0.2 # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up

scaleUpMultiplier: 1.5 # scaleUpMultiplier > 1

scaleDownMultiplier: 0.5 # 0 < scaleDownMultiplier < 1

___

The first image I have put in the first post is from the "Controller logs", and that was all information related to.

I am going to make another test using your suggestions:

Increase "runner_cool_down_period" to 600.
Decrease "runner_api_polling_interval" to 60.
Start without cleaner deployment.

This is the new suggested runners-autoscaler-config configMap:

___

apiVersion: v1

kind: ConfigMap

metadata:

data:

runners_config.yaml: |

constants:

default_sleep_time_runner_setup: 10 # seconds. Time between runners creation.

default_sleep_time_runner_delete: 5 # seconds. Time between runners deletion.

runner_api_polling_interval: 60 # seconds. Time between requests to Bitbucket API.

runner_cool_down_period: 600 # seconds. Time reserved for runner to set up.

groups:

- name: "GCP k8s" # Name of the Runner displayed in the Bitbucket Runner UI.

workspace: "testure" # TODO - Replace the workspace guid - if repository is Not specified, it creates workspace runners.

labels: # runner will be created with the following labels

- "grp1"

namespace: "runners" # namespace where runners are going to be created

strategy: "percentageRunnersIdle" # in the future more strategies will be supported

parameters:

min: 1 # recommended minimum 1 must be in UI because fails, when new build is starting

max: 10 # max number of runners

scaleUpThreshold: 0.5 # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up

scaleDownThreshold: 0.2 # The percentage of busy runners at which the number of desired runners are re-evaluated to scale up

scaleUpMultiplier: 1.5 # scaleUpMultiplier > 1

scaleDownMultiplier: 0.5 # 0 < scaleDownMultiplier < 1

___

In the new test, these are the Controller logs (most recent on top):

The same logs in text format:

___

"INFO: Autoscaler next attempt in 60 seconds..."

"✔ [GCP k8s] Successfully setup runner UUID 86e547c9-34dd-532e-b96f-522dc3c34f8e on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-86e547c9-34dd-532e-b96f-522dc3c34f8e"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] {'UNREGISTERED': 5}"

"INFO: [GCP k8s] Found 5 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: [GCP k8s] Working on runner group: workspace=NameUUIDData(name='testure', uuid='a267b0a5-4a29-4f42-b011-6e78ffbd6c4c') name='GCP k8s' namespace='runners' strategy='percentageRunnersIdle' repository=None labels={'self.hosted', 'grp1', 'linux'} parameters=PctRunnersIdleParameters(min=1, max=10, scale_up_threshold=0.5, scale_down_threshold=0.2, scale_up_multiplier=1.5, scale_down_multiplier=0.5)"

"INFO: Autoscaler runners: [GroupData(workspace=NameUUIDData(name='testure', uuid='a267b0a5-4a29-4f42-b011-6e78ffbd6c4c'), name='GCP k8s', namespace='runners', strategy='percentageRunnersIdle', repository=None, labels={'self.hosted', 'grp1', 'linux'}, parameters=PctRunnersIdleParameters(min=1, max=10, scale_up_threshold=0.5, scale_down_threshold=0.2, scale_up_multiplier=1.5, scale_down_multiplier=0.5))]"

"INFO: Autoscaler config: constants=Constants(default_sleep_time_runner_setup=10, default_sleep_time_runner_delete=5, runner_api_polling_interval=60, runner_cool_down_period=600) groups=[GroupData(workspace=NameUUIDData(name='testure', uuid='a267b0a5-4a29-4f42-b011-6e78ffbd6c4c'), name='GCP k8s', namespace='runners', strategy='percentageRunnersIdle', repository=None, labels={'self.hosted', 'grp1', 'linux'}, parameters=PctRunnersIdleParameters(min=1, max=10, scale_up_threshold=0.5, scale_down_threshold=0.2, scale_up_multiplier=1.5, scale_down_multiplier=0.5))]"

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

"INFO: Autoscaler next attempt in 60 seconds..."

"\u001b[32m✔ [GCP k8s] Successfully setup runner UUID 502456f2-b5ac-53b4-a0ea-1ff52fc327e6 on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-502456f2-b5ac-53b4-a0ea-1ff52fc327e6"

"INFO: [GCP k8s] Secret created. status=runner-oauth-credentials-502456f2-b5ac-53b4-a0ea-1ff52fc327e6"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] {'UNREGISTERED': 4}"

"INFO: [GCP k8s] Found 4 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

"INFO: Autoscaler next attempt in 60 seconds..."

"\u001b[32m✔ [GCP k8s] Successfully setup runner UUID 771c9775-3263-5525-a457-ca26deb9dd44 on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-771c9775-3263-5525-a457-ca26deb9dd44"

"INFO: [GCP k8s] Secret created. status=runner-oauth-credentials-771c9775-3263-5525-a457-ca26deb9dd44"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] {'UNREGISTERED': 3}"

"INFO: [GCP k8s] Found 3 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

"INFO: Autoscaler next attempt in 60 seconds..."

"\u001b[32m✔ [GCP k8s] Successfully setup runner UUID 44d452d7-ee08-52d7-8f74-d393e66ce494 on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-44d452d7-ee08-52d7-8f74-d393e66ce494"

"INFO: [GCP k8s] Secret created. status=runner-oauth-credentials-44d452d7-ee08-52d7-8f74-d393e66ce494"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] {'UNREGISTERED': 2}"

"INFO: [GCP k8s] Found 2 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

"INFO: Autoscaler next attempt in 60 seconds..."

"\u001b[32m✔ [GCP k8s] Successfully setup runner UUID 3d4507c7-104f-58c0-aa96-a5f6b7efecfa on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-3d4507c7-104f-58c0-aa96-a5f6b7efecfa"

"INFO: [GCP k8s] Secret created. status=runner-oauth-credentials-3d4507c7-104f-58c0-aa96-a5f6b7efecfa"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] {'UNREGISTERED': 1}"

"INFO: [GCP k8s] Found 1 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

"INFO: Autoscaler next attempt in 60 seconds..."

"\u001b[32m✔ [GCP k8s] Successfully setup runner UUID 5323962c-3b6c-5086-9a59-4ce9ee4c39fc on workspace testure"

"INFO: [GCP k8s] Job created. status=runner-5323962c-3b6c-5086-9a59-4ce9ee4c39fc"

"INFO: [GCP k8s] Secret created. status=runner-oauth-credentials-5323962c-3b6c-5086-9a59-4ce9ee4c39fc"

"INFO: [GCP k8s] Starting to setup the Kubernetes job ..."

"INFO: [GCP k8s] Runner created on Bitbucket workspace: testure"

"INFO: [GCP k8s] Starting to setup runner on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runner #1 for namespace: runners setup..."

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Runners Autoscaler. min: 1, max: 10, current: 0, desired: 1. Changing the desired capacity from 0 to 1."

"INFO: [GCP k8s] Current runners threshold: 0"

"INFO: [GCP k8s] Found IDLE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found ONLINE runners with labels {'self.hosted', 'grp1', 'linux'}: 0"

"INFO: [GCP k8s] Found 0 runners on workspace testure"

"INFO: [GCP k8s] Getting runners on Bitbucket workspace: testure ..."

"INFO: [GCP k8s] Namespace runners found."

"INFO: [GCP k8s] Getting runners namespace in Kubernetes..."

"INFO: File /opt/conf/job_template/job.yaml.template copied to /home/bitbucket/autoscaler/resources/"

"INFO: Config file provided /opt/conf/config/runners_config.yaml."

___

And the runners created:

Thank you very much for your help.

You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.

Comment

Was this helpful?

Thanks!

Bitbucket Pipelines: Runner Autoscaler for Kubernetes

Missed Team ’24? Catch up on announcements here.

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Runners in "UNREGISTERED" state

2 answers

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events

Missed Team ’24? Catch up on announcements here.

Ask a question

Start a discussion

Products

Community resources

Support

Top groups

Community resources

Support

Learn

Community resources

Support

Events

Community resources

Support

Get product advice from experts

Join a community group

Advance your career with learning paths

Earn badges and rewards

Connect and share ideas at events

Runners in "UNREGISTERED" state

2 answers

Suggest an answer

Was this helpful?

Thanks!

TAGS

Atlassian Community Events