Introducing Custom merge checks, powered by Forge

[#1]

Hi all - as many of you will know, the Bitbucket Cloud team has been hard at work bringing a significant set of new features to the product that builds on top of our initial implementation of Forge.

Today, we're excited to announce the EAP of custom merge checks, powered by Forge. The Bitbucket Cloud side of the integration will be listed as beta. You can find out more by reading the release blog here https://bitbucket.org/blog/beta-custom-merge-checks-in-bitbucket-cloud .

Technical documentation can be found here: https://developer.atlassian.com/platform/forge/manifest-reference/modules/bitbucket-merge-check/  

Note: The release of this beta also corresponds with this group being made public, so hopefully we'll see an increase in participation in the following weeks. Please continue to use this group to share what you’re working on, providing feedback and reporting any issues that you run into.

We’re very excited to see what you build with this new functionality.

@Edmund Munday (Bitbucket Product Manager), @Dan Fraser (Bitbucket Software Engineer) and @Caterina Curti (Developer Advocate)

[Nikolai Marov]

Hello, feature looks great, thanks for releasing EAP!

Currently we struggle with an issue and I don't know how to solve it. We have well defined commit message format in our documentation, and we use Squash merge strategy. The problem is that it's not possible to validate commit message when user merge PR (and developers forget to change the message before merge PR). I was hoping that custom merge checks will allow that, but merge commit message is not available in handler function.

Do you have plans to pass merge commit message as an input parameter to handler function on on-merge trigger?

Would it be possible to manipulate merge dialog from a Forge application somehow?

It would be perfect it we'll able to:

• generate default commit message (not a static text) when user clicks Merge in PR and put it into dialog by default.
• validate commit message when user clicks merge second time in Dialog.

Cheers,
Nikolai

[Dan Fraser]

Hi @Nikolai Marov

Thanks for your feedback, sorry it's taken so long for us to reply, today is my first day back for 2024.

Let me address both points individually, as I believe their independent feature requests.

Validate commit message when user clicks merge second time in Dialog

I think exposing the submitted commit message, in some form, when the merge is invoked is definitely the kind of feedback we were hoping to get from this EAP. Whether we expose that information via injecting it as a context parameter to the Forge function, or we extend an existing API to expose this information, I'm not sure yet. I've raised a public BCLOUD ticket to track this suggestion here: BCLOUD-23049. Please upvote the ticket when you get a chance.

Generate default commit message (not a static text) when user clicks Merge in PR and put it into dialog by default.

This feels like it is probably outside of the scope of the Custom merge checks implementation, and would likely be a Forge feature request for Bitbucket Cloud.

In saying that, I think Forge (in it's current implementation) can get you most of the way there. You could have a merge button in a Forge UI panel on the PR view. When the button is pressed could invoke the call to the PR Merge API (as the calling user) and you can then provide the custom/dynamic commit message in the body of the request. It's not a perfect solution, but would still get you most of the way there. I think this is the best way forward, If you feel this is an inadequate solution, please let us know. I'd like to keep the two features separate as they're not tightly related.

Hope that helps,

@Dan Fraser 

[Nikolai Marov]

Hi @Dan Fraser ,

thanks for the answers!

We will wait https://jira.atlassian.com/browse/BCLOUD-23049 then.
Option with a custom merge button via Forge is nice. Potentially I can call a custom Dialog when developer clicks custom merge button and perform validation there, but it looks more like a workaround (we still have the main merge button, a custom merge button is under sub-menu).

Regards,
Nikolai

[Aron Gombas _Midori_]

@Dan Fraser We started to build our first merge check prototype with the Forge facility you released. (We have a Bitbucket DC app that implements many of these checks, and it has 300+ installation.)

Things are looking fine so far (thanks for making this possible!), although it is just some prototyping work. But, we found a major bug related to repository permissions. It is either a bug in the functionality or the UI does not behave intuitively...

Could you please find some to have a look at this? https://ecosystem.atlassian.net/servicedesk/customer/portal/34/ECOHELP-33116

[Aron Gombas _Midori_]

@Edmund Munday @Dan Fraser 

One month later, the problem we found early Jan is still present. In the meanwhile, Atlassian replied to us that they designed custom merge checks in a way that they will actually work only if the current user has WRITE permission on the target repo?!! 🙁 

Which is not going to be true in 99% of the cases. (You collaborate using PRs exactly because you don't want individual developers have write permission on the target repo...)

Example

Say we have a merge check which tests if the PR summary is at least 10 chars long.

1. Sender creates the PR with 8 char long summary
2. The merge check evaluates to "rejected" (correct!)
3. Sender rewrites the summary to 11 chars
4. Sender can not refresh the merge check status unless he has WRITE permission: 
   1. if he clicks the icon, the spinner turns infinitely and nothing happens
   2. if he reloads the screen the merge check is still "rejected" (incorrect!)
   3. ultimately, he will never be able to make it green
5. Because of the "rejected" merge check, the PR won't be merged (incorrect!)

We reported it weeks ago: ECOHELP-33116

I hope Atlassian fixes it very soon. For now I just wanted to warn other developers that with this bug in place, the whole concept is totally broken and useless.

[Edmund Munday]

Hi @Aron Gombas _Midori_ - apologies for the slow reply, I didn't see the initial notification for your comment for some reason.

I'll review this with the team today to understand the behaviour you're experiencing.

[Edmund Munday]

Hi @Aron Gombas _Midori_

First up, thank you for raising these issues, we do genuinely appreciate it and it makes a significant difference to the direction of this work. Additionally, I want to apologise for the time it’s taken to get an official response back to you.

To reiterate the issue:

• Whilst Custom merge checks do function as expected in the scenario where a users ability to push to a repository is restricted via Branch Restrictions (rather than the removal of write access on the repo entirely).

• The behaviour of Custom merge checks is different in situations where the user is following a fork-based contribution workflow.

  • This results in a scenario where Custom merge checks are not being executed on PR’s that are originating from a forked repository, where the user does not have write access to the target repository.

The experience you encountered here is absolutely not the desired outcome for this process, and we are doing several things to address this.

1. We are resolving the UX issue you encountered with the infinite spinner and locking page. No matter what the overarching user-flow ends up being, that kind of experience is not what we want for users. This will be addressed in an upcoming update to the Custom merge check feature.

2. In regards to the overall behaviour of Custom merge checks when a user does not have write access to the target repository, I want to provide a bit more context.

   1. The reason this is not just a simple matter of changing the behaviour is due to the way this scenario would interact with the overall permissions model of Bitbucket Cloud. We need to be very comprehensive when working on any feature that impacts this space, as security is always our #1 priority.

   2. To be specific, the concern here comes about through the fact that a Forge app installed in a Workspace will often have the ability to make changes within the Workspace it is installed against, and this is no different for Custom merge check apps.

   3. What this means is, in a situation where a user does not have write access to the repo that the PR is targeted towards, but they do have the ability to execute an action that would trigger Custom merge checks to run inside that repo, that user is effectively being given the ability to potentially trigger some kind of action against the repo that could result in a change that the user themselves would not have had the permission to do.

   4. This is not, inherently, an issue - as Forge apps represent a distinct identity with distinct permissions. However, we need to be really careful, and really explicit with how we allow this functionality to work.

I can’t give you a guaranteed answer on what the outcome is going to be here right now, but I can commit to you that it is actively being looked into. The end-state goal would be for it to be possible for customers to have Custom merge checks enforced on all changes to a repository, we just need to make sure that is done in a way that allows the customer to always be in control of who/what can make changes to their Intellectual Property (IP).

We want to make sure that we get as much input from partners and customers as possible when making these kinds of behaviour defining decisions, so we have made a dedicated suggestion to track feedback on this specifically: https://jira.atlassian.com/browse/BCLOUD-23113.

Given the additional context I’ve shared here around the permissions component to the behaviour in question, if you have any additional thoughts we would really appreciate you sharing them with us.

As we make progress on addressing this, I will make sure updates are shared on the BCLOUD-23113 suggestion ticket.

Regards,

Edmund.

[Aron Gombas _Midori_]

@Edmund Munday Thanks for writing this comprehensive answer.

I appreciate that you will re-build the interaction and get rid of the infinite-spinner. I agree, this is the absolute minimum that should be done here. If you ping me when it is rolled out, we can verify if it works as expected - if it helps you.

I will discuss the other parts about the permission model with my developer colleague. If we have any feedback on that, we will post to https://jira.atlassian.com/browse/BCLOUD-23113

Thanks again!

[Gergely Juhász _Midori_]

Hi @Edmund Munday,

I noticed that Custom merge check is independent of the built-in Merge checks, that can be configured under Branch restrictions/Merge settings. They behave differently when it comes to pull request mergeability.

The built-in Merge checks (if not passed) do not prevent the pull request from being merged by default. However there is an option "Prevent a merge with unresolved merge checks" which is a Premium feature.
In contrast, the Custom merge check does prevent it by default, regardless of the plan or branch restrictions applied, so it cannot be configured.

Is this intentional behavior? Or a bug? Or do you plan to change this during the EAP?

[Edmund Munday]

Hi @Gergely Juhász _Midori_ - really excellent question, thanks for asking!

The short answer is "This is a limitation of the open beta", when we go to GA, the behaviour between the two types of Checks will be effectively harmonised, at least in terms of standard/premium - however Custom Merge Checks will have a level of enhanced capability in this regards.

We're going to be announcing more detail on this next week, but I'll give you a sneak-preview.

1. Classic Merge Checks and Custom Merge Checks are being merged into a single UI component on the PR screen so that users don't need to keep track of them separately.
2. The UI behaviour of the two types of Checks is also being standardised (following the model set by Custom Merge Checks) so that there's a single consistent set of behaviours for users to understand.
3. In terms of configuration and management, they will still be configured separately in the repository settings.
   1. There is a very good, but slightly long-winded reason for this, which relates to our plans to enable customers to configure sets of Custom Merge Checks at the Workspace and Project level and then "push" those rules down the stack.
   2. Basically, the current branch-restrictions configuration scheme which allows configuration schemes to be linked to arbitrary branch-patterns at the repository level breaks any ability we would have to create standardised "branching patterns" across workspaces, restricting us from implementing any kind of proper "inherited Checks configurations".
   3. I don't want to go super into detail on this here because it's not something we've 100% decided on the path for yet - we just know we don't want Custom Merge Checks to follow the same setup as Classic checks, so we have restricted the granularity of their configuration for the time being to "all branches" and the main branch of the repository.
   4. We do plan to add more granularity to this branch targeting in the future.
4. In terms of Standard vs Premium, once we go GA with Custom Merge Checks, repository admins in Premium workspaces will be able to configure whether they want each Check to be enabled in either a "Recommended" or a "Required" state.
   1. If a Check is "Recommended", it will run as today but will not block the merge if it fails.
      1. The result will look different in the UI to what it does today, providing a visual distinction between "checks that block the merge" and "checks that are recommendations".
      2. It will still be clear to the user that the Check failed, and the Check app will still be able to return a message to the user providing them with feedback.
   2. If a Check is "Required", it will run exactly as today, blocking the merge from going ahead.
   3. Note: This means that in a Premium workspace, repository administrators will be able to configure some Checks as Recommended, and some as Required, granting significantly more flexibility than the Classic Check scheme which sets everything to the same rule.
      1. We believe this is a significant improvement over the traditional model as it gives admins the ability to create nuanced Check schemes or gently onboard teams to following new standards or rules before gradually enforcing those rules over time.
5. In Standard workspaces, only the "Recommended" option will be available, essentially restricting the blocking of merges via Checks to Premium workspaces only.
   1. This means the scheme is effectively in-line with the behaviour of Classic checks, although with an additional level of granularity for admins in Premium workspaces.

[Gergely Juhász _Midori_]

Thanks @Edmund Munday, for your detailed answer!