How to avoid timeouts / 502 errors with long running REST tasks

[Anonymouslemming]

Hi all,

We need to delete a large number of projects from a number of JIRA instances. I've written a REST plugin to expose the ProjectService.DeleteProjectResult method from the JIRA API. 

This works fine for small projects with few issues, but for larger projects with thousands of issues, this fails. It runs for a while, then we eventually receive a 502 (Bad Gateway) error.

Attempting to delete similar sized projects in the UI has similar behaviour - we see the delete progress for a while, then we get a network timeout error. 

In both cases, the logs imply that the delete activity is ongoing after the failure. We see similar results when doing exports of large instances - a network timeout with the export continuing to run in the background.

What's the right way of keeping these kinds of long-running requests alive? 

I did consider iterating over all issues and deleting them first, but we have dozens of permission schemes in use, and most of these have no delete permission for any role. This was necessary to ensure that users can't delete things and break audit requirements. 

The code that the plugin is implementing is as follows:

ProjectService.GetProjectResult projectResult = projectService.getProjectByKey(loggedInUser, projectkey);
Project project = projectResult.getProject();

if (project == null) {
    logger.warn("Unable to lookup " + projectkey + " - Will not delete");
    return Response.status(Response.Status.NOT_FOUND).entity("Failed to lookup project " + projectkey).cacheControl(never()).build();
}
SimpleErrorCollection errorCollection = new SimpleErrorCollection();
ProjectService.DeleteProjectResult deleteProjectResult = projectService.deleteProject(loggedInUser, new ProjectService.DeleteProjectValidationResult(errorCollection, project));
if (deleteProjectResult.isValid()) {
    logger.info("Successfully deleted " + projectkey);
    return Response.status(Response.Status.OK).entity("Deleted Project").cacheControl(never()).build();
} else {
    logger.warn("Failed to delete " + projectkey);
    return Response.status(Response.Status.NOT_FOUND).entity("Failed to delete project").cacheControl(never()).build();
}

[MattS]

I've had problems with this before and ended up using a jira-python script to delete the issues slowly. Many permission schemes allow members of jira-administrators to do the same thing in multiple projects. Or can you extend your REST add-on to allow you to delete issues with the same authentication?

[Anonymouslemming]

We have permission schemes that block us from being able to delete issues. I do like the idea though and will look into whether or not we can delete issues in our own plugin without meeting the requirements of the permission schemes - that might solve the issue for us.

[Jonas Andersson]

Are you using a mod proxy infront of your Jira? By the sound of the issue, you do. If this is the case the timeouts can be configured in apache. If i am on the right track, let me know and i can tell you more. And upvote my answer  I spend a lot of time helping but few take the time to give me a click! 

[Anonymouslemming]

We're not using mod_proxy, no, sorry.

[Volodymyr Krupach]

The only thing that comes to my mind is to start separate thread and return immediately from your REST. Then in your page you need to have JavaScript that pulls other rest method that checks if the thread is alive. While it's alive you show "progress icon" and show "Done status" when the rest returns "thread completed".

[Anonymouslemming]

Thanks for the feedback. I'm not quite sure I follow though. I can see how we'd create the thread to carry out the delete action, but how would we use a different REST request to get information about that thread ? What information would we have to return from our original request to allow us to monitor the thread status (and exit status especially!) from another REST request ? Thanks,

[Volodymyr Krupach]

Rest service classes (classes with methods annotated by annotations from javax.ws.rs) are singletons. It means that all you requests will be served by the same instances so you can keep thread reference in your class and have there 2 REST methods: delete and checkStatus. Also I suggest to make sure that "delete" does not spawn new threads while deletion is in progress.

[Anonymouslemming]

Could you point me to some documentation that explains that a little better please? I'm not really a developer, I just impersonate one when we have a DevOps task like this :) I get that we could store the thread reference in our class, but I don't understand how we'd manage multiple requests. Assuming that User A kicks off a request to delete project FOO - we store the thread reference and checkstatus monitors that thread. User A tries to delete project FOO again, but we check that this is already in progress and do not allow the action. User B then kicks off a request to delete project BAR. At this point I'm lost. Are you saying that only 1 delete should be alive at a time and that user B shouldn't be able to trigger the delete of BAR until FOO is finished deleting ? Thanks!

[Volodymyr Krupach]

> Are you saying that only 1 delete should be alive at a time and that user B shouldn't be able to trigger the delete of BAR until FOO is finished deleting ? Yes, exactly. Project deletion could be very heavy operation and I recommend you to allow only one at a time.