groovy.util.ResourceException: Cannot open URL: bundle:

[Chris Solgat]

After upgrading from JIRA 6.1.6 to JIRA 6.3.15, keep getting this error during the re-index.  I have added the JVM argument to point to where our scripts are stored -

-Dplugin.script.roots=/data/atlassian/jira/scriptrunner

This is what shows up at the bottom of the stack trace - 

Caused by: groovy.util.ResourceException: Cannot open URL: file:/data/atlassian/runtime/rpd/scripts//data/atlassian/jira/scriptrunner/RPD/STRDocumentNeededforIssue-CourtOrder.groovy
	... 31 more
Caused by: groovy.util.ResourceException: Cannot open URL: file:/data/atlassian/jira/scriptrunner//data/atlassian/jira/scriptrunner/RPD/STRDocumentNeededforIssue-CourtOrder.groovy

Any ideas what could be causing this?

[JamieA]

You are probably using an absolute path name for your scripts...? It should still work fine though, so not really sure. 

Just to rule things out, can you change one instance to just: RPD/STRDocumentNeededforIssue-CourtOrder.groovy and see if that improves things.

If you're getting problems on reindex you are most likely using script fields, so you can check this by going to Admin -> Script Fields and doing a preview.

[Phillip Ponzer [Cprime]]

REMOVE THE LEADING FORWARD-SLASH! Why didn't I see this before? Ugh.... Thanks Jamie! :)

[Jeff Turner]

We just had the same problem with JIRA 6.3.12 and Groovy runner 3.0.7 failing mid-way through a reindex, with "Cannot open URL" errors like:

2015-11-26 00:09:41,948 IssueIndexer:thread-9 WARN jturner 6x4406x1 vqlbxf 172.17.8.118 /secure/admin/jira/IndexReIndex.jspa [onresolve.scriptrunner.runner.ScriptRunnerImpl] RE
groovy.util.ResourceException: Cannot open URL: bundle://156.0:0/approval_date.groovy
        at groovy.util.GroovyScriptEngine.getResourceConnection(GroovyScriptEngine.java:399)
        at groovy.util.ResourceConnector$getResourceConnection.call(Unknown Source)
        at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.runScript(ScriptRunnerImpl.groovy:398)
        at com.onresolve.scriptrunner.runner.ScriptRunner$runScript$8.callCurrent(Unknown Source)
        at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.runFileAsScript(ScriptRunnerImpl.groovy:182)
        at com.onresolve.scriptrunner.runner.ScriptRunner$runFileAsScript$6.call(Unknown Source)
        at com.onresolve.scriptrunner.customfield.GroovyCustomField.getValueFromIssue(GroovyCustomField.groovy:144)
        at com.atlassian.jira.issue.fields.CustomFieldImpl.getValue(CustomFieldImpl.java:437)
        at com.atlassian.jira.issue.index.indexers.impl.DateCustomFieldIndexer.addDocumentFields(DateCutomFieldIndexer.java:44)
        at com.atlassian.jira.issue.index.indexers.impl.DateCustomFieldIndexer.addDocumentFieldsSearchable(DateCustomFieldIndexer.java:34)
        at com.atlassian.jira.issue.index.indexers.impl.AbstractCustomFieldIndexer.addIndex(AbstractCustomFieldIndexer.java:46)
        at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:102)
...
Caused by: groovy.util.ResourceException: Cannot open URL: file:/var/atlassian/application-data/jira/6.3.12/scripts/approval_date.groovy
        at groovy.util.GroovyScriptEngine.getResourceConnection(GroovyScriptEngine.java:397)
        ... 27 more

Thanks to this thread I noticed the (infrequently) interspersed 'Too many open files' errors:

file:/var/atlassian/application-data/jira/6.3.12/scripts/util/FakeMLJIRA.groovy: /var/atlassian/application-data/jira/6.3.12/scripts/util/FakeMLJIRA.groovy (Too many open files)

We have 7 script fields. During a reindex I checked open file handles with 'lsof' and found ~110 file handles for each .groovy file, for a total of 776, well on the way to hitting the 1024 default max.

Just adding ulimit -n 10000 to the beginning of bin/setenv.sh fixes this, but the fact that there's 110 file handles open per .groovy file suggests things are severely non-optimal. It looks like the .groovy script is being re-parsed and re-processed once for every issue.

[JamieA]

I would upgrade to a more recent version, even to the latest free version. I can say categorically that it will not be compilng your script every time, you can verify that by turning DEBUG on for com.onresolve. When it compiles something you will see a line like: DEBUG anonymous [onresolve.scriptrunner.runner.WithPluginCompilationCustomiser] Compile file:/... to node ... An open file limit of 1024 is too small for a non-trivial jira instance anyway. We'd love to work out what the problem is here, if you would ideally upgrade first and if it's still a problem open a ticket at https://productsupport.adaptavist.com/servicedesk/customer/portal/2.

[Chris Solgat]

We made the changes to the ulimit settings and everything looks good.  I have run the re-index a few times, and have not seen the errors show up.  Please convert your ulimit comment to an answer and I will check it as accepted.

Secondarily, I posted a screen shot of an error I received when trying to run the Script Registry built in script.  Any ideas on why it's doing that?  We did not have the error on the previous version of Script Runner.

[Chris Solgat]

image2015-4-10 6:59:43.png

Any idea if this is related to JIRA not being able to find the Scripts?

[Chris Solgat]

We have dug into this a little bit further, but have not come up with anything.  Unfortunately, I have a new wrench to throw into the works.  So far we have upgraded four different environments (we have a total of 22) that use a decent number of Scripted Fields (between 4 - 12) that are pointed to a script living on the file system.  Two of them have not had any problems during the re-index.  Two of them 'Cannot find/open the file' containing the scripts.  We will look at the differences between them, but at the surface, there is not much difference.  One thing that we thought may be part of the problem is that we also have sub-directories within the scriptrunner home location.  I changed the home location to point to the sub-directory, but it still came out with the same error as we initially mentioned.

[JamieA]

Random failures point to the ulimit that I mentioned previously.

[Chris Solgat]

Any ideas about the new error message about Too many files open?  Our JIRA instance relies heavily on these scripted fields and we need to try to find an answer.

[JamieA]

Can you check your ulimit settings for the number of open files parameter. lsof will show you which files are open, but you probably want to talk to your sysadmin. It's not a Script Runner issue, unless SR is holding thousands of files open.

[Chris Solgat]

Here is a screen shot of the JVM argument and the system property.

JVMArgs.png

PluginScriptRoots.png

[JamieA]

That looks correct. So, I'm at a loss. But rather than testing with a full reindex, can you add one file to your <jira_home>/scripts directory. Call it test.groovy, containing just a single number, like 7. Then go to script console, and in the file tab enter: "test.groovy" (without quotes). If that works try copying your STR... script in there with a different name and running it in the same way. We would expect an exception, but I want to see if we get the ResourceException.

[Chris Solgat]

Added test.groovy to the new <jira_home>/scripts directory. It did find it and returned 7. Also, since I last added the screen shots, I restarted the JVM and ran a re-index with Script Runner disabled (think I forgot to mention that re-indexes were failing as well). The re-index completed successfully and I re-enabled Script Runner. One of our testers mentioned that the scripted fields are working correctly so far. I kicked off a background re-index (with Script Runner enabled), so he could continue testing, and so far I have not seen an error yet. The culprit may have been the lack of a completed re-index. We will continue testing and update as needed.

[Chris Solgat]

I may have spoken too early. The errors started showing up around 26%. They have since stopped at 34%. Will try moving the script into the new location a little bit later.

[Chris Solgat]

There is a new error showing up at the 26% mark. 2015-03-19 09:00:09,140 JiraTaskExectionThread-1 WARN jiraadmin 497x1696x1 1cnoccb 10.9.96.140,10.8.172.87 /secure/admin/jira/IndexReIndex.jspa [onresolve.scriptrunner.runner.ScriptRunnerImpl] Add a script root for this path: /data/atlassian/jira/scriptrunner/RPD/RPDCommencementDateDisplay5.groovy 2015-03-19 09:00:09,236 JiraTaskExectionThread-1 ERROR jiraadmin 497x1696x1 1cnoccb 10.9.96.140,10.8.172.87 /secure/admin/jira/IndexReIndex.jspa [onresolve.scriptrunner.customfield.GroovyCustomField] ************************************************************************************* 2015-03-19 09:00:09,244 JiraTaskExectionThread-1 ERROR jiraadmin 497x1696x1 1cnoccb 10.9.96.140,10.8.172.87 /secure/admin/jira/IndexReIndex.jspa [onresolve.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: RPISSSA-7119, field: Commencement Date Display 5 java.io.FileNotFoundException: /data/atlassian/jira/scriptrunner/RPD/RPDCommencementDateDisplay5.groovy (Too many open files) at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.runFileAsScript(ScriptRunnerImpl.groovy:192) at com.onresolve.scriptrunner.runner.ScriptRunner$runFileAsScript$3.call(Unknown Source) at com.onresolve.scriptrunner.customfield.GroovyCustomField.getValueFromIssue(GroovyCustomField.groovy:144) This is the error that is showing up when doing a background re-index.

[Chris Solgat]

You are correct.  They are scripted fields, pointing to our script directory with absolute path names.  I changed STRDocumentsNeededforIssue.groovy as you recommended and started another reindex.  I inserted the stack below.

2015-03-17 15:42:38,049 IssueIndexer:thread-24 WARN req85518 941x112x1 tj7h6m 10.9.96.140,10.8.172.87 /secure/admin/jira/IndexReIndex.jspa [onresolve.scriptrunner.runner.ScriptRunnerImpl] RE
groovy.util.ResourceException: Cannot open URL: bundle://159.0:0/RPD/STRDocumentsNeededforIssue.groovy
	at groovy.util.GroovyScriptEngine.getResourceConnection(GroovyScriptEngine.java:399)
	at groovy.util.ResourceConnector$getResourceConnection.call(Unknown Source)
	at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.runScript(ScriptRunnerImpl.groovy:426)
	at com.onresolve.scriptrunner.runner.ScriptRunner$runScript$18.callCurrent(Unknown Source)
	at com.onresolve.scriptrunner.runner.ScriptRunnerImpl.runFileAsScript(ScriptRunnerImpl.groovy:183)
	at com.onresolve.scriptrunner.runner.ScriptRunner$runFileAsScript$4.call(Unknown Source)
	at com.onresolve.scriptrunner.customfield.GroovyCustomField.getValueFromIssue(GroovyCustomField.groovy:144)
	at com.atlassian.jira.issue.fields.CustomFieldImpl.getValue(CustomFieldImpl.java:437)
	at com.atlassian.jira.issue.index.indexers.impl.SortableTextCustomFieldIndexer.addDocumentFields(SortableTextCustomFieldIndexer.java:44)
	at com.atlassian.jira.issue.index.indexers.impl.SortableTextCustomFieldIndexer.addDocumentFieldsSearchable(SortableTextCustomFieldIndexer.java:33)
	at com.atlassian.jira.issue.index.indexers.impl.AbstractCustomFieldIndexer.addIndex(AbstractCustomFieldIndexer.java:46)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.add(DefaultIssueDocumentFactory.java:102)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory$Builder.addAll(DefaultIssueDocumentFactory.java:91)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:60)
	at com.atlassian.jira.issue.index.DefaultIssueDocumentFactory.apply(DefaultIssueDocumentFactory.java:37)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$DefaultDocumentCreationStrategy.get(DefaultIssueIndexer.java:664)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$IndexIssuesOperation.perform(DefaultIssueIndexer.java:502)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$4$1.get(DefaultIssueIndexer.java:337)
	at com.atlassian.jira.issue.index.DefaultIssueIndexer$4$1.get(DefaultIssueIndexer.java:333)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:9)
	at com.atlassian.jira.index.SimpleIndexingStrategy.get(SimpleIndexingStrategy.java:5)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:38)
	at com.atlassian.jira.index.MultiThreadedIndexingStrategy$1.call(MultiThreadedIndexingStrategy.java:35)
	at com.atlassian.jira.util.concurrent.BoundedExecutor$2.call(BoundedExecutor.java:85)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: groovy.util.ResourceException: Cannot open URL: file:/data/atlassian/runtime/rpd/scripts/RPD/STRDocumentsNeededforIssue.groovy
	... 28 more
Caused by: groovy.util.ResourceException: Cannot open URL: file:/data/atlassian/jira/scriptrunner/RPD/STRDocumentsNeededforIssue.groovy
	at groovy.util.GroovyScriptEngine.getResourceConnection(GroovyScriptEngine.java:397)
	... 27 more

[JamieA]

Can you go to Admin -> System Info, and check that there is a property plugin.script.roots with the correct value defined? To me it looks like that property is not set correctly.