PermGen problems with Scriptrunner

[Uwe Sauerbrei]

Hi,

it seems, we have a serious problem with PermGen using ScriptRunner (Jira 5.2.7). There are about 10 Scripted fields to calculate costs but only a view users. We used VisualVM to observe the tomcat server and with active Scriptrunner permgen increases by 12Mb/hour. Deactivating the plugin stops this. I have already checked search templates and indexer, which seems to be okay.

It would be helpful to get some additional advices since I guess, a lot of people are using scriptrunnner without problem.

Thanks, Uwe

[JamieA · Accepted solution]

UPDATE: As discovered on GRV-206 this was caused by a later version of groovy installed that the one that ships with script runner. Right now, there is an issue with groovy 2+, hence I am keeping it on 1.8.5 for the moment, until I work it through. If you have any doubts about your installation, in the script runner admin panel run this:

GroovySystem.getVersion()

and assert that you 1.8.5 (for recent script runner versions). If you see 2.x.x then you need to sort that out.

Radu's answer is interesting. But note that his code will only create a new class in permgen because he is creating a new classloader... if you used the same GroovyClassLoader as script runner plugin does, then the HelloWorld class will be replaced. The total number of classes won't increase.

When compiling a script, which is what the calculated field module does, the script will get a class name like Script24, where the run() method contains the text you have entered. If you modify this text, you will then get Script25 etc. But you do not get a new class unless you change the code.

Of course this is a concern to me, but previously I have tested modifying the script for workflow functions hundreds of times, and what I observe is that the classes in permgen are GC'd, as Radu says.

Possibly there is a bug, but in a way I doubt it, because I am using calculated fields on instances with tens of thousands of users and hundreds of thousands of issues, and I would expect to see the same problem.

You mentioned in your private email that this happened when pressing Preview when designing the calculated field... maybe different code is used here (quite likely, and there may well be a bug here - there is some hackery such that the preview works without actually changing the calculated field definition) , when the script is run in the context of an issue there should not be a problem.

I do recommend the following properties:

-XX:PermSize=384m -XX:ReservedCodeCacheSize=128m

You should size as appropriate according to physical memory available.

[JamieA]

I just checked this with jvisualvm. There is some problem with calculated fields in that a new class is compiled each time the field is calculated... (GRV-206). However what I see is that the classes are GC'd from perm gen, so I don't see how this could cause an OOM.

[Uwe Sauerbrei]

The difference for me is, there is no gc and finally I get OOM.

server: tomcat6

jdk: 1.6.0_34

[Radu Dumitriu]

Hey Jamie, I just tried to get some q's off your chest :)

[JamieA]

@Radu - I appreciate it!

Uwe: Hrm. Can you give me the jira version and script runner version, ideally attach logs to https://jamieechlin.atlassian.net/browse/GRV-206. Also please provide your calc'd field script.

Also JVM options if they're not in the log file.

[Uwe Sauerbrei]

I have already send you some screenshots and the script

Jira Version: 5.2.7

Plugin Version: 2.1. Beta-8

Which log do you mean?

Added screenshot with JVM options to GRV-206

[Andy Grace]

Hi

We also hit perm gen problem when I was adding script fields to production system yesterday.

For me also it seemed to happen on the preview..

These were my steps:

Create 4 scripted fields (no script behind them yet)

Admin - Script Fields and edit one of them - add the logic and click preview for am issue

It just timed out - tried again, same

Log files showed the Perm Gen out of memory exception

Confirm groovy version 1.8.5

Our settings at the time were:

-XX:MaxPermSize=256m -Xms256m -Xmx2048m -XX:ReservedCodeCacheSize=64m

(We set the ReservedCodeCacheSize after problems enabling the Script Runner plugin in that enviornment - per https://jamieechlin.atlassian.net/browse/GRV-56

Have increased XX:ReservedCodeCacheSize to 128 and will try again in our next windown of opportunity next week

On our test system I had no such problems with the preview or with the scripted fields themselves

It would be good to know what happens when a field is added - does the system run through every issue that the field would be added to (there will be 80K+) then and there or just when they are pulled up, accessed by filter? Will this have memory / performance impact as they are added?

In this instance, since I'd never added the script I guess it wouldn't do anything other than running the preview

[Uwe Sauerbrei]

Our problems with PermGen where caused by another groovy lib in the path.

[Fabian Meier]

I have asked a related questions here:

https://answers.atlassian.com/questions/145473/permgen-s-code-cache-memory-pool-constantly-at-98

I suspect a ScriptRunner listener to leak into the code cache pool. I do not get an OOM, but the code cache fills up and after a day stays constantly at 98%. This does not happen in the staging system so I expect it to be related to the user activity / listener instances.

Can anyone help me to find out what exactly is filling up my code cache?

[Radu Dumitriu]

I will answer instead of Jamie, but your question seems a little unclear.

1/ First, do you run into: OutOfMemoryError: PermGen Space error ? Why are you worried about ?

2/ Second, PermGen is used for static storage, classloaders, classes, bytecode, pointers to members etc. It is obvious that the usage of permgen depends on the script being run. If your script does crazy stuff or just call crazy stuff (a lot of static storage routines etc) this behaviour is just, well, normal.

Permgen simply (despite of it's name) should be gatbage collected at some point - well that depends on the flags of the JVM, actually - and you should not worry about it.

3/ Third - the real reason you should see this - is the dynamic nature of the language (in your case, I suppose, it's Groovy). Look below:

ClassLoader parent = getClass().getClassLoader();
GroovyClassLoader loader = new GroovyClassLoader(parent);
Class groovyClass = loader.parseClass(new File("src/test/groovy/script/HelloWorld.groovy"));

GroovyObject groovyObject = (GroovyObject) groovyClass.newInstance();
Object[] args = {};
groovyObject.invokeMethod("run", args);

Hello ! This is defining some class from a groovy script ! Remember from the point 2 above that classes definitions are stored into permgen ? If yes, this is your answer ! :)

[Uwe Sauerbrei]

Hi Radu, thanks for your answer. Here some more details:

1. Yes, our Jira instance runs out of memory if we activate script runner plugin

2. The script (groovy) adds two values together (template: number field, searcher: Number Searcher) and PermGen/loaded classes are not released (observed with VisualVM)

We have:

-XX:MaxPermSize=440m

If I edit the appropriate source fields for calculation frequently, PermGen grows continously, as well as the number of loaded classes. I suppose, the same classes are loaded more than once but not released. Are there some special settings for GC? Any ideas are welcome! :-)

[Radu Dumitriu]

On old Java you should enable permgen GC with -XX:+CMSPermGenSweepingEnabled. If there's a leak there, you will need to hunt it down ...
-XX:-TraceClassLoading -XX:-TraceClassUnloading (something like that)

[Uwe Sauerbrei]

I'll try to use some additional flags as proposed...