How can I find bottlenecks in Tempo performance?

[Nathan Funk]

Viewing timesheets or reports in Tempo with a month or more time period for a project can take over 1/2minute in many cases. Each time we change the level of detail in a report, it takes just as long to regenerate the report. So it is very annoying to work with.

I see very high CPU usage (over 70% of a single core and occasionally multi-core use) of the java process on our JIRA VM while the report is being generated.

We are using Jira 6.1 and Tempo 2.1 in a VMWare virtual machine with a MySQL database on a Windows Server. Both are VMs on a dual Xeon server. Storage of the VMs is on a SAN. I would gladly provide additional information - I'm just not sure what's all relevant for performance issues.

[Nathan Funk · Accepted solution]

After posting this question on the Tempo support system they confirmed that performance for larger time periods can be poor so my experience isn't unusual.

[Eric Almquist]

Nathan,

Did you have a ticket opened that I could vote on regarding the poor performance? Unfortunately, we are experiencing the same issue since we are also a decent-sized Enterprise environment.

[Nathan Funk]

https://tempoplugin.jira.com/browse/JTS-1052

[Eric Almquist]

Thank you. I'll pass the word.

[Marc Minten _EVS_]

Hi,

We have the same huge performance issues with Tempo. It is NOT about reporting, but just filling in a user timesheet. We experienced somewhat, and it seems perfomance is about linear with the number of issues in the timesheet rows, and the number of dates in the columns (a timesheet over the current week goes nearly 4 times faster as a timesheet over the current month).

It becomes particular unworkable if there are more items (lines) in the timesheet than fit on the screen : scrolling is aweful (read:unusable).

We investigated server performance, there are no problems with memory and/or cpu, but the implementation seems very badly designed...

We are using JIRA 6.3.12 and Tempo 7.9.2

Did someone already opened a ticket for this (so we can add comments ansd votes ) ?

[Tim Kelley]

It's painful and a major annoyance at the workplace. Timesheet entry should be simple and fast or it is useless.

[Roman Mlodawski]

Not an Answer but we also have the identical problem on any long running issues (the ones we use for general tasks like "general admin"). Tried using the worklog calendar to enter times and also entering a worklog in the issues page. All have the same non-responsive result. The Devs traced it back to a REST call taking a long time (>10s) after you attempt to submit the logged time. I presume there is some data read going on for all the worklogs ever logged so as time goes by this becomes a problem. Worst, if you add a log entry via the Worklog Calendar it appears to work fast however if you don't wait until the previous entry is completed adding another one overwrites the "uncommitted task". In the "User Timesheet" view the app prevents you moving off the just entered "Log Work" popup page until the commit has happended. Either way you have to wait a long time. Like Gabrielle the workaround of closing and creating a new task for logging the time is not an acceptable option.

[Eric Almquist]

Our overseas developers have taken to using an Excel Macro to log their time each day since it's faster and they don't have to wait on it. I know the answer is a "federated" JIRA environment, but we have 24x7 project work that can be linked and networked to other applications on-prem, so we are stuck with one JIRA/Tempo environment and the performance penalties that brings.

[Bjarni Thorbjornsson]

Hi all,

We at Tempo are constantly working on performance improvements. We did some significant changes in both Tempo Timesheets 7.11 and in Timesheets 7.12. Can you please upgrade to Tempo 7.12 and tell us how that goes.

Also note that we are working with Atlassian on an issue regarding "large issues" (long running issues with a large issue history). Please watch this JIRA issue https://jira.atlassian.com/browse/JRA-40161 for the status.

Best regards,

-Bjarni (Tempo)

[Nathan Funk]

Thanks for the idea! I tried and found there was a `No Expression` text in the field. I removed that, saved, but the performance did not change. Is it normal that Tempo would issue thousands of SQL queries for a time frame of several months? We have over 14000 worklog entries after about 1 year of usage. So over 1000/month. Is that unusual?

[Bjarni Thorbjornsson]

Hi again,

1000 worklogs pr. month is not unusual. Which version of JIRA and Tempo are you using? We did some preformance improvements in Tempo 7.8.3 so you should upgrade ASAP.

It sounds like you should open a support ticket in support system: https://tempoplugin.jira.com/browse/JTS

Best regards,

-Bjarni

[Nathan Funk]

We are already at Tempo 7.8.3. Jira version is v6.1#6144. The 2.1 version I had posted originally was actually the Tempo Teams version we were using. We have since upgraded to the latest.

[Bjarni Thorbjornsson]

Hi Nathan,

Have you defined a "validation expression" like we describe here: https://tempoplugin.jira.com/wiki/display/TEMPO/Configuring+Fields+and+Properties#ConfiguringFieldsandProperties-Expression

Can you try to remove the expression if you have one defined and check if that makes any difference?

Best regards,

-Bjarni (Tempo support)

[Mehmet Kazgan]

You wrote "We are using Jira 6.1 and Tempo 2.1 in a VMWare virtual machine with a MySQL database on a Windows Server. " above. You meant MySQL is running on Windows?

I have a feeling your db ports have something to do with it. Interesting, because we are running Tempo reports and I have not experienced those type of performance issues.

There is a comment here saying that there was a performance issue before JIRA 6.1 and it is fixed

https://jira.atlassian.com/browse/JRA-31581

But you are already at 6.1. I would definitely check the ports for db.

[Nathan Funk]

MySQL is running in a Windows Server VM and the Jira server itself is in a Ubuntu Linux VM. The jira issue you linked to does sound interesting because we do have many people logging time on common issues for admin time, etc.

Not sure what you mean by "ports for db" though. Network ports for the MySQL database? Isn't there just a single port that the connections are established through?

I checked the connection pool settings and we have 20 connections and it's not establishing more during the report generation. So it doesn't feel like that's the problem.

[Mehmet Kazgan]

Hi Nathan,

For starters, moving JIRA to a Linux OS will double the performance. I can assure you from 10 years of usage experience in both Windows and Linux environments.

Tempo is a JIRA plugin which interacts with JIRA and database a lot. So, you can start checking your Database monitoring Administration and see if there are any issues there.

Try to index JIRA everyday. We do that by running a daily script at night, that also should definitely help.

From Tempo side of things, I have not seen any issues- but I strongly suggest you keep the version up-to-date.

Hope this helps,

Mehmet

[Nathan Funk]

Jira is already running in a Linux VM. I just started monitoring our MySQL DB while tempo runs a big timesheet/report. It appears Tempo is very inefficient in its DB queries. I see over 1500 queries per second for about the first half of the report generation (so over 15 seconds).