@Krzysztof: For how long was your Kunagi instance running?

It would be very nice, if you could send me the logs (wi@koczewski.de). Otherwise I have no idea how to analyse the problem.

Problem occured again few minutes ago. I did restart hour ago tomcat and problem occured again. In logs is silent - I am still logged in and can work - but catalina.out does not grow up. I will send logs in few minutes. Thanks for fast response.

Thank you for this info. Only thing I can recommend for now, ist to increase memory (heap and permgen) for your JVM.

@Krzysztof: Did you restart Kunagi by restarting Tomcat? Or did you just a "redeploy"?

Please do always a Tomcat restart, it seams we have some issues with threads, which do not stop correctly.

Even more, I had to kill -9 Tomcat process, as shutdown did not work on Tomcat running Kunagi. Before current startup I set up JVM parameters Xmx to 3GB and permgen to 256MB. We will observe situation next week, as now more people (about 15) will be using this instance. I will set up suggested Tomcat restart in cron next week.

I have question about how much is Kunagi storing in User Session context? Does it store all opened entities, what about displayed entities lists? I'm wondering how many people would be able to work on one Kunagi instance in parallel, what are boundaries here. When we should think about installing second instance... Currently on this instance works in parallel about 15 people and this number is to be increased...

Yesterday and today - the same problem, Everynight restart was configured - did not helped. Please advice what can we do.

Should we install more kunagi instances? Question is how many parallel users and active projects one Kunagi instance can handle?

Tomcat shutdown does not work again (had to kill-9) - lots (>100) of following entries at log.

Nov 23, 2011 2:12:37 PM org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
Nov 23, 2011 2:12:38 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/kunagi] appears to have started a thread named [ilarkesto.core.logging.Log] but has failed to stop it. This is very likely to create a memory leak.
Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/kunagi] appears to have started a thread named [app:kunagi] but has failed to stop it. This is very likely to create a memory leak.
Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/kunagi] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the u
nloadDelay attribute of the standard Context implementation.
Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/kunagi] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the u
nloadDelay attribute of the standard Context implementation.


and on the end of file lots of:

Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/kunagi] appears to have started a thread named [<no context>] but has failed to stop it. This is very likely to create a memory leak.
Nov 23, 2011 2:12:41 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/kunagi] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@18fdd2b]) and a value of type [ilarkesto.di.Context] (value [app:kunagi]) bu
t failed to remove it when the web application was stopped. This is very likely to create a memory leak.


I think this is a load problem. Fewer users/projects per instance should be a workaround. Just copy the complete content of kunagi-data into a new instance. Then delete half of the projects in each instance.

If you could email (wi@koczewski.de) me your kunagi-data directory, I could try to reproduce the problem on my own computer. Otherwise please give us some infos, how much data you have on your instance. How many users and how many projects. And how big is your kunagi-data/entities/ directory.

Did you already upgrade to 0.22?

I have noticed that Kunagi took 50% processor time in period that was "hanged".
Next time it happens I will send SIGQUIT to process running Kunagi and send you process stack. Maybe it can help in resolving issue.

It looks quite small.
Our kunagi-data directory:
11M core
12M entities
140M entities-rescue
14M files

Active daily users: ~20 from 35 registered
Projects: daily active 3 from 9 created

We did not upgrade, but I will do it today.

Our own instance runs with more entities without problems. Based on your info about processor usage, I would say it is a problem with your data constilation. Perhaps some infinite loop running.

Did you delete users or projects? Or removed users from projects?

Yes, as I can remember all of mentioned operations were performed in this Kunagi instance. How can I check if data consistency is preserved? Are there any way to clean up data?

Kunagi "repairs" itself automatically on startup. This functionality sometimes caused infinite loops in the past.

We have just published a bugfix release which contains a new administrators page. It shows active tasks, threads, sessions, memory usage, etc.

Hope it helps with identifying the cause of the problem.

The new page is accesible when logged in as system administrator from the header bar or directly from /kunagi/admin.html

I've just updated instance to 22.2 and will observe these statistics in daily manner.
Problem is that in case of unavailability - probably I will not be able to login as admin...
So far no more hangups from 23.11

I will let you know in case I will have any updates.

The login is a plain Java Servlet. The new admin/monitoring page is a simple Servlet too. After login Kunagi redirects to the GWT application. If this fails, you can try to call /kunagi/admin.html directly.

Just for update. We did not have any more problems like described above. We perform restart every night and everything is working ok.