Issue #1407: stress test -> com.objectdb.o.InternalException: null

I am running a stress test with 2 users, each making a request to the JSON API (which uses as storage ObjectDB) each 50 milliseconds. An API call might have between 1-4 db calls, but they are relatively light, and the db has just a few MB at the moment.

After a few minutes I got this exception. If I decrease the load, it doesn't happen. If I increase it, I get it very often.

Any solution/fix? A sample project is very hard, if not impossible, to create, because it uses all the system.

May 20, 2014 9:56:03 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [rest_29133222] in context with path [] threw exception [Error processing webservice request] with root cause
[ObjectDB 2.5.5] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) Client VM 1.8.0_05 (on Windows 8.1 6.3).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.TYM.an(TYM.java:724)
at com.objectdb.o.LDR.U1(LDR.java:994)
at com.objectdb.o.CST.ab(CST.java:491)
at com.objectdb.o.CST.U8(CST.java:408)
at com.objectdb.o.LDR.F(LDR.java:536)
at com.objectdb.o.LDR.E(LDR.java:470)
at com.objectdb.o.LDR.z(LDR.java:269)
at com.objectdb.o.OBC.aP(OBC.java:1058)
at com.objectdb.o.OBC.aN(OBC.java:976)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:616)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:536)
at org.apache.openejb.persistence.JtaEntityManager.find(JtaEntityManager.java:168)
at com.fitbird.control.p2.WorkoutPlanC.find(WorkoutPlanC.java:56)
at sun.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:180)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:80)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:212)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:181)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:268)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:263)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
at com.fitbird.control.p2.WorkoutPlanC$$LocalBeanProxy.find(com/fitbird/control/p2/WorkoutPlanC.java)
at com.fitbird.boundary.api.util.PlanAccessVerifier.<init>(PlanAccessVerifier.java:40)
at com.fitbird.boundary.api.PlanApi.getPlan(PlanApi.java:37)
at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.intercept.AbstractInvocationContext.proceed(AbstractInvocationContext.java:110)
at org.apache.webbeans.intercept.InterceptorInvocationContext.realProceed(InterceptorInvocationContext.java:72)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:66)
at com.fitbird.boundary.api.util.LogPerformanceInterceptor.logMethodEntry(LogPerformanceInterceptor.java:22)
at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.component.InterceptorBean.intercept(InterceptorBean.java:151)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:62)
at org.apache.webbeans.intercept.DefaultInterceptorHandler.invoke(DefaultInterceptorHandler.java:138)
at com.fitbird.boundary.api.PlanApi$$OwbInterceptProxy0.getPlan(com/fitbird/boundary/api/PlanApi.java)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89)
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:66)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.onMessage(CxfRsHttpListener.java:170)
at org.apache.openejb.server.rest.RsServlet.service(RsServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.fitbird.boundary.security.StatelessSecurityFilter.doFilter(StatelessSecurityFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

This is the second stack trace that I get, almost always and continuously if I increase the load:

May 20, 2014 10:13:54 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [rest_29133222] in context with path [] threw exception [Error processing webservice request] with root cause
com.objectdb.o.UserException: Mismatch client-server protocol prefix
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.SCO.b(SCO.java:210)
at com.objectdb.o.NTS.z(NTS.java:287)
at com.objectdb.o.CLS.S(CLS.java:349)
at com.objectdb.o.CLS.Q(CLS.java:282)
at com.objectdb.o.CST.U8(CST.java:407)
at com.objectdb.o.LDR.F(LDR.java:536)
at com.objectdb.o.LDR.E(LDR.java:470)
at com.objectdb.o.LDR.z(LDR.java:269)
at com.objectdb.o.OBC.aP(OBC.java:1058)
at com.objectdb.o.OBC.aN(OBC.java:976)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:616)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:536)
at org.apache.openejb.persistence.JtaEntityManager.find(JtaEntityManager.java:168)
at com.fitbird.control.p2.WorkoutPlanC.find(WorkoutPlanC.java:56)
at sun.reflect.GeneratedMethodAccessor194.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:180)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor110.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:80)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:212)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:181)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:268)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:263)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
at com.fitbird.control.p2.WorkoutPlanC$$LocalBeanProxy.find(com/fitbird/control/p2/WorkoutPlanC.java)
at com.fitbird.boundary.api.util.PlanAccessVerifier.<init>(PlanAccessVerifier.java:40)
at com.fitbird.boundary.api.PlanApi.getPlan(PlanApi.java:37)
at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.intercept.AbstractInvocationContext.proceed(AbstractInvocationContext.java:110)
at org.apache.webbeans.intercept.InterceptorInvocationContext.realProceed(InterceptorInvocationContext.java:72)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:66)
at com.fitbird.boundary.api.util.LogPerformanceInterceptor.logMethodEntry(LogPerformanceInterceptor.java:22)
at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.component.InterceptorBean.intercept(InterceptorBean.java:151)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:62)
at org.apache.webbeans.intercept.DefaultInterceptorHandler.invoke(DefaultInterceptorHandler.java:138)
at com.fitbird.boundary.api.PlanApi$$OwbInterceptProxy0.getPlan(com/fitbird/boundary/api/PlanApi.java)
at sun.reflect.GeneratedMethodAccessor141.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89)
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:66)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.onMessage(CxfRsHttpListener.java:170)
at org.apache.openejb.server.rest.RsServlet.service(RsServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.fitbird.boundary.security.StatelessSecurityFilter.doFilter(StatelessSecurityFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Another information:

I'm using Apache Tomee as app server. The moment the errors mentioned above start to happen, after a short while, I have to restart Tomee, because none of the DB queries work after those errors. It seems that all entity managers become corrupted / broken. (see stack trace below)

Should it be something in the connector? (the code that connects to ObjectDB - (which is still part of your code))

com.objectdb.o.UserException: Mismatch client-server protocol prefix
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.SCO.b(SCO.java:210)
at com.objectdb.o.NTS.z(NTS.java:287)
at com.objectdb.o.CLS.S(CLS.java:349)
at com.objectdb.o.CLS.Q(CLS.java:282)
at com.objectdb.o.CST.U8(CST.java:407)
at com.objectdb.o.LDR.F(LDR.java:536)
at com.objectdb.o.LDR.E(LDR.java:470)
at com.objectdb.o.LDR.z(LDR.java:269)
at com.objectdb.o.OBC.aP(OBC.java:1058)
at com.objectdb.o.OBC.aN(OBC.java:976)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:616)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:536)
at org.apache.openejb.persistence.JtaEntityManager.find(JtaEntityManager.java:168)
at com.fitbird.control.p2.WorkoutPlanC.find(WorkoutPlanC.java:56)
at sun.reflect.GeneratedMethodAccessor125.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:180)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor111.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:80)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:212)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:181)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:268)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:263)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
at com.fitbird.control.p2.WorkoutPlanC$$LocalBeanProxy.find(com/fitbird/control/p2/WorkoutPlanC.java)
at com.fitbird.boundary.api.util.PlanAccessVerifier.<init>(PlanAccessVerifier.java:40)
at com.fitbird.boundary.api.PlanApi.getPlan(PlanApi.java:37)
at sun.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.intercept.AbstractInvocationContext.proceed(AbstractInvocationContext.java:110)
at org.apache.webbeans.intercept.InterceptorInvocationContext.realProceed(InterceptorInvocationContext.java:72)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:66)
at com.fitbird.boundary.api.util.LogPerformanceInterceptor.logMethodEntry(LogPerformanceInterceptor.java:22)
at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.webbeans.component.InterceptorBean.intercept(InterceptorBean.java:151)
at org.apache.webbeans.intercept.InterceptorInvocationContext.proceed(InterceptorInvocationContext.java:62)
at org.apache.webbeans.intercept.DefaultInterceptorHandler.invoke(DefaultInterceptorHandler.java:138)
at com.fitbird.boundary.api.PlanApi$$OwbInterceptProxy0.getPlan(com/fitbird/boundary/api/PlanApi.java)
at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89)
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:66)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.onMessage(CxfRsHttpListener.java:170)
at org.apache.openejb.server.rest.RsServlet.service(RsServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.fitbird.boundary.security.StatelessSecurityFilter.doFilter(StatelessSecurityFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

The first exception seems to indicate a problem in live multithreaded type registration. Unfortunately this is an issue that we are aware to but couldn't reproduce and solve yet.

It was reported at least once previously, and a workaround that seems to work is to separate building the schema (i.e. registration of all the managed types) from the load test, i.e. start the multithreaded load test with a database that already has all the types. Types can be registered by persisting instances to the database. I think that invoking Metamodel methods (e.g. entity) can register types without persisting instances.

The "Mismatch client-server protocol prefix" is another issue that was reported in the past rarely and we don't know to reproduce and solve it yet. If it is related to the first issue, maybe the workaround above could solve both of them. If not, we will need a sample project (even large and with no source code) in order to reproduce and fix it.

Ok. I will help with a sample then, because it's a very important issue.

I will come up with a sample that reproduces both errors in 1-2 weeks.

That will be very helpful. Thanks.

I attached 3 new errors found during load testing of the same project.

Besides those 3 new ones, the previous ones keep appearing as well, especially: 

com.objectdb.o.UserException: Mismatch client-server protocol prefix.

All these exceptions are related to breaking the synchronization between the client and the server.

It would be interesting to check the same test, if possible, in embedded mode. This could narrow the problem.

But eventually a test case that demonstrates the exception will be needed.

Interesting. I have to mention that now there is only one client using the db server, running on Apache Tomee, and there is no redeployment or schema modification happening. Tomee is started (with the already deployed app), then the load test begins.

It seems to be related on what load I put on the server through jMeter. If the load is relatively low, the errors start to appear after some minutes or half an hour, or even an hour, depending on the load.

Another point is that ObjectDB doesn't pass 5% CPU utilization during the load on an Intel i3 2nd generation on my personal machine, so the load is very low.

In the same time, Tomee stays around 10-20% CPU utilization during the load.

I'll do the embedded experiment and create a sample next weekend, hopefully.

PS. I'm still answering from IE 11.

Hi! First of all, the website is pretty fast now. It's a pleasure to use it.

I'm working on a sample right now, but I need a way to transfer it to you without making it public, because it's confidential.

So, let me know how I can send the sample to you, in such a way that no one else will be able to see it.

One more error, 523

23-May-2014 22:44:24.296 SEVERE [http-bio-8080-exec-22] org.apache.openejb.core.transaction.EjbTransactionUtil.handleSystemException EjbTransactionUtil.handleSystemException: Mismatch client-server protocol prefix
[ObjectDB 2.5.5] javax.persistence.PersistenceException
Mismatch client-server protocol prefix (error 523)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:621)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:536)
at org.apache.openejb.persistence.JtaEntityManager.find(JtaEntityManager.java:168)
at com.fitbird.control.p1.UserC.findEmailLogin(UserC.java:100)
at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:180)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:99)
at sun.reflect.GeneratedMethodAccessor96.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:182)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:164)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:80)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:212)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:181)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:268)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:263)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:86)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:303)
at com.fitbird.control.p1.UserC$$LocalBeanProxy.findEmailLogin(com/fitbird/control/p1/UserC.java)
at com.fitbird.boundary.api.LoginApi.loginEmail(LoginApi.java:92)
at sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89)
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:66)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:262)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:237)
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.onMessage(CxfRsHttpListener.java:170)
at org.apache.openejb.server.rest.RsServlet.service(RsServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at com.fitbird.boundary.security.StatelessSecurityFilter.doFilter(StatelessSecurityFilter.java:58)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.objectdb.o.UserException: Mismatch client-server protocol prefix
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.SCO.b(SCO.java:210)
at com.objectdb.o.NTS.z(NTS.java:287)
at com.objectdb.o.CLS.S(CLS.java:349)
at com.objectdb.o.CLS.Q(CLS.java:282)
at com.objectdb.o.CST.U8(CST.java:407)
at com.objectdb.o.LDR.F(LDR.java:536)
at com.objectdb.o.LDR.E(LDR.java:470)
at com.objectdb.o.LDR.z(LDR.java:269)
at com.objectdb.o.OBC.aP(OBC.java:1058)
at com.objectdb.o.OBC.aN(OBC.java:976)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:616)
... 63 more

Good news! I've tried in embedded mode. None of the errors seem to appear here, even with 3 API requests / 1 millisecond, while in client-server mode it doesn't even resist to 2 req / 100 milliseconds.

When you tell me how to send you the sample in a private mode, you'll get it. It's ready.

Please mention which operating system you're going to use for testing, to know how to set the sample.

Good/bad news! It seems that Tomee runs out of heap memory when ObjectDB is used in embedded mode.

Both Tomee and ObjectDB in server mode seem to be perfect in terms of memory usage, but using ODB as embedded changes the situation.

With a heap of 100 MB, Tomee runs out of memory in ~20 minutes. With a heap of 512 MB, in around 240 min (4 hours). Both cases were tested with 9 API requests / millisecond from custom samplers in JMeter.

When ObjectDB is used in server mode, Tomee's average heap memory usage is 50MB, regardless of how long is load tested.

I forgot to take a screenshot of the 20 minutes experiment in jVisualVM, but I took for the 4 hours one: It's split in part 1 and two, to see how the memory usage goes.

Running on Oracle JDK 8u5 x32.

I suspect ObjectDB is the cause for this memory leak, because nothing was changed in the application nor in Tomee, excepting that persistence.xml was set to use ObjectDB in embedded mode.

Thank you for the updates.

Could you please send the test application in a new support ticket?

Error found in embedded mode today while running under the same load test. I was runnning this test on Tomee 1.7.0 JAX-RS snapshot from 2014.05.26, but I don't think it has anything to do with Tomee.

[ObjectDB 2.5.5] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) Client VM 1.8.0_05 (on Windows 8.1 6.3).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: Negative snapshot user count
com.objectdb.o.InternalException: Negative snapshot user count
at com.objectdb.o.SNP.D(SNP.java:344)
at com.objectdb.o.SFL.aa(SFL.java:802)
at com.objectdb.o.MST.Vf(MST.java:1775)
at com.objectdb.o.PRG.au(PRG.java:1168)
at com.objectdb.o.PRG.ag(PRG.java:718)
at com.objectdb.o.PRG.af(PRG.java:553)
at com.objectdb.o.QRM.Vb(QRM.java:285)
at com.objectdb.o.MST.Vb(MST.java:949)
at com.objectdb.o.WRA.Vb(WRA.java:311)
at com.objectdb.o.WSM.Vb(WSM.java:115)
at com.objectdb.o.QRR.g(QRR.java:244)
at com.objectdb.o.QRR.f(QRR.java:153)
at com.objectdb.jpa.JpaQuery.executeUpdate(JpaQuery.java:775)
at com.fitbird.control.p2.WorkoutPlanC.finishPlan(WorkoutPlanC.java:69)
at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:191)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:181)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:100)
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:191)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:233)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:198)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.synchronizedBusinessMethod(EjbObjectProxyHandler.java:308)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler.businessMethod(EjbObjectProxyHandler.java:303)
at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(EjbObjectProxyHandler.java:92)
at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(BaseEjbProxyHandler.java:308)
at com.fitbird.control.p2.WorkoutPlanC$$LocalBeanProxy.finishPlan(com/fitbird/control/p2/WorkoutPlanC.java)
at com.fitbird.boundary.api.WorkoutApi.end(WorkoutApi.java:300)
at sun.reflect.GeneratedMethodAccessor226.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:191)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
at com.fitbird.boundary.api.util.LogPerformanceInterceptor.logMethodEntry(LogPerformanceInterceptor.java:22)
at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:191)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
at org.apache.openejb.monitoring.StatsInterceptor.record(StatsInterceptor.java:181)
at org.apache.openejb.monitoring.StatsInterceptor.invoke(StatsInterceptor.java:100)
at sun.reflect.GeneratedMethodAccessor70.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext$Invocation.invoke(ReflectionInvocationContext.java:191)
at org.apache.openejb.core.interceptor.ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
at org.apache.openejb.core.interceptor.InterceptorStack.invoke(InterceptorStack.java:85)
at org.apache.openejb.core.stateless.StatelessContainer._invoke(StatelessContainer.java:233)
at org.apache.openejb.core.stateless.StatelessContainer.invoke(StatelessContainer.java:198)
at org.apache.openejb.util.proxy.ProxyEJB$Handler.invoke(ProxyEJB.java:74)
at com.fitbird.boundary.api.WorkoutApi$$LocalBeanProxy.end(com/fitbird/boundary/api/WorkoutApi.java)
at sun.reflect.GeneratedMethodAccessor225.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.openejb.server.cxf.rs.OpenEJBEJBInvoker.performInvocation(OpenEJBEJBInvoker.java:88)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165)
at org.apache.openejb.server.cxf.rs.OpenEJBEJBInvoker.invoke(OpenEJBEJBInvoker.java:62)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:89)
at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(AutoJAXRSInvoker.java:64)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:240)
at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.onMessage(CxfRsHttpListener.java:181)
at org.apache.openejb.server.rest.RsServlet.service(RsServlet.java:53)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.fitbird.boundary.security.StatelessSecurityFilter.doFilter(StatelessSecurityFilter.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:44)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1720)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

The client-server exceptions (posts #1 - #11 above) have been fixed in build 2.5.5_11.

The cause was unexpected order of invocation of afterCompletion in TomcatEE by Geronimo:

• Firstly to OpenEJB that closed the EntityManager and moved its connection to the pool.
• Then to ObjectDB (on the just closed EntityManager) that had to commit changes using the connection (which has already been moved to the pool and could be allocated to other operations).

The fix includes delaying the release of the connection to the pool and additional check to verify that connection that was released to the pool will not be used anymore by its previous owner.

The "Negative snapshot user count" error and a memory leak that its causes have not been fixed yet.

There is a good progress with the "Negative snapshot user count" exception and the consequent memory leak.

The bug can be reproduced now by the following test case:

import javax.persistence.*;


public class T1407 {

    public static void main(String[] args) throws Exception {
       
        EntityManagerFactory emf =
            Persistence.createEntityManagerFactory(
                "objectdb:db/test.tmp;drop");
       
        EntityManager em1 = emf.createEntityManager();
        em1.getTransaction().begin();
        em1.persist(new TestEntity());
        em1.flush();
       
        EntityManager em2 = emf.createEntityManager();
        em2.getTransaction().begin();
        em2.persist(new TestEntity());
        em2.getTransaction().commit();
        em2.close();
       
        em1.createQuery("UPDATE TestEntity SET x = 1").executeUpdate();
        em1.getTransaction().commit();
        em1.close();

        for (int i = 0; i < 1000000; i++)
        {
            EntityManager em = emf.createEntityManager();
            em.getTransaction().begin();
            em.persist(new TestEntity());
            em.getTransaction().commit();
            em.close();
        }

        emf.close();
    }

    @Entity
    public static class TestEntity
    {
        private int x;
    }
}

It is caused as a result of running an UPDATE query on uncommitted flushed changes in some circumstances.

A fix will be released soon.

Build 2.5.5_12 should complete the repair of these bugs.

Thank you again for reporting and for the test program.

That sounds perfectly awesome.

Is the memory leak from embedded mode fixed as well?

You're welcome, and thank you as well for your support. I appreciate it a lot. That's one of the things that makes me using ObjectDB.

I'll try it out.

Yes, the memory leak is a result of the "Negative snapshot user count" exception (reducing the wrong user count, and consequently avoiding resource release).