Issue #610: Internal exception on flush

Today I received the exception below on running the following code:

IExecutionGraph loadedgraph = em.find(ExecutionGraph.class, id);  
   if(loadedgraph == null) {
    em.persist(instance);
    em.flush();
    loadedgraph = instance;
   }

[ObjectDB 2.3.3_06] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_26 (on Windows 2003 5.2).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.InternalException.f(InternalException.java:236)
at com.objectdb.o.IPW.ar(IPW.java:119)
at com.objectdb.o.PGW.am(PGW.java:201)
at com.objectdb.o.UPT.F(UPT.java:254)
at com.objectdb.o.UPT.m(UPT.java:171)
at com.objectdb.o.TSK.k(TSK.java:184)
at com.objectdb.o.TSK.i(TSK.java:157)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.UPT.r(UPT.java:157)
at com.objectdb.o.PGT.q(PGT.java:109)
at com.objectdb.o.UPT.B(UPT.java:121)
at com.objectdb.o.URT.l(URT.java:171)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:87)
at com.objectdb.o.UTT.z(UTT.java:365)
at com.objectdb.o.UTT.l(UTT.java:203)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.MST.U9(MST.java:1233)
at com.objectdb.o.WRA.U9(WRA.java:355)
at com.objectdb.o.WSM.U9(WSM.java:146)
at com.objectdb.o.OBM.bL(OBM.java:945)
at com.objectdb.o.OBM.bI(OBM.java:818)
at com.objectdb.o.OBM.flush(OBM.java:739)
at rbccm.felix.objectdb.workflow.ObjectDbGraphContainer.putIfAbsent(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.processSpawn(Unknown Source)
at rbccm.digest2.execution.ExecutionGraph.splitWorkflow(Unknown Source)
at rbccm.digest2.execution.ExecutionGraph.spawnWorkflows(Unknown Source)
at rbccm.digest2.execution.ExecutionGraph.run(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.processGraph(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.processWorkflows(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.processWorkflow(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.workflowCompleteForSomeTasks(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.framework.service.ServiceRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)

Running the database (attached) through objectdb doctor showed broken references. I'm not sure of the exact sequence that has caused the problem as until now this part of the process has been running fine but I'll post a test after further investigation. If there is any info from the exception which could help with this it'd be appreciated.

and this exception came shortly after - not sure if its related though

[ObjectDB 2.3.3_06] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_26 (on Windows 2003 5.2).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: java.lang.ArrayIndexOutOfBoundsException: -122
java.lang.ArrayIndexOutOfBoundsException: -122
at com.objectdb.o.PGC.w(PGC.java:210)
at com.objectdb.o.SNP.B(SNP.java:366)
at com.objectdb.o.RTT.l(RTT.java:127)
at com.objectdb.o.RST.l(RST.java:24)
at com.objectdb.o.RTT.C(RTT.java:179)
at com.objectdb.o.RST.r(RST.java:110)
at com.objectdb.o.PGT.q(PGT.java:109)
at com.objectdb.o.RST.A(RST.java:93)
at com.objectdb.o.RTT.l(RTT.java:132)
at com.objectdb.o.RST.l(RST.java:24)
at com.objectdb.o.RTT.C(RTT.java:179)
at com.objectdb.o.RST.r(RST.java:110)
at com.objectdb.o.PGT.q(PGT.java:109)
at com.objectdb.o.RST.A(RST.java:93)
at com.objectdb.o.RTT.l(RTT.java:132)
at com.objectdb.o.RST.l(RST.java:24)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:87)
at com.objectdb.o.RTT.C(RTT.java:177)
at com.objectdb.o.RST.r(RST.java:110)
at com.objectdb.o.PGT.q(PGT.java:109)
at com.objectdb.o.RST.A(RST.java:93)
at com.objectdb.o.RTT.l(RTT.java:132)
at com.objectdb.o.RST.l(RST.java:24)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:87)
at com.objectdb.o.MST.aU(MST.java:598)
at com.objectdb.o.MST.U2(MST.java:541)
at com.objectdb.o.WRA.U2(WRA.java:260)
at com.objectdb.o.LDR.w(LDR.java:382)
at com.objectdb.o.LDR.v(LDR.java:293)
at com.objectdb.o.LDR.s(LDR.java:211)
at com.objectdb.o.OBC.aM(OBC.java:1056)
at com.objectdb.o.OBC.aK(OBC.java:969)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:551)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:474)
at rbccm.felix.objectdb.workflow.ObjectDbGraphContainer.get(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.processWorkflows(Unknown Source)
at rbccm.digest2.execution.Workflow2Runner.startWorkflow(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.framework.service.ServiceRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)

Both stack traces expose unexpected internal states of ObjectDB but unfortunately they do not provide much info. I also don't see a direct connection between these two exceptions.

Maybe both exceptions are caused by using ObjectDB under heavy multithreading. It seems that your application uses  multithreading much more extensively than other applications that use ObjectDB. Probably ObjectDB has still some multithreading issues that require fixes. Any test that you can post and will demonstrate such exceptions may be very helpful.

The database itself seems healthy (except the dangling references which are not related directly to the exceptions).

The stack trace at #2 reflects a bug that was reproduced and fixed using the test in issue #625.

It is possible that this bug caused also the exception at #1 so this issue will be marked as fixed.