Hello,
we introduced a new index upon a String field Order.typeName, added to package.jdo as
<class name="Order" persistence-capable-superclass="BDO"> <field name="attributeMap" embedded="true"> <map embedded-key="true" embedded-value="true"/> </field> <field name="productOidToItemHistoryMap" embedded="true"> <map embedded-key="true" embedded-value="true"/> </field> <extension vendor-name="objectdb" key="index" value="storeOid"/> <extension vendor-name="objectdb" key="index" value="orderDateEpochDay"/> <extension vendor-name="objectdb" key="index" value="typeName"/> </class>
After starting our app with the new index on one database we got the following exception:
2019-02-01 04:42:37,641 | ERROR | Exception caught in thread: ODB-IndexActivation [ObjectDB 2.7.6] Unexpected exception (Error 990) Generated by Java HotSpot(TM) 64-Bit Server VM 1.8.0_201 (on Linux 4.4.0-141-generic). Please report this error on http://www.objectdb.com/database/issue/new com.objectdb.o.InternalException: null com.objectdb.o.InternalException at com.objectdb.o.InternalException.f(InternalException.java:235) at com.objectdb.o.SPW.ad(SPW.java:174) at com.objectdb.o.PGW.ak(PGW.java:212) at com.objectdb.o.UPT.s(UPT.java:135) at com.objectdb.o.UPT.g(UPT.java:111) at com.objectdb.o.TSK.b(TSK.java:146) at com.objectdb.o.TSK.c(TSK.java:96) at com.objectdb.o.UPT.l(UPT.java:158) at com.objectdb.o.PGT.j(PGT.java:110) at com.objectdb.o.UPT.s(UPT.java:122) at com.objectdb.o.URT.g(URT.java:173) at com.objectdb.o.TSK.b(TSK.java:146) at com.objectdb.o.TSK.c(TSK.java:96) at com.objectdb.o.TSM.a(TSM.java:88) at com.objectdb.o.UTT.r(UTT.java:377) at com.objectdb.o.UTT.g(UTT.java:210) at com.objectdb.o.TSK.b(TSK.java:146) at com.objectdb.o.TSK.c(TSK.java:96) at com.objectdb.o.TSM.a(TSM.java:88) at com.objectdb.o.MST.Y4(MST.java:1421) at com.objectdb.o.IXA.a(IXA.java:312) at com.objectdb.o.IXA.run(IXA.java:266) at java.lang.Thread.run(Thread.java:748)
When starting it on another instance, the app got just hanging for hours, JConsole stating there was no deadlock, but most of the threads were waiting for ObjectDB. I am attaching threaddump of the threads related to the issue:
Name: Thread-6 State: WAITING on java.lang.Thread@7dc3b129 Total blocked: 0 Total waited: 1 Stack trace: java.lang.Object.wait(Native Method) java.lang.Thread.join(Thread.java:1252) java.lang.Thread.join(Thread.java:1326) com.objectdb.o.IXA.i(IXA.java:176) com.objectdb.o.IXM.Yz(IXM.java:100) com.objectdb.o.MST.X(MST.java:1194) - locked com.objectdb.o.MST@1881a6d5 com.objectdb.o.MST.Z2(MST.java:1181) - locked com.objectdb.o.MST@1881a6d5 com.objectdb.o.IXA$1.run(IXA.java:418) - locked com.objectdb.o.LKM@7b5315ca ====================================================================== Name: ODB-FileWriter State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6 Total blocked: 111 Total waited: 75 Stack trace: com.objectdb.o.DFL.run(DFL.java:1009) java.lang.Thread.run(Thread.java:748) ====================================================================== Name: ODB-WeakRefPurger State: WAITING on java.lang.ref.ReferenceQueue$Lock@6a33f62e Total blocked: 0 Total waited: 1 Stack trace: java.lang.Object.wait(Native Method) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) com.objectdb.o.RTM$1.run(RTM.java:157) ====================================================================== Name: ODB-IndexActivation State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6 Total blocked: 15 Total waited: 2,171 Stack trace: com.objectdb.o.IXA.a(IXA.java:294) com.objectdb.o.IXA.run(IXA.java:266) java.lang.Thread.run(Thread.java:748) ====================================================================== Name: ODB-IndexActivation State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6 Total blocked: 5 Total waited: 65 Stack trace: com.objectdb.o.IXA.a(IXA.java:294) com.objectdb.o.IXA.run(IXA.java:266) java.lang.Thread.run(Thread.java:748) ====================================================================== Name: pool-1-thread-7 State: BLOCKED on com.objectdb.o.MST@1881a6d5 owned by: Thread-6 Total blocked: 1 Total waited: 11 Stack trace: com.objectdb.o.DFL.Y8(DFL.java:796) com.objectdb.o.MST.at(MST.java:470) com.objectdb.o.MST.Zc(MST.java:446) com.objectdb.o.WRA.Zc(WRA.java:268) com.objectdb.o.LDR.u(LDR.java:558) com.objectdb.o.LDR.Yr(LDR.java:475) com.objectdb.o.LDR.n(LDR.java:274) com.objectdb.o.OBC.ab(OBC.java:1085) com.objectdb.o.OBC.ai(OBC.java:1003) com.objectdb.o.OBC.q(OBC.java:698) com.objectdb.jdo.PMImpl.getObjectById(PMImpl.java:457) - locked com.objectdb.jdo.PMImpl@39ce27f2 eu.extech.serverImpl.jdo.JDOConnection.objectById(Unknown Source) eu.extech.serverImpl.QuantServerImpl.retrievePlanogramStorage(Unknown Source) eu.extech.serverImpl.MiddlewareQServer.getPlanogramStorage(Unknown Source) eu.extech.quant.data.planogram_lifecycle.RackPlanogram.getPlanogram(Unknown Source) eu.extech.serverImpl.QuantServerImpl$13.run(Unknown Source) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)
Please advice how to solve this issues. Last time it happened to us few monthes earlier and back then we could not get the new index working at all.
