New Index causing troubles

#1

Hello,

we introduced a new index upon a String field Order.typeName, added to package.jdo as

<class name="Order" persistence-capable-superclass="BDO">
   <field name="attributeMap" embedded="true">
      <map embedded-key="true" embedded-value="true"/>
   </field>
   <field name="productOidToItemHistoryMap" embedded="true">
      <map embedded-key="true" embedded-value="true"/>
   </field>
   
   <extension vendor-name="objectdb" key="index" value="storeOid"/>
   <extension vendor-name="objectdb" key="index" value="orderDateEpochDay"/>
   <extension vendor-name="objectdb" key="index" value="typeName"/>
   
</class>

After starting our app with the new index on one database we got the following exception:

2019-02-01 04:42:37,641 | ERROR | Exception caught in thread: ODB-IndexActivation 
[ObjectDB 2.7.6] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.8.0_201 (on Linux 4.4.0-141-generic).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
        at com.objectdb.o.InternalException.f(InternalException.java:235)
        at com.objectdb.o.SPW.ad(SPW.java:174)
        at com.objectdb.o.PGW.ak(PGW.java:212)
        at com.objectdb.o.UPT.s(UPT.java:135)
        at com.objectdb.o.UPT.g(UPT.java:111)
        at com.objectdb.o.TSK.b(TSK.java:146)
        at com.objectdb.o.TSK.c(TSK.java:96)
        at com.objectdb.o.UPT.l(UPT.java:158)
        at com.objectdb.o.PGT.j(PGT.java:110)
        at com.objectdb.o.UPT.s(UPT.java:122)
        at com.objectdb.o.URT.g(URT.java:173)
        at com.objectdb.o.TSK.b(TSK.java:146)
        at com.objectdb.o.TSK.c(TSK.java:96)
        at com.objectdb.o.TSM.a(TSM.java:88)
        at com.objectdb.o.UTT.r(UTT.java:377)
        at com.objectdb.o.UTT.g(UTT.java:210)
        at com.objectdb.o.TSK.b(TSK.java:146)
        at com.objectdb.o.TSK.c(TSK.java:96)
        at com.objectdb.o.TSM.a(TSM.java:88)
        at com.objectdb.o.MST.Y4(MST.java:1421)
        at com.objectdb.o.IXA.a(IXA.java:312)
        at com.objectdb.o.IXA.run(IXA.java:266)
        at java.lang.Thread.run(Thread.java:748)

 

When starting it on another instance, the app got just hanging for hours, JConsole stating there was no deadlock, but most of the threads were waiting for ObjectDB. I am attaching threaddump of the threads related to the issue:

Name: Thread-6
State: WAITING on java.lang.Thread@7dc3b129
Total blocked: 0  Total waited: 1

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1252)
java.lang.Thread.join(Thread.java:1326)
com.objectdb.o.IXA.i(IXA.java:176)
com.objectdb.o.IXM.Yz(IXM.java:100)
com.objectdb.o.MST.X(MST.java:1194)
   - locked com.objectdb.o.MST@1881a6d5
com.objectdb.o.MST.Z2(MST.java:1181)
   - locked com.objectdb.o.MST@1881a6d5
com.objectdb.o.IXA$1.run(IXA.java:418)
   - locked com.objectdb.o.LKM@7b5315ca

======================================================================
Name: ODB-FileWriter
State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6
Total blocked: 111  Total waited: 75

Stack trace: 
com.objectdb.o.DFL.run(DFL.java:1009)
java.lang.Thread.run(Thread.java:748)

======================================================================
Name: ODB-WeakRefPurger
State: WAITING on java.lang.ref.ReferenceQueue$Lock@6a33f62e
Total blocked: 0  Total waited: 1

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
com.objectdb.o.RTM$1.run(RTM.java:157)

======================================================================
Name: ODB-IndexActivation
State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6
Total blocked: 15  Total waited: 2,171

Stack trace: 
com.objectdb.o.IXA.a(IXA.java:294)
com.objectdb.o.IXA.run(IXA.java:266)
java.lang.Thread.run(Thread.java:748)

======================================================================
Name: ODB-IndexActivation
State: BLOCKED on com.objectdb.o.LKM@7b5315ca owned by: Thread-6
Total blocked: 5  Total waited: 65

Stack trace: 
com.objectdb.o.IXA.a(IXA.java:294)
com.objectdb.o.IXA.run(IXA.java:266)
java.lang.Thread.run(Thread.java:748)

======================================================================
Name: pool-1-thread-7
State: BLOCKED on com.objectdb.o.MST@1881a6d5 owned by: Thread-6
Total blocked: 1  Total waited: 11

Stack trace: 
com.objectdb.o.DFL.Y8(DFL.java:796)
com.objectdb.o.MST.at(MST.java:470)
com.objectdb.o.MST.Zc(MST.java:446)
com.objectdb.o.WRA.Zc(WRA.java:268)
com.objectdb.o.LDR.u(LDR.java:558)
com.objectdb.o.LDR.Yr(LDR.java:475)
com.objectdb.o.LDR.n(LDR.java:274)
com.objectdb.o.OBC.ab(OBC.java:1085)
com.objectdb.o.OBC.ai(OBC.java:1003)
com.objectdb.o.OBC.q(OBC.java:698)
com.objectdb.jdo.PMImpl.getObjectById(PMImpl.java:457)
   - locked com.objectdb.jdo.PMImpl@39ce27f2
eu.extech.serverImpl.jdo.JDOConnection.objectById(Unknown Source)
eu.extech.serverImpl.QuantServerImpl.retrievePlanogramStorage(Unknown Source)
eu.extech.serverImpl.MiddlewareQServer.getPlanogramStorage(Unknown Source)
eu.extech.quant.data.planogram_lifecycle.RackPlanogram.getPlanogram(Unknown Source)
eu.extech.serverImpl.QuantServerImpl$13.run(Unknown Source)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

Please advice how to solve this issues. Last time it happened to us few monthes earlier and back then we could not get the new index working at all.

#2

Thank you for this report.

Could you please check if starting your application with no activity, until the new index is built (and only then resuming normal activity) solves this issue? You should see a message in the log file "Index activation batch completed..." when the new index building is done. If it doesn't help, could you please send the new thread dump?

ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)
#3

I implemented the workaround from here 

https://www.objectdb.com/forum/1860

so my app is opening the DB, but does not do any operation until it checks if there is the IndexActiovation thread and joins it. It seems it helped.

Thank You!

#4

Thank you for the update and for posting the link to the workaround here.

ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)

Post Reply

To post a reply and/or subscribe to update notifications - please login