Issue #187: Querying error - java.lang.ClassCastException: com.objectdb.o.STV

I'm getting the ClassCastException below when querying objects. As yet I haven't been able to track down the exact circumstance in which it occurs - saving and querying objects works fine on attempts to recreate - but presumably at some point an object gets saved which objectdb doesn't like.

Any pointers as to what is wrong or approaches to investigate further would be much appreciated.

[odb:*:fatal] java.lang.ClassCastException: com.objectdb.o.STV
at com.objectdb.o.PBI.p(PBI.java:91)
at com.objectdb.o.OBI.U4(OBI.java:235)
at com.objectdb.o.BQI.Vd(BQI.java:134)
at com.objectdb.o.PRG.ac(PRG.java:679)
at com.objectdb.o.PRG.aa(PRG.java:609)
at com.objectdb.o.QRM.UR(QRM.java:256)
at com.objectdb.o.MST.UR(MST.java:856)
at com.objectdb.o.WRA.UR(WRA.java:285)
at com.objectdb.o.WSM.UR(WSM.java:112)
at com.objectdb.o.WRA.UR(WRA.java:285)
at com.objectdb.o.WSN.UR(WSN.java:416)
at com.objectdb.o.QRR.g(QRR.java:210)
at com.objectdb.o.QRR.b(QRR.java:138)
at com.objectdb.jpa.JpaQuery.getResultList(JpaQuery.java:566)
at rbccm.felix.gridservice.admin.dao.SimpleWorkflowInstanceDao.pagedQuery(Unknown Source)
at rbccm.felix.gridservice.admin.dao.SimpleWorkflowInstanceDao.query(Unknown Source)
at rbccm.felix.gridservice.admin.server.WorkflowAdminClientImpl.query(Unknown Source)
at rbccm.felix.gridservice.api._WorkflowAdminClientAPIDisp.___query(Unknown Source)
at rbccm.felix.gridservice.api._WorkflowAdminClientAPIDisp.__dispatch(Unknown Source)
at IceInternal.Incoming.invoke(Incoming.java:147)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2249)
at Ice.ConnectionI.message(ConnectionI.java:1362)
at IceInternal.ThreadPool.run(ThreadPool.java:782)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:1242)

Unfortunately the stack trace indicates an internal ObjectDB bug. This is the first report about this bug so any help in exploring it would be appreciated.

It seems related to handling small / large objects. ObjectDB splits large objects (> 2000 bytes) into pages. Somehow the stack trace indicates an unexpected state in which the same object is marked as small in one page and as large in another page.

Maybe this is the result of switching between the two states when the object is updated and become larger / smaller.

As a first step - could you please run the ObjectDB Doctor on the database and send the result report?

Could you reproduce this exception on a new database? If you can - we can try disabling some optimization, which I suspect might cause this issue, in order to isolate the problem.

Build 2.1.1_08 should generate an exception with additional details about the problem.

If you can post the generated output it might help.

In addition, a temporary property on the client side disables the suspected optimization:

System.setProperty("com.objectdb.disable.optimization.small", "true");

But it might help (if at all) only with a new database file.

The suggestion about handling of small/large objects is possible. When the object is first saved it contains a (potentially) large amount of information which allows a process to be recovered/replayed if necessary. There is then a "cleanup" which removes this information from the object after a set expiry time. Thus the object can go from being large to small.

The output from ObjectDB Doctor is below. I'll try your other suggestions later this morning.

ObjectDB Doctor [version 2.0.5_02]
Copyright (c) 2011, ObjectDB Software. All rights reserved.

Scanning the database file...
.. 5MB (total)

Analyzing database structure...
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Global Value Errors
-------------------
[1] Unexpected last index ID: 4622168676387389538 (expected -100)
[2] Unexpected last large value ID: 3631415547023958017 (expected 2403)
[3] Unexpected last type ID: 7221069219291554102 (expected 11)
[4] Unexpected total object count: 3691044274576634723 (expected 1378)
[5] Unexpected total page count: 8719319783396618851 (expected 2816)

Page Relation Errors
--------------------
[1] Page #474 first key is 'db80fd29770c41a892d8caf5b3685845'#0/3-0/4011, parent page #882 key is 'db898c01f7274d6c8447acb3457673ed'
[2] Page #2245 first key is '66b9ba37579d4956b0f6c3a7804abfb6'#0/15-0/28789, parent page #1800 key is '66c0f7d0f73043909fe79c4dcd09d7ee'
 

The page relation errors might indicate a deeper problem in the database file structure that could cause this exception.

Please upgrade to the last build (2.1.1_08) and either start with a clean database or fix this database by running the Doctor in repair mode (and then verify the fix by using the Doctor in diagnosis mode).

Check if your application can bring the database again to a state in which similar errors are reported by the Doctor, with and without setting of the property at #3.

Build 2.1.1_08 produces the exception below. I've tried repairing the database with Doctor but the new file produced seems to contain the same error.

I'll start again with a clean database file and see if the problem re-occurs.

[ObjectDB 2.1.1_08] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.5.0_21 (on Windows 2003 5.2).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: merge2 rbccm.felix.gridservice.admin.execution.SimpleWorkflowInstance:'04e0cc3e99a14edd97792c453d43cc95' => merger[2998]-missing:1
com.objectdb.o.InternalException: merge2 rbccm.felix.gridservice.admin.execution.SimpleWorkflowInstance:'04e0cc3e99a14edd97792c453d43cc95' => merger[2998]-missing:1
at com.objectdb.o.PBI.z(PBI.java:119)
at com.objectdb.o.PBI.p(PBI.java:93)
at com.objectdb.o.OBI.U5(OBI.java:235)
at com.objectdb.o.BQI.Ve(BQI.java:134)
at com.objectdb.o.PRG.ac(PRG.java:679)
at com.objectdb.o.PRG.aa(PRG.java:609)
at com.objectdb.o.QRM.UR(QRM.java:256)
at com.objectdb.o.MST.UR(MST.java:866)
at com.objectdb.o.WRA.UR(WRA.java:286)
at com.objectdb.o.WSM.UR(WSM.java:113)
at com.objectdb.o.WRA.UR(WRA.java:286)
at com.objectdb.o.WSN.UR(WSN.java:417)
at com.objectdb.o.QRR.g(QRR.java:216)
at com.objectdb.o.QRR.b(QRR.java:139)
at com.objectdb.jpa.JpaQuery.getResultList(JpaQuery.java:575)
at rbccm.felix.gridservice.admin.dao.SimpleWorkflowInstanceDao.pagedQuery(Unknown Source)
at rbccm.felix.gridservice.admin.dao.SimpleWorkflowInstanceDao.query(Unknown Source)
at rbccm.felix.gridservice.admin.server.WorkflowAdminClientImpl.query(Unknown Source)
at rbccm.felix.gridservice.api._WorkflowAdminClientAPIDisp.___query(Unknown Source)
at rbccm.felix.gridservice.api._WorkflowAdminClientAPIDisp.__dispatch(Unknown Source)
at IceInternal.Incoming.invoke(Incoming.java:147)
at Ice.ConnectionI.invokeAll(ConnectionI.java:2249)
at Ice.ConnectionI.message(ConnectionI.java:1362)
at IceInternal.ThreadPool.run(ThreadPool.java:782)
at IceInternal.ThreadPool.access$100(ThreadPool.java:12)
at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:1242)

Following your report I could reproduce the exception, I found the bug and fixed the problem.

Version 2.2.0 that was just released includes the fix.

This bug is now marked as critical since it can corrupt database files. However, at least in my tests - no data was lost and I could fix the corrupted database files by running the Doctor in repair mode (with 2 arguments):

> java -cp objectdb.jar com.objectdb.Doctor old.odb new.odb

Thank you very much for your helpful report.