Issue #414: InternalException - error reading field from "queue" database

I trying to use an objectdb database as a persitent queue but have started getting the following error on the receiver side:

2011-07-25 10:54:25,023 AdminServer-Callback-Instance-2/0 ERROR ServiceRunner - Error processing message.
rbccm.felix.framework.ApplicationException: Error taking message from ObjectDB queue
at rbccm.felix.objectdb.messaging.ObjectDbMessagePipe.take(Unknown Source)
at rbccm.felix.objectdb.messaging.ObjectDbMessagePipe.take(Unknown Source)
at rbccm.felix.framework.service.ServiceRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.RuntimeException: Error getting message from data
at rbccm.felix.objectdb.messaging.ObjectDbMessagePipe.getData(Unknown Source)
at rbccm.felix.objectdb.messaging.ObjectDbMessagePipe.getMessage(Unknown Source)
... 4 more
Caused by: com.objectdb.o._PersistenceException: Failed to read the value of field field rbccm.felix.objectdb.messaging.ObjectDbMessage._payload using reflection
at com.objectdb.o._PersistenceException.b(_PersistenceException.java:45)
at com.objectdb.o.JPE.g(JPE.java:140)
at com.objectdb.o.ERR.f(ERR.java:59)
at com.objectdb.o.OBC.onObjectDBError(OBC.java:1448)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:554)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:472)
... 6 more
Caused by: com.objectdb.o.UserException: Failed to read the value of field field rbccm.felix.objectdb.messaging.ObjectDbMessage._payload using reflection
at com.objectdb.o.MSG.d(MSG.java:74)
at com.objectdb.o.UMR.M(UMR.java:860)
at com.objectdb.o.UMR.z(UMR.java:575)
at com.objectdb.o.UML.v(UML.java:528)
at com.objectdb.o.MMM.ag(MMM.java:1042)
at com.objectdb.o.UTY.aH(UTY.java:1243)
at com.objectdb.o.UTY.aG(UTY.java:1215)
at com.objectdb.o.ENH.b(ENH.java:99)
at com.objectdb.o.LDR.I(LDR.java:437)
at com.objectdb.o.LDR.j(LDR.java:658)
at com.objectdb.o.CST.aa(CST.java:471)
at com.objectdb.o.CST.UP(CST.java:390)
at com.objectdb.o.LDR.H(LDR.java:348)
at com.objectdb.o.LDR.G(LDR.java:286)
at com.objectdb.o.LDR.D(LDR.java:204)
at com.objectdb.o.OBC.aK(OBC.java:1027)
at com.objectdb.o.OBC.aI(OBC.java:927)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:549)
... 7 more
Caused by: com.objectdb.o.InternalException
at com.objectdb.o.TYM.al(TYM.java:696)
at com.objectdb.o.TYR.aB(TYR.java:560)
at com.objectdb.o.TYR.completeRead(TYR.java:286)
at com.objectdb.o.TYR.readElement(TYR.java:263)
at com.objectdb.o.UTY.readAndAdjust(UTY.java:1322)
at com.objectdb.o.UMR$P.A(UMR.java:941)
at com.objectdb.o.UMR.z(UMR.java:569)
... 22 more

The _payload member referenced in the exception is a simple String but is referenced as type Object.

I'll try to put together an repeatable example but this may be difficult as this has worked fine in our test cases so far.

What I shouldve added is that this db is running in server mode.

I've taken a copy of the database which is causing problems. Its 83MB so I cant upload it but you're welcome to a copy of it if theres a way i can get it to you.

Looking at it in explorer shows a lot of "Broken References" (see screenshot) so I guess the problem is actually on writting the objects to the db.

Running the database through Database Doctor reported no errors.

When I try to access this database via a test case it throws the following error:

com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.TYM.al(TYM.java:696)
at com.objectdb.o.TYR.aB(TYR.java:560)
at com.objectdb.o.TYR.completeRead(TYR.java:286)
at com.objectdb.o.TYR.readElement(TYR.java:263)
at com.objectdb.o.UTY.readAndAdjust(UTY.java:1322)
at com.objectdb.o.UMR.readAndAdjust(UMR.java:603)
at rbccm.felix.objectdb.messaging.ObjectDbMessage.__odbReadContent(ObjectDbMessage.java:1)
at com.objectdb.o.MMM.ag(MMM.java:1040)
at com.objectdb.o.UTY.aH(UTY.java:1243)
at com.objectdb.o.UTY.aG(UTY.java:1215)
at com.objectdb.o.ENH.b(ENH.java:99)
at com.objectdb.o.LDR.I(LDR.java:437)
at com.objectdb.o.LDR.D(LDR.java:156)
at com.objectdb.o.OBC.aK(OBC.java:1027)
at com.objectdb.o.OBC.aI(OBC.java:917)
at com.objectdb.o.OBC.UG(OBC.java:802)
at com.objectdb.o.SRB.l(SRB.java:149)
at com.objectdb.o.QRR.b(QRR.java:180)
at com.objectdb.jpa.JpaQuery.getSingleResult(JpaQuery.java:654)

Could you put the database file on the Internet and send a link (here or in a private support ticket)?

Upload options are limited as I'm on a restricted company network but it turns out that rar compresses the file to less than 32MB so I've attached here.

Please let me know if theres any further information I can give you as I'm very keen to understand what is going on here. I'm trying to put together a smal test case to replicate the problem but I dont know how successful this will be.

It seems that the problem is in the _payload field. It contains instances of an unknown type whose schema is not stored in the database, i.e. there are broken references from entity object field values (probably in _payload) to a type that is missing in this database.

What do you store in that field? entity objects? embeddable objects? serializable objects?

Did you use that specific database in replication? Maybe this is related to the other reported problem of inability to update the schema in a replicated database?

It would be very helpful if you could prepare a test that reproduces this problem.

Currently across the application it can be a string or a byte[]. In this particular database it should always be a byte[] and the db is not replicated.

Understood about the test - the only problem is I have been unable to get it to break in via a test! I'll update as soon as I have something.

Apparently this specific database file is corrupted but the cause is unknown.

I am closing this issue now. If the problem repeats please open it and provide additional information.