InternalException when reading a HashMap

Hi,

our app threw this exception during saving an object after running for several days (using the database) and we have the proper activation key set in objectdb.conf file.

Caused by: com.objectdb.o._JdoDataStoreException: Too many persistable types (>10) - exceeds evaluation limit
        at com.objectdb.o.JDE.d(JDE.java:119)
        at com.objectdb.o.ERR.h(ERR.java:56)
        at com.objectdb.o.JDE.h(JDE.java:54)
        at com.objectdb.o.OBC.onObjectDBError(OBC.java:1588)
        at com.objectdb.jpa.EMImpl.commit(EMImpl.java:289)
        ... 16 more

a few days later an instance of the same class was saving fine - no exception thrown.

8 days later the same exception was thrown for another class and another DB file in the same running application.

Apart from those 2 exceptions the database for all DB files was working well.

Today we restarted the app and it threw the following exception during startup while trying to read the instances of the first class which reported the exception above:

[ObjectDB 2.8.3_05] Unexpected exception (Error 990)
  Generated by OpenJDK 64-Bit Server VM 17.0.3 (on Linux 4.15.0-65-generic).
Please report this error on http://www.objectdb.com/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
        at com.objectdb.o.TYM.y(TYM.java:794)
        at com.objectdb.o.TYR.ao(TYR.java:606)
        at com.objectdb.o.TYR.completeRead(TYR.java:320)
        at com.objectdb.o.TYR.readElement(TYR.java:296)
        at com.objectdb.o.UTY.readAndAdjust(UTY.java:1616)
        at com.objectdb.o.MPT.readArray(MPT.java:427)
        at com.objectdb.o.MPT.readArray(MPT.java:407)
        at com.objectdb.o.MPT.r(MPT.java:333)
        at com.objectdb.o.TYT.extractMap(TYT.java:109)
        at com.objectdb.o.ENT.extractMap(ENT.java:1608)
        at objectdb.java.util.HashMap.__odbBeforeAccess(Unknown Source)
        at objectdb.java.util.HashMap.keySet(Unknown Source)
        at quant.phdsc_new.utility.Tool.deepCopy(Unknown Source)
 

The Doctor did not find any errors in the DB file.

We were then searching for backups of the database which would still work and found out that the latest backup which works is the one just before the first report of "Too many persistable types".

We were then testing the same database files on our latest code.

It was throwing a different exception, but also on the first access of some objects from the corrupted file: 

java.lang.InstantiationError: eu.extech.quant.channels.interfaces.IChannels$WithChannelType
        at com.objectdb.o.JDE.d(JDE.java:128)
        at com.objectdb.o.ERR.h(ERR.java:56)
        at com.objectdb.o.JDE.h(JDE.java:54)
        at com.objectdb.o.JDE.j(JDE.java:166)
        at com.objectdb.o.JDE.d(JDE.java:75)
        ... 19 more
Caused by: com.objectdb.o._JdoUserException: Failed to construct an instance of java.lang.Object using a no-arg constructor
NestedThrowables:
java.lang.InstantiationError: eu.extech.quant.channels.interfaces.IChannels$WithChannelType
        at com.objectdb.o.JDE.d(JDE.java:128)
        at com.objectdb.o.ERR.h(ERR.java:56)
        at com.objectdb.o.JDE.h(JDE.java:54)
        at com.objectdb.o.JDE.j(JDE.java:166)
        at com.objectdb.o.JDE.d(JDE.java:75)
        ... 23 more
Caused by: java.lang.InstantiationError: eu.extech.quant.channels.interfaces.IChannels$WithChannelType
        at jdk.internal.reflect.GeneratedSerializationConstructorAccessor24.newInstance(Unknown Source)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at com.objectdb.o.TYH.e(TYH.java:975)
        at com.objectdb.o.UTY.newInstance(UTY.java:1316)
        at com.objectdb.o.UTY.readStrictly(UTY.java:1578)
        at com.objectdb.o.TYR.ao(TYR.java:609)
        at com.objectdb.o.TYR.completeRead(TYR.java:320)
        at com.objectdb.o.TYR.readElement(TYR.java:296)
        at com.objectdb.o.UTY.readAndAdjust(UTY.java:1616)
        at com.objectdb.o.MPT.readArray(MPT.java:427)
        at com.objectdb.o.MPT.readArray(MPT.java:407)
        at com.objectdb.o.MPT.r(MPT.java:333)
        at com.objectdb.o.TYT.extractMap(TYT.java:109)
        at com.objectdb.o.ENT.extractMap(ENT.java:1608)
        at objectdb.java.util.HashMap.__odbBeforeAccess(Unknown Source)
        at objectdb.java.util.HashMap.values(Unknown Source)
        at com.objectdb.o.MPT.visitRefs(MPT.java:174)
        at com.objectdb.o.TVS.e(TVS.java:170)
        at com.objectdb.o.TVS.cascade(TVS.java:157)
        at com.objectdb.o.TVS.b(TVS.java:106)
        at com.objectdb.o.TVS.b(TVS.java:94)
        at com.objectdb.jdo.PMImpl.deletePersistentAll(PMImpl.java:380)
        ... 15 more

 

We then tried it with ObjectDB 2.7.0, but that did not make any change.

Please give us some advice how to repair the damaged file and how to recognize this kind of mistakes earlier next time.

Hi,

Usually running the Doctor on every backup (and at least daily) can help in detecting issues early. Maybe in this case, as you have an error that is not detected by the Doctor, running your own tests in addition to running the Doctor on a daily backup could help.

If you can share the damaged database (providing a link to download it in a private support ticket) and also provide information regarding the damaged objects, we may be able to improve the Doctor and also fix the database.

Another thing that you can try is running the Doctor in repair mode (even if errors are not found) and then, maybe the result database would work well.

Although the stack trace indicates an error while reading the hash map, if some objects in the database are corrupted then unfortunately, the exact stack trace is not very helpful.

Finally, there is also a possibility that the data in the database is ok but something fails with the automatic schema evolution, if you changed your persistable classes recently.

For now I just add, that something change on our server probably, although we did not do anything intentionally.
Also the network interface was not changed. But from some moment old activation key stopped working. We have running many services on that server, so we found out this when trying to update some of them. They are all independent services running with it's own objectdb.conf.

So we added the new activation key to all of them, to prevent the Too many persistable types for future updates. However this one service thrown this error even without the restarting of JVM.

Is this wanted behaviour?

We'll keep investigating with the broken DB and we'll try the repair mode. I am not sure if we would be able to provide the database, but we'll try to.

I see now, that we tried to run the doctor in repair mode. It does not helped unfortunatelly.

What helped was deleting the entity completely in explorer including all it's instances (deleting only instances did not helped)

But that way we found, that another entities are also corrupt.

> Is this wanted behaviour?

Indeed it seems restrictive. We plan to switch to a new automatic activation system, so ObjectDB would be able to re-activate automatically in such incidents, if it has internet access to our activation server.