Issue #1236: Internal Exception and Index Problems

Dear Support Team,

we have an urgent issue in a productive database. The objectdb doctor finds no error but one string field of one database item is not editable (even not using the objectdb explorer). The corrupt entry is in entity "UserModelData" with id="7d29d5eb-69a6-4d46-9bf1-802cbb6e439b". Try to change the userName attribute of this entry then the following error occurs:

Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) Client VM 1.7.0_25 (on Windows 8 6.2).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.InternalException.f(InternalException.java:236)
at com.objectdb.o.SPW.aq(SPW.java:109)
at com.objectdb.o.PGW.am(PGW.java:210)
at com.objectdb.o.UPT.C(UPT.java:134)
at com.objectdb.o.URT.l(URT.java:171)
at com.objectdb.o.TSK.i(TSK.java:145)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:86)
at com.objectdb.o.UTT.A(UTT.java:365)
at com.objectdb.o.UTT.l(UTT.java:218)
at com.objectdb.o.TSK.i(TSK.java:145)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:86)
at com.objectdb.o.MST.Vc(MST.java:1315)
at com.objectdb.o.WRA.Vc(WRA.java:372)
at com.objectdb.o.WSM.Vc(WSM.java:183)
at com.objectdb.o.STC.v(STC.java:514)
at com.objectdb.o.SHN.an(SHN.java:563)
at com.objectdb.o.SHN.K(SHN.java:170)
at com.objectdb.o.HND.run(HND.java:132)
at java.lang.Thread.run(Thread.java:724)

Cannot upload  the corrupt database because if I try to upload the file I get the following error:

An unrecoverable error occurred. This form was missing from the server cache. Try reloading the page and submitting again.

Uploading should now work. Could you please try again to upload the db file?

I uploaded the database.

The data in that specific object in field userName is not synchronized with the index on that field.

It is unclear how this happened, and unfortunately the Doctor doesn't include checks to identify this mismatch.

However it is very easy to fix the index by running the Doctor in repair mode:

> java com.objectdb.Doctor error.odb error2.odb

The new generated database doesn't have this problem.

Thanks, it solved the issue.

It would be great if you could find out how this happened because a lot of our databases are affected by this error. Maybe the reason was that we added a new field in a super class from that some entities derive from. Another reason could be that a entity class that didn't derive from that super class in previous versions now derives from this super class. In our case the super class is annotated with @Entity and not with @MappedSuperClass.

This exception can be thrown due to any addition or removal of persistent fields in an entity class, including moving a field in the hierarchy (which includes removing it from one class and adding it to another super/sub class), and due to any inheritance change (such as adding, removing or changing the super class).

You can avoid this issue by restarting the ObjectDB server after every such schema change.

Future versions of ObjectDB may support schema changes without restarting the server, or at least should produce better error messages in these cases. Currently the direction is to restart the server after schema changes.

But in our case a server restart doesn't help. The only way I figured out to migrate the old databases to the new schema is to access the database with the new database model (e.g. adding a new entry in the database) so the database gets corrupted and after that repairing the database with the doctor in repair mode. The main issue we have is that we don't get any exception from the database server when we access the corrupted database. Some queries just don't give any result if they access a not synchronized field (e.g. querying a UserModelData object by the corrupted field userName=...). Even the doctor finds no error which makes it pretty hard to find out afterwards which modification of the schema caused the issue.

Sorry, the answer in #7 was written under the impression that we were discussing the other issue that you filed (issue #1232), in which restarting the server solves the problem.

Regarding this issue, we need more information to understand the cause, but possibly this issue and issue #208 are related. ObjectDB assumes that always one schema version is in use. If a client has a new class schema and the server is not restarted and still uses the old schema, it is possible that wrong data will be used by the server when an object is indexed. We were aware to issues that are caused by schema changes against a running server, but until these reports we were not aware to the possibility that it may also corrupt the indexes. Now, when discussed, we will open this issue.

As a workaround, please remember to stop the server before future schema changes.

If the problem happens again (and hopefully it will not repeat with no schema changes against a running server) please report here.