Issue #693: InternalException

I'm getting a new InternalException (see below). No repeatable example as yet but will work on that and let you know as soon as we have one. If there's anything in the exception which could help us narrow down the problem please let me know.

[ObjectDB 2.3.7_02] javax.persistence.RollbackException
Failed to commit transaction:  (error 613)
at com.objectdb.jpa.EMImpl.commit(EMImpl.java:279)
at rbccm.felix.objectdb.workflow.ObjectDbInstanceContainer.commit(Unknown Source)
at rbccm.digest.workflow.execution.WorkflowRunner.cleanup(Unknown Source)
at rbccm.digest.workflow.execution.WorkflowRunner.workflowCompleteforThisRunner(Unknown Source)
at rbccm.digest.workflow.execution.WorkflowRunner.workflowCompleteForSomeTasks(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.framework.service.ServiceRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)
Caused by: com.objectdb.o.InternalException
at com.objectdb.o.InternalException.b(InternalException.java:88)
at com.objectdb.o.VAL.w(VAL.java:601)
at com.objectdb.o.UTY.aK(UTY.java:1387)
at com.objectdb.spi.DetachedTracker.<init>(DetachedTracker.java:82)
at com.objectdb.o.ENT.getDetachTracker(ENT.java:915)
at com.objectdb.o.ENT.w(ENT.java:329)
at com.objectdb.o.STA.ae(STA.java:819)
at com.objectdb.o.STM.J(STM.java:599)
at com.objectdb.o.OBM.bN(OBM.java:1042)
at com.objectdb.o.OBM.bI(OBM.java:825)
at com.objectdb.o.OBM.bG(OBM.java:719)
at com.objectdb.jpa.EMImpl.commit(EMImpl.java:276)
... 9 more

The exception is thrown during detachment of an object (at the end of commit).

Apparently the specific detached object that causes the exception has no id (primary key) fields. ObjectDB supports automatic ids (when no id fields are defined) but then the type of the automatic id is always long. In this specific case the object has an id that is not long.

I think that such an exception is possible as a result of schema change, in which id fields are removed. Notice that changing the primary key of an existing entity class with instances in the database is not supported.

When you say schema change do you mean changing the code for the entity objects and using it against an existing database? In this case the app is starting against a clean database (literally a new file) and the error occurred mid-run so there is no opportunity for code change.

I've also found the following error in the logs so it seems that there is definitely something going wrong with the ids but at the moment I'm confused as to how this is possible. We're using an application id as the primary key so I would expect to see wider app failures if it was null.

2012-02-24 18:39:55,719 pdnsrtlonmo42-WSAgent-1-Existing/0 ERROR WorkflowRunner - The identity must not be null.
javax.jdo.JDONullIdentityException: The identity must not be null.
at javax.jdo.identity.SingleFieldIdentity.assertKeyNotNull(SingleFieldIdentity.java:92)
at javax.jdo.identity.ObjectIdentity.<init>(ObjectIdentity.java:60)
at com.objectdb.o.UTY.aK(UTY.java:1404)
at com.objectdb.spi.DetachedTracker.<init>(DetachedTracker.java:82)
at com.objectdb.o.ENT.getDetachTracker(ENT.java:910)
at com.objectdb.o.ENT.v(ENT.java:324)
at com.objectdb.o.STA.ae(STA.java:819)
at com.objectdb.o.STM.J(STM.java:599)
at com.objectdb.o.OBM.bM(OBM.java:1042)
at com.objectdb.o.OBM.rollbackTransaction(OBM.java:1009)
at com.objectdb.jpa.EMImpl.rollback(EMImpl.java:292)
at rbccm.felix.objectdb.workflow.ObjectDbInstanceContainer.rollback(Unknown Source)
at rbccm.digest.workflow.execution.WorkflowRunner.workflowCompleteForSomeTasks(Unknown Source)
at rbccm.digest.workflow.execution.WorkflowRunner.workflowCompleteForSomeTasks(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.gridservice.workflow.WorkflowService.process(Unknown Source)
at rbccm.felix.framework.service.ServiceRunner.run(Unknown Source)
at java.lang.Thread.run(Thread.java:662)

In that case it is not a schema change problem of course.

Please run the Doctor to check the database file. Can you also upload a sample database?

I've attached 3 affected databases and the doctor output which is similar for all of them.

I still don't have a simple recreation but a few changes in the app-objectdb interaction seems to work around the problem.

In the error version I'm keeping the entityManager at the thread level. It is created the first time a thread accesses a database and is stored for future use. The app is then effectively doing the following actions in a loop:

1. Open a new transaction
2. Load an object from the database
3. Update the object
4. Commit the transaction, storing the updates.

When the app is finished with the object, the actions are:

1. Open a new transaction
2. Load an object from the database
3. Remove the object
4. Commit the transaction, storing the updates.

If another thread has updated the same object the RollbackException is caught and the transaction is re-attempted from scratch.

In the workaround version a new entityManager is opened & closed for each action (load, remove, etc). When an update has occurred the actions are now:

1. Open a new EntityManager
2. Open a new transaction
3. Merge the updated object (entityManager.merge)
4. Commit the transaction

Could you please check if building the database with the schema (i.e. create a first EntityManager, register all the entity classes in the database by persisting and removing dummy objects, and then close the EntityManager) before the ordinary working threads start and their EntityManager instances are instantiated makes any difference?

That is effectively already happening. The app registers all entity classes using the em.getMetamodel().entity() method on start-up (otherwise we used to get errors on empty database queries). The app also runs for a period of time before the first error occurs so several objects will have been saved by then.

I'll try to create a test case based on the workaround I described above - it gives me a starting point at least - and will let you know if/when I have any success.

Any news about this issue?

None I'm afraid - I haven't been able to look at it further as the push has been on getting the application out using the workaround described above. Hopefully I'll be able to work on it further in the next week or so.

Feel free to close the issue until I can report back with more detail if you wish.