Issue #2333: Removed entities can be found by query in the same transaction

Hello,

we opened a transaction, load an entity from database and removed this entity by em.remove().

Afterwards the entity can be load again by query although the resulted entity have an attribute __odbtracker/m (type STA, value=Persistent-Deleted-Flushed) which shows that the entity is deleted and flushed.

The bug was introduced with ObjectDB Version 2.7.4_04, therefore in the moment we are using 2.7.4_03.

Unfortunately it is difficult to create an example, because the error only occurs in a complex scenario.

Maybe you have a hint how we can better analyze the issue or do you have an idea what is the problem?

best regards BTC

If an entity was deleted from the database and the change was flushed but not committed yet then the database maintains at least two different snapshots of objects - one with the deleted object and the other without the deleted object.

Apparently the query was directed to the wrong snapshot.

Are you sure that this bug exists in 2.7.4._04 but not in 2.7.4_03?

In that case we can investigate the exact changes between these two builds.

Yes, I have just tested it again.

The error didn't occur with 2.7.4_03, but with 2.7.4_04.

Build 2.7.4_04 fixes issue #2280 but seems to cause new issues. We will check.

We may have to reverse the changes of build 2.7.4_04.

We will continue this discussion in the thread of issue #2280.

It is unclear why the change in version 2.7.4_04 fails, but it was an essential change that we should keep. Unfortunately we couldn't reproduce this issue yet or understand what may cause it.

Some explanation on the change in build 2.7.4_04:

ObjectDB uses snapshots in order to isolate transactions, i.e. to hide uncommitted flush changes of one transaction from other transactions, but still make them available to that transaction. Every snapshot represents a picture of the database as seen by a particular transaction. When changes are flushed, a snapshot is generated, and further queries in that transaction uses that snapshot. That snapshot is only visible to that transaction.

The problem before build 2.7.4_04 was that as soon as flush was called and a snapshot was generated, changes to the database by other transactions after that flush were not visible to that transaction (until commit). Build 2.7.4_04 fixed it by creating a new snapshot on query execution if changes have been committed after the last flush by other transactions. The new snapshot includes all the up to date changes including changes in the current transaction. Therefore, the cause of this new issue is unclear.

Hopefully this information may assist in isolating this issue.

We have found the problem. There were 'DELETE FROM' querys on another entity type which cause the problems.

But the another entity type has not any relation to the failed entity type. Can you explain why the deletion of entities causes strange visibility of other entities which have another type?

It is possible that the fix in build 2.7.4_04, which improves visibility of committed changes in other transactions after flush conflicts with preserving visibility of changes when a DELETE query is executed. DELETE queries bypass the ordinary transaction management on the client side and may have different effect than ordinary operation within transactions.

You wrote that you found the problem but it seems that you still have unexpected behaviour that requires an additional fix, right? Are you now in a position to submit a test case or maybe alter the test case of issue #2280 to demonstrate it? maybe by adding a DELETE query in a specific timing? If a fix is needed and you haven't already solved it somehow.

Build 2.7.6 includes an additional related fix. It solves another issue (unexpected optimistic lock exception) but you may want to check if it also solves this issue.