Tracking changes to new collections (in enhancement mode) after flush

Hello,

We have an issue with a list as a value in a hashmap.

Within one transaction a new key-value pair is added to an existing hashmap. The key is a String, while the value is an ArrayList. During the same large transaction three elements are added to the list step by step, with many other db operations being performed in between.

Once the transaction is committed, the issue is that the last element in the list is not persisted and cannot be retrieved in another transaction. The last element is missed in the list.
Do you have an idea why?

@Entity

@Access (AccessType.FIELD)
public class MappingImpl implements Mapping {
    //the UID of the Mapping
    @Id
    @Access (AccessType.FIELD)
    @Column (name = "uid")
    private String uid;
    @OneToOne (fetch = FetchType.EAGER, cascade = {CascadeType.REFRESH, CascadeType.DETACH })
    private ModelElementImpl element;
    @OneToMany (fetch = FetchType.EAGER, cascade = {CascadeType.REFRESH, CascadeType.DETACH })
    private HashMap<String, List<ModelElementImpl>> mappingMap = new HashMap<>(1); (edited)

Several questions:

1. Is it always the last element in the ArrayList? Is there a special operation (e.g. flush) just before adding the last element?
2. Does it happen consistently? is it reproducible?
3. Does it happen with enhanced classes or in reflection mode?
4. What ObjectDB version are you using? Can you check if it also happens with version 2.8.4?

• yes, it is always the last element
• yes, there is probably a flush before adding the last element
• yes, it is always reproducible in our complex scenario with the same entity and uid
• the issue happens with enhanced classes, without enhanced classes all works correctly as expected
• currently we use ObjectDB version 2.8.0.b04

I will check it with 2.8.4 yet.

We found out a workaround.

If we touch again the map by map.put(key, list), although the key/value pair is already a part of the map, then it works also correctly.

But this approach is actually not the best way.

It seems like a failure in automatic tracking of changes in enhanced classes in this specific case. In reflection mode the entire content of the object is compared to the last saved content, so changes are detected. Enhancement mode is optimized to skip this comparison, relying on automatic invocation of change events, including in embedded collections. We should check how this is implemented for nested collections.

Note that this structure of a List as a value in a Map field is not standard (unsupported by JPA), although it should be supported by ObjectDB.

As a workaround you may try add a change to a direct field in the embedded object.

> If we touch again the map by map.put(key, list), although the key/value pair is already a part of the map, then it works also correctly.

Yes, this is a simple workaround.

> But this approach is actually not the best way.

Thank you for this report. We will check what can be done.

A further hint is also that the affected list in the map is still a java.util.ArrayList, not an objectdb.java.util.ArrayList, if the element is added to list which is missed in the next transaction.

The issue does also happen, if we use ObjectDB 2.8.4.

The following simple test program tries to reproduce the issue, but with no success, i.e. it works correctly with and without reflection. If you can change it so it would fail (based on what you have in your application), we should probably be able to fix the issue. If not, we will have to wait for more information.

import java.util.*;

import javax.persistence.*;

public class F2640 {
    public static void main(String[] args) {
        
        com.objectdb.Enhancer.enhance(F2640.class.getName());
        com.objectdb.Enhancer.enhance(F2640.ListInMap.class.getName());
        
        String dbUrl = "objectdb:F2640.tmp;drop";
        EntityManagerFactory emf = Persistence.createEntityManagerFactory(dbUrl);
        EntityManager em = emf.createEntityManager();
        
        em.getTransaction().begin();
        ListInMap entity = new ListInMap();
        em.persist(entity);
        entity.addValue();
        em.flush();
        long id = entity.id;
        entity.addValue();
        em.getTransaction().commit();
        em.clear();
        
        entity = em.find(ListInMap.class, id);
        System.out.println("Expected 2 values, found: " + entity.getCount());
        
        em.close();
        emf.close();
    }

    @Entity
    public static class ListInMap {
        private @Id long id;
        private Map<String, List<String>> map;
        
        public void addValue() {
            if (map == null)
                map = new HashMap<>();
            List<String> list = map.get("list");
            if (list == null)
                map.put("list", list = new ArrayList<>());
            list.add("item-" + list.size());
        }
        
        public int getCount() {
            return map.get("list").size();
        }
    }
}

We have an example:

public class MappingTest {

    public static void main(String[] args) {

        EntityManagerFactory emf = Persistence
            .createEntityManagerFactory("objectdb:c:\\test.tmp;drop");

        EntityManager em;

        //-------------------------------
        // invalid mapping between TL and CC
        em = emf.createEntityManager();
        em.setFlushMode(FlushModeType.AUTO);
        em.getTransaction().begin();

        ModelElementImpl m1 = new ModelElementImpl();
        em.persist(m1);
        ModelElementImpl m2 = new ModelElementImpl();
        em.persist(m2);
        MappingImpl mappingInvalid1 = new MappingImpl();
        mappingInvalid1.uid = m1.getUid();
        mappingInvalid1.element = m1;
        em.persist(mappingInvalid1);
        List<ModelElementImpl> mappedElements = new ArrayList<>(3);
        mappedElements.add(m2);
        mappingInvalid1.mappingMap.put("I", mappedElements);
        //-------------------------------

        //-------------------------------
        em.getTransaction().commit();
        em.close();

        em = emf.createEntityManager();
        em.setFlushMode(FlushModeType.AUTO);
        em.getTransaction().begin();
        //-------------------------------

        //-------------------------------
        // create observer elements
        ModelElementImpl m3 = new ModelElementImpl();
        em.persist(m3);
        //-------------------------------

        //-------------------------------
        // mapping model load
        TypedQuery<MappingImpl> query =
            em.createQuery("select m from MappingImpl m", MappingImpl.class);
        List<MappingImpl> mappingList = query.getResultList();
        for (MappingImpl mapping : mappingList) {
            mapping.element.getUid();
            mapping.mappingMap.isEmpty();
        }
        //-------------------------------

        //-------------------------------
        // mapping model save
        MappingImpl mapping1 = em.find(MappingImpl.class, m1.getUid());
            // mappingInvalid1 with invalid mapping
        mappedElements = new ArrayList<>(3);
        mappedElements.add(m3);
        mapping1.mappingMap.put("M", mappedElements);
        //-------------------------------

        //-------------------------------
        // create observer elements
        ModelElementImpl m5 = new ModelElementImpl();
        em.persist(m5);
        //-------------------------------

        //-------------------------------
        // mapping model load
        query = em.createQuery("select m from MappingImpl m", MappingImpl.class);
        mappingList = query.getResultList();
        for (MappingImpl mapping : mappingList) {
            mapping.element.getUid();
            mapping.mappingMap.isEmpty();
        }
        //-------------------------------

        //-------------------------------
        // mapping model save
        List<ModelElementImpl> mappings = mapping1.mappingMap.get("M");
        mappings.contains(m5);
        mappings.add(m5);
        //-------------------------------

        //-------------------------------
        // create harness elements
        ModelElementImpl m7 = new ModelElementImpl();
        em.persist(m7);
        //-------------------------------

        //-------------------------------
        // mapping model save
        mappings = mapping1.mappingMap.get("M");
        mappings.contains(m7);
        mappings.add(m7);
        //-------------------------------

        //-------------------------------
        em.getTransaction().commit();
        em.close();
        em = emf.createEntityManager();
        em.setFlushMode(FlushModeType.AUTO);
        em.getTransaction().begin();
        //-------------------------------

        mapping1 = em.find(MappingImpl.class, m1.getUid());
        mappings = mapping1.mappingMap.get("M");
        if (!mappings.contains(m3)) {
            throw new RuntimeException();
        }
        if (!mappings.contains(m5)) {
            throw new RuntimeException();
        }
        if (!mappings.contains(m7)) {
            throw new RuntimeException();
        }

        em.getTransaction().commit();
        em.close();

        emf.close();
    }

    @Entity
    public static class MappingImpl {

        @Id
        private String uid;

        @OneToOne (fetch = FetchType.EAGER, cascade = {CascadeType.REFRESH, CascadeType.DETACH })
        private ModelElementImpl element;

        @OneToMany (fetch = FetchType.EAGER, cascade = {CascadeType.REFRESH, CascadeType.DETACH })
        private HashMap<String, List<ModelElementImpl>> mappingMap = new HashMap<>(1);
    }

    @Entity
    public static class ModelElementImpl {

        private static int uidCounter = 1;

        @Id
        private String uid = "" + uidCounter++;

        public ModelElementImpl() {
        }

        public String getUid() {
            return this.uid;
        }

        @Override
        public boolean equals(Object obj) {
            return uid.equals(((ModelElementImpl)obj).getUid());
        }

        @Override
        public int hashCode() {
            return uid.hashCode();
        }
    }
}

Thank you for this important test case, which helped to find the bug. Actually the issue is not related to nested collections (and therefore the title of this thread was updated), but to tracking changes after flush to new collections that are added to an object of an enhanced entity classes before that flush. This can be demonstrated by the simpler test case below, with no nested collections. Build 2.8.4_02 should fix this issue.

import java.util.*;

import javax.persistence.*;

public class F2640c {
    public static void main(String[] args) {
        
        com.objectdb.Enhancer.enhance(F2640c.class.getName());
        com.objectdb.Enhancer.enhance(F2640c.EntityWithList.class.getName());
        
        String dbUrl = "objectdb:F2640c.tmp;drop";
        EntityManagerFactory emf = Persistence.createEntityManagerFactory(dbUrl);

        EntityManager em = emf.createEntityManager();
        em.getTransaction().begin();
        EntityWithList entity = new EntityWithList();
        em.persist(entity);
        em.flush();
        long id = entity.id;
        em.getTransaction().commit();
        em.clear();
        em.close();
        
        em = emf.createEntityManager();
        entity = em.find(EntityWithList.class, id);
        em.getTransaction().begin();
        entity.list = new ArrayList();
        em.flush();
        entity.list.add("not-tracked");
        em.getTransaction().commit();
        em.close();
        
        em = emf.createEntityManager();
        entity = em.find(EntityWithList.class, id);
        System.out.println("Expected one value, found: " + entity.list.size());
        em.close();
        
        emf.close();
    }

    @Entity
    public static class EntityWithList {
        private @Id long id;
        private List<String> list;
    }
}

The build 2.8.4_02 solves the issue. Thank you very much.