em.flush(); em.clear(); loosing data and not persisting managed objects

#1

I'm having trouble with flush() and clear() in a loop inside a transaction not persisting modified objects to database.

Sometimes it does other times it doesn't.

Here is my Entity.

import javax.persistence.Basic;
import javax.persistence.Entity;
import javax.persistence.Id;
import javax.persistence.Lob;
import javax.persistence.Table;

@Entity
@Table(name = "TestEntity", schema = "mySchema")
public class TestEntity
{
    @Id
    private int    id;
    @Basic
    @Lob
    private String  remark;

    public int getId()
    {
      return id;
    }

    public void setId(int id)
    {
      this.id = id;
    }

    public String getRemark()
    {
      return remark;
    }

    public void setRemark(String remark)
    {
      this.remark = remark;
    }

}

Here is my Test program that fails.

import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.Persistence;

public class Test
{
private final EntityManagerFactory emf;
private final EntityManager      em;

public Test()
{
  emf = Persistence.createEntityManagerFactory("$objectdb/db/test.odb");
  em = emf.createEntityManager();
}

public static void main(String[] args)
{
  Test t = new Test();
  t.doit();
}

public void doit()
{
  String sbs = "This is a test String";
  em.getTransaction().begin();
  for(int i=0;i<Integer.MAX_VALUE;i++)
  {
   TestEntity v = new TestEntity();
   v.setId(i);
   em.persist(v);
   v.setRemark(sbs);
   if(i % 10 == 0)
   {
    System.out.println("flushing");
    em.flush();
    em.clear();
   }
   System.out.println("i="+i);
   TestEntity t1 = em.find(TestEntity.class, i);
   String rem = t1==null ? null : t1.getRemark();
   if(rem==null || rem.compareTo(sbs)!=0)
   {
    System.out.println("Validate failed for i="+i);
    System.out.println("t1="+t1);
    System.out.println("rem="+rem);
    System.exit(1);
   }
  }
  em.getTransaction().commit();
}
}

Here is my failure output.

i=75
i=76
i=77
i=78
i=79
flushing
i=80
Validate failed for i=80
t1=null
rem=null

#2

Thank you very much for this useful test. Please try build 2.2.3_06 that should fix this bug.

ObjectDB Support
#3

This fix works except when the primary key is a string.

Attached is my test code above modified with a String as a key.

The output when I run the test is

i=127
i=128
i=129
flushing
i=130
i=131
i=132
i=133
i=134
i=135
i=136
i=137
i=138
i=139
flushing
Exception in thread "main" [ObjectDB 2.2.3_06] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_24 (on Linux 2.6.32-71.24.1.el6.x86_64).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.InternalException.f(InternalException.java:236)
at com.objectdb.o.UTT.G(UTT.java:528)
at com.objectdb.o.UTT.E(UTT.java:491)
at com.objectdb.o.PGW.at(PGW.java:414)
at com.objectdb.o.PGW.am(PGW.java:229)
at com.objectdb.o.UPT.A(UPT.java:134)
at com.objectdb.o.UPT.l(UPT.java:110)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.UPT.q(UPT.java:157)
at com.objectdb.o.PGT.p(PGT.java:104)
at com.objectdb.o.UPT.A(UPT.java:121)
at com.objectdb.o.URT.l(URT.java:171)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.TSM.e(TSM.java:87)
at com.objectdb.o.UTT.z(UTT.java:362)
at com.objectdb.o.UTT.l(UTT.java:203)
at com.objectdb.o.TSK.i(TSK.java:146)
at com.objectdb.o.TSK.f(TSK.java:95)
at com.objectdb.o.MST.UW(MST.java:1127)
at com.objectdb.o.WRA.UW(WRA.java:351)
at com.objectdb.o.WSM.UW(WSM.java:146)
at com.objectdb.o.WRA.UW(WRA.java:351)
at com.objectdb.o.WSN.UW(WSN.java:612)
at com.objectdb.o.OBM.bL(OBM.java:935)
at com.objectdb.o.OBM.bI(OBM.java:811)
at com.objectdb.o.OBM.flush(OBM.java:732)
at Test.doit(Test.java:38)
at Test.main(Test.java:19)

 

#4

Thank you for this bug report. Build 2.2.3_09 should fix it.

As a side note - ObjectDB doesn't limit the length of primary keys, but large strings as demonstrated by this test should be avoided, if possible, because a primary key whose length is larger than about 45 bytes is much less efficient.

ObjectDB Support
#5

I"m storing data from files that was originally in a RDMS and exported as csv like files (1 file per table with one row per entry).

These tables are the data from the backend of an antiquated source code/bug tracking management system called CMVC, since I don't have direct access to that database to get the data directly.

The code is stored on the file system in SCCS, but the history of each change and defects are stored in a database.

Each table has hundreds of thousands of entries.

I'm trying to write some custom code to analyze these tables and maybe migrate this history to a more modern distributed revision cotrol system (DRCS) like hg and git.

But before that can begin I have to import these csv files into a real database so the data can be queries and analyzed to convert it to a format sutable for the new DRCS.

I chose JPA since it was not specific to a peticular database.

I chose to try openDB since it directly supports JPA.

Most of these tables map directly to Objects very well and have a sequencial integer as the primary key.

But of all the file/tables 3 or 4 or them do not have primary keys.

The data that I'm using in these entity has multiple fields that are related but takes too long to access when using embededId's. Maybe my embedable class's hashcode and equal functions are the problem here.

5+ hrs to import the data (if it doesn't fail)  vs 15min when using the scheme below for a Primary Key.

 

So I created a string representation of the the key fields and calculate a SHA1 of that string and use it as the ID.

This gives me a String object that is always a 40 digit hex string and should be unique among the key fields.

 

For one of those three tables Notes the method used is below.

The key fields are

1 - Another Entity object ( I'm using its integer id since the original RDMS had a FK ref to the int PK of another table)

2 - Primative int field (using it as is. This is just a sequence number)

3 - A Date object (I'm using its .toString() representation. This is the timestamp of when the Note was taken)

I then concantinate them with "/" seperating them and use the resulting string as my Primary Key.

When doing this the em.find() method seems to be fast.

 

But even with 2.2.3_09 applied I still get a internal error at some point while importing the Notes data.

 

But since I'm having trouble with String Primary Keys, I'm going to try the other multi ID method  of @IdClass and multiple @Id fields and a non annotated POJO as shown in the manual

 

#6

I doubt that ID class would give you better results than embedded IDs since they are very similar.

Actually it is very strange that just switching to string primary keys provides performance boost of x20.

If you can change your test case to demonstrate this - it is certainly something that I will explore.

Regarding the new internal exception - could you please attach the stack trace?

ObjectDB Support
#7
Updating note d60/0/0/Tue Jun 04 06:53:20 EDT 1996
Updating note d56/0/0/Tue Jun 04 07:34:10 EDT 1996
Updating note d61/0/0/Tue Jun 04 10:24:56 EDT 1996
com.objectdb.o.InternalException
[ObjectDB 2.2.3_09] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_24 (on Linux 2.6.32-71.24.1.el6.x86_64).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.VUT.k(VUT.java:677)
at com.objectdb.o.VUT.j(VUT.java:346)
at com.objectdb.o.STV.I(STV.java:282)
at com.objectdb.o.PAG.aq(PAG.java:956)
at com.objectdb.o.MST.aQ(MST.java:476)
at com.objectdb.o.MST.UO(MST.java:427)
at com.objectdb.o.WRA.UO(WRA.java:244)
at com.objectdb.o.WRA.UO(WRA.java:244)
at com.objectdb.o.WSN.UO(WSN.java:302)
at com.objectdb.o.LDR.H(LDR.java:348)
at com.objectdb.o.LDR.G(LDR.java:286)
at com.objectdb.o.LDR.D(LDR.java:204)
at com.objectdb.o.OBC.aM(OBC.java:1028)
at com.objectdb.o.OBC.aK(OBC.java:929)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:542)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:465)
at com.ibm.check.cmvc2git.database.Database.getNote(Database.java:672)
at com.ibm.check.cmvc2git.database.NotesSync.updateDatabase(NotesSync.java:76)
at com.ibm.check.cmvc2git.database.CMVCdbSync.syncDB(CMVCdbSync.java:99)
at com.ibm.check.cmvc2git.CMVCSync.syncDB(CMVCSync.java:200)
at com.ibm.check.cmvc2git.CMVCSync.runCommand(CMVCSync.java:115)
at com.ibm.check.cmvc2git.CMVCSync.main(CMVCSync.java:61)

 

#8
[2011-05-05 22:28:02 #132 query.manager]
<queryRequest query="SELECT MAX($1.addDate) FROM Notes $1 WHERE $1.addDate IS NOT NULL" args="null" transactionId="-1" />

[2011-05-05 22:28:02 #133 query.tokenizer]
<tokenList>
    <token name="SELECT" code="132" postion="0:6" />
    <token name="MAX" code="118" postion="7:10" />
    <token name="(" code="14" postion="10:11" />
    <token name="$1" code="1" postion="11:13" />
    <token name="." code="6" postion="13:14" />
    <token name="addDate" code="1" postion="14:21" />
    <token name=")" code="15" postion="21:22" />
    <token name="FROM" code="103" postion="23:27" />
    <token name="Notes" code="1" postion="28:33" />
    <token name="$1" code="1" postion="34:36" />
    <token name="WHERE" code="150" postion="37:42" />
    <token name="$1" code="1" postion="43:45" />
    <token name="." code="6" postion="45:46" />
    <token name="addDate" code="1" postion="46:53" />
    <token name="IS NOT NULL" code="62" postion="54:65" />
</tokenList>

[2011-05-05 22:28:02 #134 query.tokenizer]
<token name="QUERY" code="67" postion="0:0">
    <token name="SELECT" code="132" postion="0:6">
        <token name="MAX" code="118" postion="7:10">
            <token name="(" code="14" postion="10:11">
                <token name="." code="6" postion="13:14">
                    <token name="$1" code="1" postion="11:13" />
                    <token name="addDate" code="1" postion="14:21" />
                </token>
                <token name=")" code="15" postion="21:22" />
            </token>
        </token>
    </token>
    <token name="FROM" code="103" postion="23:27">
        <token name="$1" code="1" postion="34:36">
            <token name="Notes" code="1" postion="28:33" />
        </token>
    </token>
    <token name="WHERE" code="150" postion="37:42">
        <token name="IS NOT NULL" code="62" postion="54:65">
            <token name="." code="6" postion="45:46">
                <token name="$1" code="1" postion="43:45" />
                <token name="addDate" code="1" postion="46:53" />
            </token>
        </token>
    </token>
</token>

[2011-05-05 22:28:02 #135 query.compiler]
<queryCompilation query="SELECT MAX($1.addDate) FROM Notes $1 WHERE $1.addDate IS NOT NULL">
    <symbolManager>
        <parameters />
        <variables>
            <$1 varId="0" type="com.ibm.check.cmvc2git.entities.Notes" subtypes="include" typeId="5" binding="null" nav="1">
                <keys />
                <navigations>
                    <addDate member="addDate" memberType="java.util.Date" id="0" />
                </navigations>
            </$1>
        </variables>
    </symbolManager>
    <queryCompData>
        <query exp="query(where(($1.addDate!=null)),select(max($1.addDate)))" var="1" directVar="0" ext="1">
            <where exp="where(($1.addDate!=null))" var="1" directVar="0" ext="1">
                <notEquals exp="($1.addDate!=null)" type="boolean" var="1" directVar="0" ext="1">
                    <member exp="$1.addDate" type="java.util.Date" var="1" directVar="0" ext="1" memberName="addDate" />
                    <literal exp="null" type="java.lang.Object" var="0" directVar="0" ext="0" />
                </notEquals>
            </where>
            <select exp="select(max($1.addDate))" var="1" directVar="0" ext="1">
                <max exp="max($1.addDate)" type="java.util.Date" var="1" directVar="0" ext="1">
                    <member exp="$1.addDate" type="java.util.Date" var="1" directVar="0" ext="1" memberName="addDate" />
                </max>
            </select>
        </query>
        <globals globalVariables="0" globalNavigations="1" />
    </queryCompData>
</queryCompilation>

[2011-05-05 22:28:02 #136 query.compiler]
<query-planning filter="($1.addDate!=null)">
    <step step="1" var="$1" filter="($1.addDate!=null)">
        <allPlans>
            <filterPlan plan="filter(extract($1,type(Notes[all])),($1.addDate!=null))" eval="3.4426" var="1" prm="1" reg="1" filter="($1.addDate!=null)" />
        </allPlans>
        <selectedPlans>
            <filterPlan plan="filter(extract($1,type(Notes[all])),($1.addDate!=null))" eval="3.4426" var="1" prm="1" reg="1" filter="($1.addDate!=null)" />
        </selectedPlans>
    </step>
</query-planning>

[2011-05-05 22:28:02 #137 query.compiler]
<finalPlans>
    <groupPlan plan="group(filter(extract($1,type(Notes[all])),($1.addDate!=null)))" eval="3.6979" var="1" prm="1" reg="1" group="null" eval="[max($1.addDate)]">
        <filterPlan plan="filter(extract($1,type(Notes[all])),($1.addDate!=null))" eval="3.4426" var="1" prm="1" reg="1" filter="($1.addDate!=null)">
            <extractPlan plan="extract($1,type(Notes[all]))" eval="3.4214" var="1" prm="1" reg="1" variable="$1">
                <btreePlan plan="type(Notes[all])" eval="3.4214" var="1" prm="1" reg="0" variable="$1" />
            </extractPlan>
        </filterPlan>
    </groupPlan>
</finalPlans>

[2011-05-05 22:28:02 #138 query.manager]
<newQueryProgram>
    <groupPlan plan="group(filter(extract($1,type(Notes[all])),($1.addDate!=null)))" eval="3.6979" var="1" prm="1" reg="1" group="null" eval="[max($1.addDate)]">
        <filterPlan plan="filter(extract($1,type(Notes[all])),($1.addDate!=null))" eval="3.4426" var="1" prm="1" reg="1" filter="($1.addDate!=null)">
            <extractPlan plan="extract($1,type(Notes[all]))" eval="3.4214" var="1" prm="1" reg="1" variable="$1">
                <btreePlan plan="type(Notes[all])" eval="3.4214" var="1" prm="1" reg="0" variable="$1" />
            </extractPlan>
        </filterPlan>
    </groupPlan>
</newQueryProgram>

[2011-05-05 22:28:02 #139 query.manager]
<newQueryResponse isUnique="true" fetchSize="0">
    <SimpleResultBuilder name="MAX($1.addDate)" type="java.util.Date" />
</newQueryResponse>

[2011-05-05 22:28:03 #140 *]
[ObjectDB 2.2.3_09] Unexpected exception (Error 990)
  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_24 (on Linux 2.6.32-71.24.1.el6.x86_64).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.VUT.k(VUT.java:677)
at com.objectdb.o.VUT.j(VUT.java:346)
at com.objectdb.o.STV.I(STV.java:282)
at com.objectdb.o.PAG.aq(PAG.java:956)
at com.objectdb.o.MST.aQ(MST.java:476)
at com.objectdb.o.MST.UO(MST.java:427)
at com.objectdb.o.WRA.UO(WRA.java:244)
at com.objectdb.o.WRA.UO(WRA.java:244)
at com.objectdb.o.WSN.UO(WSN.java:302)
at com.objectdb.o.LDR.H(LDR.java:348)
at com.objectdb.o.LDR.G(LDR.java:286)
at com.objectdb.o.LDR.D(LDR.java:204)
at com.objectdb.o.OBC.aM(OBC.java:1028)
at com.objectdb.o.OBC.aK(OBC.java:929)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:542)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:465)
at com.ibm.check.cmvc2git.database.Database.getNote(Database.java:672)
at com.ibm.check.cmvc2git.database.NotesSync.updateDatabase(NotesSync.java:76)
at com.ibm.check.cmvc2git.database.CMVCdbSync.syncDB(CMVCdbSync.java:99)
at com.ibm.check.cmvc2git.CMVCSync.syncDB(CMVCSync.java:200)
at com.ibm.check.cmvc2git.CMVCSync.runCommand(CMVCSync.java:115)
at com.ibm.check.cmvc2git.CMVCSync.main(CMVCSync.java:61)

This is the last entries in the log with log level debug.

#9

As far as the String vs @EmbedableId it will take me a while to create test code that shows my environment. I'm not going to have much time to work on this project till after next week so it may be a while before I post it.

 

#10

Unfortunately it seems that the stack trace will not be sufficient in this case. If and when you have time to generate a test case that demonstrates the exception it would be very helpful. Thank you very much for your useful reports.

ObjectDB Support

Reply