Empty temporary dat files in the temp folder

Hello,
We have questions regarding the pagesX.dat files in the temp directory.

We detected a problem on a machine with ‘only’ 2GB free memory on disc drive C:/ because during using our Product the %temp%/objectdb_123... folder growth up to 2GB. And then we run into some following errors due to lack of memory.

Some information to our scenario:
We use one main database file to which we continuously have a connection (open EntityManagerFactory). For each user action we create an EntityManager, begin a transaction, do some stuff (find, persist, remove …) and then commit or in the case of errors rollback the transaction and closing the EntityManager.
In some scenarios we use additional databases as large data sink. And for optimized data access we hold the connection to the EntityManager.

In the observed use case the database’s are very small (main DB ~ 3MB and three additional DB’s with just 1 MB)
If we access entities from one of the additional DB’s we get five new pagesX.dat files (each 8MB) in the TEMP directory. If we take a look into the dat files they are all (with exception of the first one) empty. Or rather pre filled with null.

We observed that the directory is cleaned up sporadic. But for us is the worst case very important. We need to know how big can be the directory.
- Is there a threshold?
   - Is the threshold adjustable?
- Is it possible to reduce the initial size (actually 8MB)?
- What performance advantages bring these files? Is this possible/ advisable to disable?
- What could we do wrong?
- Is our scenario with multiple DBs the reason for this behavior?

The pages*.dat files are generated when objectdb.temp.page-file is enabled (as part of the new experimental feature of supporting very large transactions, as requested by you here). Note that you may be the only ObjectDB users that are currently using this new feature.

Only one such file should be created per open database file with the initial size of 1000 pages (default page size is 2KB but it may be 8KB in your configuration). The file will expand to contain large transactions, so its maximum size depends on the size of your transactions.

Your post title mentions high memory usage. Did you mean high RAM usage in addition to these files?

Hello,

with memory (in the title) we mean the unexpected high disk usage of the *.dat files, not RAM.

I prepared a TestCase to demonstrate the behavior:

import javax.persistence.Basic;
import javax.persistence.Entity;
import javax.persistence.EntityManager;
import javax.persistence.EntityManagerFactory;
import javax.persistence.FlushModeType;
import javax.persistence.Persistence;

public class pagesTempFiles {

public static void main(String[] args) {

  System.setProperty("objectdb.temp.page-file", "true");

  String dbA = "objectdb:$objectdb/db/testA.tmp";
  String dbB = "objectdb:$objectdb/db/testB.tmp";
  String dbC = "objectdb:$objectdb/db/testC.tmp";
  String dbD = "objectdb:$objectdb/db/testD.tmp";

  // no page dat file!
  createDB(dbA);
  // first *.dat file (pages0.dat) created
  createDB(dbB);
  // second *.dat file (pages1.dat) created
  createDB(dbC);
  // third *.dat file (pages2.dat) created
  createDB(dbD);
  // fourth *.dat file (pages3.dat) created

  acessDB(dbA);
  // new *.dat file (pages4.dat) created
  acessDB(dbB);
  // new *.dat file (pages5.dat) created
  acessDB(dbC);
  // new *.dat file (pages6.dat) created
  acessDB(dbD);
  // new *.dat file (pages7.dat) created
  acessDB(dbA);
  // new *.dat file (pages8.dat) created
  acessDB(dbA);
  // new *.dat file (pages9.dat) created

  System.out.println("add a breakpoint - and take a look into temp -> "
    + "All Database connections are closed but temp contains 10 *.dat files");

  for (int i = 0; i < 100; i++) {
   acessDB(dbA);
  }

  System.out.println("take a look into temp -> temp contains 110 *.dat files");

}

public static void createDB(String db) {

  EntityManagerFactory emf = Persistence.createEntityManagerFactory(db + ";drop");

  EntityManager em = emf.createEntityManager();
  em.setFlushMode(FlushModeType.AUTO);
  em.getTransaction().begin();

  MyEntity myEntity = new MyEntity();
  myEntity.name = "DBFILE: " + db;
  em.persist(myEntity);

  em.getTransaction().commit();
  em.close();
  emf.close();
}

public static void acessDB(String db) {

  EntityManagerFactory emf = Persistence.createEntityManagerFactory(db);

  EntityManager em = emf.createEntityManager();
  em.setFlushMode(FlushModeType.AUTO);
  em.getTransaction().begin();

  // do anything - access an entity
  MyEntity myEntity = em.find(MyEntity.class, 1);

  em.getTransaction().commit();
  em.close();
  emf.close();
}


@Entity
public static class MyEntity {
  @Basic
  String name;
}

}

Why are the files are not removed?

Thank you for this test case. Please try build 2.6.8_08.

Hello,

thanks, we will try the fix tomorrow.

But again, for us it is importent to understand the behavior of the dat files. Because we released our product with the previos ObjectDB version 2.6.8_06.

What's in the previous version the trigger for the clean up?

It is very important that we can appraise how critical the problem for our application is.

A new temporary file of this type is created every time an ObjectDB database file is created or opened, if objectdb.temp.page-file is enabled.

In the new build (2.6.8_08) the temporary file is deleted when the database file (represented by an EntityManagerFactory) is closed. Unfortunately that was missing until build 2.6.8_07. Since this new feature is only used by you, we had no information about this bug until your report today.

Still, ObjectDB deletes all unused temporary files periodically (also in 2.6.8_06). This happens when a new temporary file is created if the last "all unused temporary files deletion" operation was completed more than 2 minutes ago.

Thank you,

the patch (2.6.8_08) solve the problem.

And also many thanks for the description of the '2 minutes' clean up. With this information we are able to rate the impact of the problem.