EntityManagerFactory objects being locked during EntityManager creation

Issue Details :
We've been observing lately an issue with ObjectDB wherein the EntityManagerFactory objects are being locked during EntityManager object creation. 
Please refer the below stacktrace. Complete thread dump is also attached to the ticket for your reference.

"https-openssl-nio-8443-exec-158" - Thread t@356
   java.lang.Thread.State: BLOCKED
        at com.objectdb.o.OMF.w(OMF.java:674)
        - waiting to lock <76bf096d> (a com.objectdb.jpa.EMF) owned by "https-openssl-nio-8443-exec-132" t@330
        at com.objectdb.jpa.EMF.createEntityManager(EMF.java:170)
        at com.adheris.cap.capfoundation.objectdb.AbstractRetailerObjectDatabase.createEntityManager(AbstractRetailerObjectDatabase.java:96)
        at com.adheris.cap.consumerframework.database.ReprintCacheDatabase.createDao(ReprintCacheDatabase.java:60)
        
        
"https-openssl-nio-8443-exec-132" - Thread t@330
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
        - locked <19e9ce09> (a java.io.BufferedInputStream)
        at com.objectdb.o.NTS.n(NTS.java:324)
        at com.objectdb.o.NTS.m(NTS.java:284)
        at com.objectdb.o.CLS.y(CLS.java:350)
        at com.objectdb.o.CLS.z(CLS.java:283)
        at com.objectdb.o.CLS.w(CLS.java:172)
        at com.objectdb.o.CLS.x(CLS.java:141)
        at com.objectdb.o.CST.<init>(CST.java:79)
        at com.objectdb.o.CSF.Z0(CSF.java:90)
        at com.objectdb.o.OMF.w(OMF.java:707)
        - locked <76bf096d> (a com.objectdb.jpa.EMF)
        at com.objectdb.jpa.EMF.createEntityManager(EMF.java:170)
        at com.adheris.cap.capfoundation.objectdb.AbstractRetailerObjectDatabase.createEntityManager(AbstractRetailerObjectDatabase.java:96)
        at com.adheris.cap.consumerframework.database.ReprintCacheDatabase.createDao(ReprintCacheDatabase.java:60)
        at com.adheris.cap.consumerframework.database.ReprintCacheDatabase.getDao(ReprintCacheDatabase.java:49)

Impact :
This issue is causing thread spikes on our application servers and also on our ObjectDB servers which is in turn causing performance degradation during this spike interval.

We would like to understand any underlying EntityManagerFactory locking mechanism to get deeper insight into this issue.

Note: We have tried to create a sample program to recreate this issue, but were not successful in doing so.

Thanks in anticipation.

> We've been observing lately an issue with ObjectDB wherein the EntityManagerFactory objects are being locked during EntityManager object creation. 

This is the normal behavior of ObjectDB. Creation operations of EntityManager instances were always serial because they lock the ower EntityManagerFactory. Changing this behavior is not trival, as it would change the order of locks, which must be preserved to avoid deadlocks (although this would be considered as a possible future improvement).

Usually this lock should not be a problem, because creating an EntityManager should be fast. In this specific case, the locking thread's stack trace indictes a slower operation of creating a new connection to the server.

Possible directions you may consider:

1. Increasing the size of the connection pool to reduce the need for new connections.
2. managing your own pool of unused EntityManager instances (the ObjectDB pool contains client-server connections, which are only parts of the EntityManager instances).
3. Using multiple EntityManagerFactory Instances.