EntityManagerFactory Fails To Swtich Over To Slave When Master Is Unavailable

I have a simple task to validate the sample code will switch over to the slave database when the master is unavailable. I simulate the master unavailable by stopping the master server; which generates the following exception below. What am I doing wrong? It appears the connection manager fails to try the second URL in the list: objectdb://10.9.2.15:9998//10.9.2.15:9999/test.odb;user=admin;password=password

Thank you in advance.

[ObjectDB 2.5.0_04] javax.persistence.PersistenceException
Failed to connect to server 10.9.2.15:9999 (Connection refused: connect) (error 522)
at com.objectdb.jpa.EMF.createEntityManager(EMF.java:253)
at tutorial.Main.main(Main.java:22)
Caused by: com.objectdb.o.UserException: Failed to connect to server 10.9.2.15:9999 (Connection refused: connect)
at com.objectdb.o.MSG.d(MSG.java:74)
at com.objectdb.o.CLS.M(CLS.java:153)
at com.objectdb.o.CST.<init>(CST.java:65)
at com.objectdb.o.CSF.UK(CSF.java:87)
at com.objectdb.o.OMF.al(OMF.java:688)
at com.objectdb.jpa.EMF.createEntityManager(EMF.java:250)
... 1 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.objectdb.o.CLS.M(CLS.java:122)
... 5 more

 

Master Configuration

<objectdb>

        <general>
                <temp path="$temp" threshold="64mb" />
                <network inactivity-timeout="5" />
                <url-history size="50" user="true" password="true" />
                <log path="$objectdb/log/" max="8mb" stdout="true" stderr="true" />
                <log-archive path="$objectdb/log/archive/" retain="90" />
                <logger name="*" level="info" />
        </general>

        <database>
                <size initial="256kb" resize="256kb" page="2kb" />
                <recovery enabled="true" sync="true" path="." max="128mb" />
                <recording enabled="true" sync="true" path="." mode="write" />
                <locking version-check="true" />
                <processing cache="64mb" max-threads="10" />
                <query-cache results="32mb" programs="500" />
                <extensions drop="temp,tmp" />
                <!-- <activation code="XXXX-XXXX-XXXX-XXXX-XXXX" /> -->
        </database>

        <entities>
                <enhancement agent="false" reflection="warning" />
                <cache ref="weak" level2="0" />
                <persist serialization="false" />
                <cascade-persist always="auto" on-persist="false" on-commit="true" />
                <dirty-tracking arrays="false" />
        </entities>

        <schema>
        </schema>

        <server>
                <connection port="9999" max="100" />
                <data path="$objectdb/db" />
        </server>

        <users>
                <user username="admin" password="password">
                        <dir path="/" permissions="access,modify,create,delete" />
                </user>
                <user username="$default" password="$$$###">
                        <dir path="/$user/" permissions="access,modify,create,delete">
                                <quota directories="5" files="20" disk-space="5mb" />
                        </dir>
                </user>
                <user username="user1" password="password" />
        </users>

        <ssl enabled="false">
                <server-keystore path="$objectdb/ssl/server-kstore" password="pwd" />
                <client-truststore path="$objectdb/ssl/client-tstore" password="pwd" />
        </ssl>

</objectdb>

Slave Configuration

<objectdb>

        <general>
                <temp path="$temp" threshold="64mb" />
                <network inactivity-timeout="5" />
                <url-history size="50" user="true" password="true" />
                <log path="$objectdb/log/" max="8mb" stdout="true" stderr="true" />
                <log-archive path="$objectdb/log/archive/" retain="90" />
                <logger name="*" level="info" />
        </general>

        <database>
                <size initial="256kb" resize="256kb" page="2kb" />
                <recovery enabled="true" sync="false" path="." max="128mb" />
                <recording enabled="false" sync="false" path="." mode="all" />
                <locking version-check="true" />
                <processing cache="64mb" max-threads="10" />
                <query-cache results="32mb" programs="500" />
                <extensions drop="temp,tmp" />
                <!-- <activation code="XXXX-XXXX-XXXX-XXXX-XXXX" /> -->
        </database>

        <entities>
                <enhancement agent="false" reflection="warning" />
                <cache ref="weak" level2="0" />
                <persist serialization="false" />
                <cascade-persist always="auto" on-persist="false" on-commit="true" />
                <dirty-tracking arrays="false" />
        </entities>

        <schema>
        </schema>

        <server>
                <connection port="9998" max="100" />
                <data path="$objectdb/db" />
                <replication url="objectdb://localhost:9999/test.odb;user=admin;password=password" />
        </server>

        <users>
                <user username="admin" password="password">
                        <dir path="/" permissions="access,modify,create,delete" />
                </user>
                <user username="$default" password="$$$###">
                        <dir path="/$user/" permissions="access,modify,create,delete">
                                <quota directories="5" files="20" disk-space="5mb" />
                        </dir>
                </user>
                <user username="user1" password="password" />
        </users>

        <ssl enabled="false">
                <server-keystore path="$objectdb/ssl/server-kstore" password="pwd" />
                <client-truststore path="$objectdb/ssl/client-tstore" password="pwd" />
        </ssl>

</objectdb>

 

Test Code

package tutorial;

import javax.persistence.*;
import java.util.*;

public class Main {

private static boolean WRITE_DATA_TO_DATABASE = false;

public static void main(String[] args) {
  EntityManagerFactory emf = null;
  EntityManager em = null;
  try {
   emf = Persistence
     .createEntityManagerFactory("objectdb://10.9.2.15:9999/test.odb;user=admin;password=password|"
       + "objectdb://10.9.2.15:9998//10.9.2.15:9999/test.odb;user=admin;password=password");
   for (int i = 0; i < 100; i++) {
   
    boolean flag = true;
    while(flag) {
     try {
      em = emf.createEntityManager();
      flag = false;
     }catch(Exception e) {
      e.printStackTrace();
     }
    }
   
    queryDatabase(em);
    System.out.println("COUNT = " + i);
    em.close();
    try {
     Thread.sleep(1500);
    } catch (InterruptedException e) {
     // noop
    }
   }
  } catch (Exception e) {
   e.printStackTrace();
  } finally {
   emf.close();
  }
}

private static void queryDatabase(EntityManager em) {
  if (WRITE_DATA_TO_DATABASE) {
   em.getTransaction().begin();
   for (int i = 0; i < 1000; i++) {
    Point p = new Point(i, i);
    em.persist(p);
   }
   em.getTransaction().commit();
  }

  // Retrieve all the Point objects from the database:
  TypedQuery<Point> query = em.createQuery("SELECT p FROM Point p",
    Point.class);
  List<Point> results = query.getResultList();
  int count = 0;
  for (Point p : results) {
   System.out.println(p);
   if(count++>5) {
    break;
   }
  }

  // Find the number of Point objects in the database:
  Query q1 = em.createQuery("SELECT COUNT(p) FROM Point p");
  System.out.println("Total Points: " + q1.getSingleResult());

  // Find the average X value:
  Query q2 = em.createQuery("SELECT AVG(p.x) FROM Point p");
  System.out.println("Average X: " + q2.getSingleResult());

}

}

 

package tutorial;

import java.io.Serializable;
import javax.persistence.*;


@Entity
public class Point implements Serializable {
    private static final long serialVersionUID = 1L;

    @Id @GeneratedValue
    private long id;

    private int x;
    private int y;

    public Point() {
    }

    Point(int x, int y) {
        this.x = x;
        this.y = y;
    }

    public Long getId() {
        return id;
    }

    public int getX() {
         return x;
    }

    public int getY() {
         return y;
    }

    @Override
    public String toString() {
        return String.format("(%d, %d)", this.x, this.y);
    }
}

Slightly modified code to view the exception message.

package tutorial;

import javax.persistence.*;
import java.util.*;

public class Main {

private static boolean WRITE_DATA_TO_DATABASE = false;

public static void main(String[] args) {
  EntityManagerFactory emf = null;
  EntityManager em = null;
  try {
   emf = Persistence
     .createEntityManagerFactory("objectdb://10.9.2.15:9999/test.odb;user=admin;password=password|"
       + "objectdb://10.9.2.15:9998//10.9.2.15:9999/test.odb;user=admin;password=password");
   for (int i = 0; i < 100; i++) {
   
    boolean flag = true;
    while(flag) {
     try {
      em = emf.createEntityManager();
      flag = false;
     }catch(Exception e) {
      e.printStackTrace();
     }
    }
   
    try {
     queryDatabase(em);
     System.out.println("COUNT = " + i);
     em.close();
    }catch(Exception e) {
     e.printStackTrace();
    }
   
   
    try {
     Thread.sleep(1500);
    } catch (InterruptedException e) {
     // noop
    }
   }
  } catch (Exception e) {
   e.printStackTrace();
  } finally {
   emf.close();
  }
}

private static void queryDatabase(EntityManager em) {
  if (WRITE_DATA_TO_DATABASE) {
   em.getTransaction().begin();
   for (int i = 0; i < 1000; i++) {
    Point p = new Point(i, i);
    em.persist(p);
   }
   em.getTransaction().commit();
  }

  // Retrieve all the Point objects from the database:
  TypedQuery<Point> query = em.createQuery("SELECT p FROM Point p",
    Point.class);
  List<Point> results = query.getResultList();
  int count = 0;
  for (Point p : results) {
   System.out.println(p);
   if(count++>5) {
    break;
   }
  }

  // Find the number of Point objects in the database:
  Query q1 = em.createQuery("SELECT COUNT(p) FROM Point p");
  System.out.println("Total Points: " + q1.getSingleResult());

  // Find the average X value:
  Query q2 = em.createQuery("SELECT AVG(p.x) FROM Point p");
  System.out.println("Average X: " + q2.getSingleResult());

}

}

Your test and configuration files are fine. It was a problem in ObjectDB.

Please try build 2.5.0_05 that should fix it. Thank you for this report.

Hello. I'm not seeing the URL switching fix in build 2.5.0_05.  My test is:

1.) Start ObjectDB server as master on port 10,000.

2.) On same server, start ObjectDB server as slave on port 10,001.

3.) Run test application (Main.java) with "private static boolean WRITE_DATA_TO_DATABASE = true;" to insert a few records. Stop Main.java.

4.) Re-run (Main.java) with WRITE_DATA_TO_DATABASE = false;

Sample output:

COUNT = 14
Entity manager is: ObjectManager of /10.9.2.15:10000 [2013-05-08 11:45:07 -> 2013-05-08 11:45:07] - 15
(0, 0)
(1, 1)
(2, 2)
(3, 3)
(4, 4)
(5, 5)
(6, 6)
Total Points: 5000
Average X: 499.5

5.) Shut down master on port 10,000. I see the following stack trace. Notice EntityManagerFactory switched over to the slave; however, the connection is closed.

COUNT = 8
Entity manager is: ObjectManager of /10.9.2.15:10001 [2013-05-08 11:46:08 -> 2013-05-08 11:46:26] - 9
[ObjectDB 2.5.0_05] javax.persistence.PersistenceException
Connection is closed (error 526)
at com.objectdb.jpa.JpaQuery.getResultList(JpaQuery.java:695)
at tutorial.Main.queryDatabase(Main.java:75)
at tutorial.Main.main(Main.java:39)
Caused by: com.objectdb.o.UserException: Connection is closed
at com.objectdb.o.MSG.d(MSG.java:74)
at com.objectdb.o.CLS.R(CLS.java:330)
at com.objectdb.o.CLS.w(CLS.java:245)
at com.objectdb.o.CLS.v(CLS.java:229)
at com.objectdb.o.CST.U6(CST.java:498)
at com.objectdb.o.QRR.g(QRR.java:244)
at com.objectdb.o.QRR.f(QRR.java:153)
at com.objectdb.jpa.JpaQuery.getResultList(JpaQuery.java:686)
... 2 more

6.) If I re-start the master (10,000), I connect successfully and see the following results. Notice, now I'm able to connect to the slave database (10,001) and get results, just before EntityManagerFactory switches back over to the master (10,000).

COUNT = 9
Entity manager is: ObjectManager of /10.9.2.15:10001 [2013-05-08 11:49:08 -> 2013-05-08 11:49:27] - 10
(0, 0)
(1, 1)
(2, 2)
(3, 3)
(4, 4)
(5, 5)
(6, 6)
Total Points: 5000
Average X: 499.5
COUNT = 10
Entity manager is: ObjectManager of /10.9.2.15:10000 [2013-05-08 11:49:08 -> 2013-05-08 11:49:29] - 11
(0, 0)
(1, 1)
(2, 2)
(3, 3)
(4, 4)
(5, 5)
(6, 6)
Total Points: 5000
Average X: 499.5

7.) Also, if I start both servers, and change the URL to connect only to the slave, I get the following exception.

COUNT = 0
Failed to create entity manager.
COUNT = 1
[ObjectDB 2.5.0_05] javax.persistence.PersistenceException
Replicated database objectdb://10.9.2.15:10000/test.odb;user=admin;password=password is unavaliable (error 538)
at com.objectdb.jpa.EMF.createEntityManager(EMF.java:253)
at tutorial.Main.main(Main.java:31)
Caused by: com.objectdb.o.UserException: Replicated database objectdb://10.9.2.15:10000/test.odb;user=admin;password=password is unavaliable
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.SMR.ac(SMR.java:721)
at com.objectdb.o.SHN.Y(SHN.java:251)
at com.objectdb.o.SHN.K(SHN.java:123)
at com.objectdb.o.HND.run(HND.java:132)
at java.lang.Thread.run(Thread.java:662)
Failed to create entity manager.

Code modification to Main.java is below to connect to the slave database.

emf = Persistence
    .createEntityManagerFactory("objectdb://10.9.2.15:10001//10.9.2.15:10000/test.odb;user=admin;password=password");

 

8.) I also noticed, if EntityManagerFactory is created while the master (10,000) is already down, we fail to connect to the slave (10,001).

COUNT = 0
Failed to create entity manager.
COUNT = 1
[ObjectDB 2.5.0_05] javax.persistence.PersistenceException
Failed to connect to server 10.9.2.15:10000 (Connection refused: connect) (error 522)
at com.objectdb.jpa.EMF.createEntityManager(EMF.java:253)
at tutorial.Main.main(Main.java:33)
Caused by: com.objectdb.o.UserException: Failed to connect to server 10.9.2.15:10000 (Connection refused: connect)
at com.objectdb.o.MSG.d(MSG.java:74)
at com.objectdb.o.CLS.M(CLS.java:153)
at com.objectdb.o.CST.<init>(CST.java:65)
at com.objectdb.o.CSF.UK(CSF.java:87)
at com.objectdb.o.OMF.am(OMF.java:743)
at com.objectdb.jpa.EMF.am(EMF.java:232)
at com.objectdb.o.OMF.al(OMF.java:684)
at com.objectdb.jpa.EMF.createEntityManager(EMF.java:250)
... 1 more
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.objectdb.o.CLS.M(CLS.java:122)
... 7 more
Failed to create entity manager.

 

I've attached the objectdb.conf files for both master and slave.

Other output:

master db directory

/data/objectdb-2.5.0_05/db> dir
-rw-r--r-- 1 pcrx pcrx 786432 2013-05-08 11:58 test.odb
drwxr-xr-x 2 pcrx pcrx   4096 2013-05-08 11:58 test.odr

slave db directory

/data/objectdb-2.5.0_05-2/db/$replication> dir
-rw-r--r-- 1 pcrx pcrx 786432 2013-05-08 11:51 test.odb
-rw-r--r-- 1 pcrx pcrx     20 2013-05-08 11:51 test.odb$

 

 

Thank you for this revised test.

Unfortunately I couldn't reproduce the problems in steps 5 yet. Main switched successfully from the master to the slave. I got error messages from the slave server that failed to connect to the master server (which is down), but not the errors that you got. Please check the steps. Maybe there is an additional operation that I have to do in order to see the problem.

In step 7, I could connect to the slave server directly (using the emf creation in comment in your test) after stopping the master server with no errors, but I got the reported error message if the slave is started when the master is down. This should be fixed. Please check if this is actually what you get.

I can confirm that the problem at step 8 exists as described. Switching urls works only when the initial connection is to the master server, and this should be fixed.

Update: Issue #1140 has been filed to cover the limitations of composite url and automatic server switch.