Server connection management

The server does not appear to release broken connections. My typical development pattern is to keep reloading my war file in tomcat until the permgen space runs out, and then I kill and restart tomcat. After a few days of this, the server stops responding, and I see the following in the server logs:

[odb:server:fatal] com.objectdb.o.UserException: Too many open connections (100)
        at com.objectdb.o.MSG.d(MSG.java:61)
        at com.objectdb.o.SMR.O(SMR.java:469)
        at com.objectdb.o.SMR.N(SMR.java:406)
        at com.objectdb.o.SMR.L(SMR.java:271)
        at com.objectdb.o.SMR.n(SMR.java:166)
        at com.objectdb.o.TOL.run(TOL.java:115)
        at com.objectdb.Server.runCommand(Server.java:200)
        at com.objectdb.Server.run(Server.java:107)
        at com.objectdb.Server.main(Server.java:66)

It would be useful to have a configuration setting that would tell the server to close connections after a period of inactivity.

Also, how does the client deal with broken connections or server restarts. Does the client have an internal connection pool that checks the health of a connection and reconnects if needed? It would be useful to have configuration settings to control the connection pool (min/max connections, etc...).

Thanks,

Carl

The following highlights should answer your questions.

Broken Connections on the Server

The server handles every connection using a dedicated thread. This thread uses blocking read to listen to client requests. When the connection is closed on the client side (including on client side JVM shutdown) - the server gets an EOF exception and automatically closes the thread and the socket. I just checked now and it seems to work well.

Broken Connections on the Client

If connection fails on the client side - the client automatically tries to reconnect to the server and this is transparent to the application. Replication (introduced in ObjectDB 2.1) enables also specifying replicated servers that can be connected automatically by the client (in read only mode) in case the main (master) server is unreachable. The client automatically switches back to the master server in a few seconds after it becomes accessible again.

Inactivity Timeout

You can set the inactivity timeout in seconds and it affects both the client and the server side:

  <general>
    ...
    <network inactivity-timeout="3600" />
    ...
  </general>

Connection Pool (Client Side)

You should be able to set the connection pool size by a property in persistence.xml:

  <properties>
    <property name="javax.jdo.option.MaxPool" value="10"/>
  </properties>

The property can also be specified in properties map when instantiating the EntityManagerFactory.

Currently the default pool size is 50.

This is very useful information.

I do, however, get the exception reported above at least once a week. When you say that objectdb auto-detects the closed connection on JVM shutdown, does that include JVM failure (out of permgen space) and forced quit?

I believe that if the client JVM process terminates - the operating system should close the socket and then the server gets an EOF notification. I just checked it on Windows XP. But it might depends on the operating system. Maybe a firewall also has some effect.

Anyway, try using the inactivity timeout that probably may solve the problem.

I'm getting the same error with a test that I'm trying to run but each opened connection should be closed properly, i.e. no broken connections.

The test runs a loop which saves an object and then queries it back to check the save has been successful. The persist and save are done via different EntityManagers with the transaction commited and manager closed at the end of each action.

Is there anything further which needs to be done in order for the open connection to be released?

The exception is:

[odb:server:fatal] com.objectdb.o.UserException: Too many open connections (100)
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.SMR.O(SMR.java:469)
at com.objectdb.o.SMR.N(SMR.java:406)
at com.objectdb.o.SMR.L(SMR.java:271)
at com.objectdb.o.SMR.n(SMR.java:166)
at com.objectdb.o.TOL.run(TOL.java:115)
at com.objectdb.Server.runCommand(Server.java:194)
at com.objectdb.Server.run(Server.java:107)
at com.objectdb.Server.main(Server.java:66)

I just ran a quick test and couldn't reproduce the problem:

package com.objectdb.test.bug;

import javax.persistence.*;

public final class ConnectionTest
{
    public static void main(String[] args)
    {
        EntityManagerFactory emf =
            Persistence.createEntityManagerFactory(
                "objectdb://localhost/test.odb;user=admin;password=admin");
       
        for (int i = 1; i <= 1000; i++)
        {
            EntityManager em = emf.createEntityManager();
            em.getTransaction().begin();
            em.persist(new MyEntity(i));
            em.getTransaction().commit();
            em.close();
            System.out.println(i);
        }

        emf.close();
    }

    @Entity
    public static final class MyEntity {
        @Id @GeneratedValue long id;
        int value;
        MyEntity(int value) {
            this.value = value;
        }
    }
}

Maybe if you post your test it would help in finding the problem.

Apologies - this was my fault.

Your example forced me back to basics which worked fine and eventually led me to a connection which was being left open on the client side. Thanks for your help!

No problem. Note that the limit (100) can be removed by changing the configuration to 0, but as you can see sometimes the limit is useful in revealing problems.

It would be useful to have a query string that can return:

• # of open connections to server
• for each open connection, see:
  • database name that it is connected to
  • Client username and IP address

Also, a command to force any or all connections closed would be great.

Good ideas.

I added them to the Monitoring and Performance counters feature request.

You may subscribe to this issue and suggest additional ideas.