ObjectDB is opening too many sockets

#1

Hi, we are trying to switch from embedded mode to client-server to support database clustering, and we are now facing following issue.

Object DB server is creating too many sockets.
Currently the ObjectDB Server process has 3895 opened files, and from that number 3823 are opened sockets.
I am supposing, that it is because of some mistake, so I would like from you to help me diagnose it instead of increasing system limits.
We are using Ubuntu 18.04.5 LTS and after some time it causes following error and the ObjectDB is restarted automatically:
java[3545]: Too many open files

Few lines from /proc/<ODBServerPID>/fd 
lrwx------ 1 odb odb 64 Mar 19 11:34 995 -> 'socket:[507755621]'
lrwx------ 1 odb odb 64 Mar 19 11:34 996 -> 'socket:[507755622]'
lrwx------ 1 odb odb 64 Mar 19 11:34 997 -> 'socket:[507755623]'
lrwx------ 1 odb odb 64 Mar 19 11:34 998 -> 'socket:[507755624]'
lrwx------ 1 odb odb 64 Mar 19 11:34 999 -> 'socket:[507755625]'

We think, that it does not corresponding with opened DB connections as we are monitoring them, and they are staying under 20.

#2

If you open connections (create EntityManager / PersistenceManager instances) without closing them then the result could be many open sockets.

ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)
#3

I guess I will need more information from you about this :-(

1. I don't know, that is the purpose of that sockets, I don't think, it is related to the connection to much

Let's look on the results of lsof command for ODB Server process, from the 2 lines the first one is TCP connection from explorer.
This disappear when explorer is closed (or File->Close Connection is used), the same way, when our application is running, those connections are not created indefinitely (we are using up to 20), but the socket stays there for longer period of time.

java    17888  odb   14u  IPv6           31439922       0t0      TCP localhost:zabbix-agent->localhost:52842 (ESTABLISHED)
java    17888  odb   65u  sock                0,9       0t0 31404976 protocol: TCPv6

2. I did some testing with running ObjectDB Server, and did all connections from explorer only.
I was working with 2 DB files (firstdb and seconddb) and on the end of the line I wrote number of sockets, which ObjectDB Server has currently in use. I guess it kind of work correctly as the sockets were not increasing indefinitely, but still I don't get, why some times it openes 70 sockets at once, and if I tried same operation again with freshly started ObjectDB Server it did not opened single one.
The results written bellow are steps done on 1 running ObjectDBServer without restarts, also explorer was single instance, just closing and reopening again the connection. So I assume in bad case when opening more database files it can easily pass file systems limits. I had 72 sockets opened at one time using single TCP connection.


The results were following:
01_firstdb open (connected using explorer) 22 sockets open (in total)
02_firstdb closed connection (File->Close connection) 22 sockets open
03_firstdb open second time 22 sockets open
04_firstdb closed connection 22 sockets open
05_seconddb open and closed 22 sockets open
06_seconddb open 22 sockets open
07_seconddb open entity in tree view 0 sockets open
08_firstdb open 0 sockets
08_2_firstdb closed 0 sockets
09_firstdb open 72 sockets open
09_2_firstdb closed 72 sockets open
10_firstdb open 72 sockets open
10_2_firstdb closed 72 sockets open
11_seconddb open 72 sockets open
12_seconddb open entity in tree view 46 sockets open
12_seconddb closed 46 sockets open
13_seconddb open 46 sockets open
14_seconddb open entity in tree mode 47 sockets open
15_firstdb open 47 sockets open
16_firstdb open entity in tree view 47 sockets open
16_2_firstdb closed 47 sockets open
17_seconddb open and also open entity in tree view 1 socket open
18_firstdb open 1 socket open
18_2 firstdb closed 1 socket open
19_firstdb open 1 socket open
19_2firstdb closed
20_firstdb open and open entity in tree view 1 socket open
20_2firstdb closed
21_seconddb open 1 socket open 1 socket open
22_seconddb open entity in tree view 0 sockets open
22_2 seconddb closed
23_firstdb open 0 sockets open
24_seconddb open entity in tree view 1 socket open

#4

This is an interesting test, thanks for sharing. It is difficult to understand and explain some of the data, such as the 22 open sockets on just opening the first database at the beginning.

In principle sockets are associated with connections. Every connection requires an open socket. Connections (and their sockets) are not released immediately when you close them due the connection pool mechanism, which keeps some sockets open for reuse. This is a commonly used optimization in working with databases in general. You can limit the size of the connection pool (the default in ObjectDB is currently 50), by using the JDO method setMaxPool.

ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)
#5

Sorry, I did not get that

You can limit the size of the connection pool (the default in ObjectDB is currently 50), by using the JDO method setMaxPool

How can I do that for ObjectDB server? Shouldn't that be in some config? Or set by command argument?

#6

Actually the connection pool is on the client side and the client holds the sockets. On the server you can set an expiration time for unused connections. See post #2 on this related forum thread.

ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)
#7

I am sorry, but I did not found a way, how to set it up.
I did not found setMaxPool anywhere in documentation.
I was also searching in PersistenceManagerFactory and PersistenceManager with no success.

Can you explain more, how to call setMaxPool?

#8

You can specify the max pool size when you initialize the EntityManagerFactory:

    Map<String, String> properties = new HashMap<>();
    properties.put("javax.jdo.option.MaxPool", "10");
    EntityManagerFactory emf =
        Persistence.createEntityManagerFactory("test.odb", properties);
    EntityManager em = emf.createEntityManager();
ObjectDB Support
ObjectDB - Fast Object Database for Java (JPA/JDO)

Post Reply

To post a reply and/or subscribe to update notifications - please login