Intermittent Server Shutdowns Due to Out of Memory (OOM) Issues

Dear ObjectDB Community,

I am encountering a recurring issue where the ObjectDB server is frequently being terminated by the Linux Out-Of-Memory (OOM) Killer. Despite allocating substantial memory resources to the server (90GB total memory on the VM), the java process hosting ObjectDB is being killed due to exceeding memory limits.

Here are the specifics of my setup and the issue:

• Server Configuration:

  • ObjectDB Version: 2.8.8
  • VM Total Memory: 90GB
  • ObjectDB Configuration:
    • Processing Cache: 32GB (<processing cache="32000mb" />)
    • Other settings: Stated below

• Problem Description:

  • The ObjectDB server starts normally but gets terminated intermittently.
  • The dmesg log indicates that the process is killed by the OOM Killer.
  • The relevant dmesg output line is: Out of memory: Killed process 149910 (java) total-vm:97947796kB, anon-rss:83478100kB.
  • System resource checks confirm ample free memory and disk space before the server is started.

I have already tried reducing the processing cache and monitoring system resource usage, but the problem persists.

Could you provide insights or recommendations on how to configure ObjectDB and the hosting environment to prevent these OOM terminations? Are there specific JVM settings or ObjectDB configurations that I should look into to manage memory usage more effectively?

Thank you in advance for your assistance.

<!-- ObjectDB Configuration -->

<objectdb>

    <general>
        <temp path="$temp" threshold="4000mb" />
        <network inactivity-timeout="0" />
        <url-history size="100" user="true" password="true" />
        <log path="$objectdb/log/" max="1600mb" stdout="false" stderr="false" />
        <log-archive path="$objectdb/log/archive/" retain="90" />
        <logger name="*" level="info" />
               <!--
                <logger name="query" level="debug" />
               -->
    </general>

    <database>
        <size initial="1000mb" resize="1000mb" page="20kb" />
        <recovery enabled="true" sync="false" path="." max="2000mb" />
        <recording enabled="false" sync="false" path="." mode="write" />
        <locking version-check="true" />
        <processing cache="32000mb" max-threads="720" />
        <query-cache results="160mb" programs="2000" />
        <extensions drop="temp,tmp" memory="mem" />
        <activation code="AN52-7Y7H-GR1Y-VDYT-VF84" />
    </database>    

    <entities>
        <enhancement agent="true" reflection="warning" />
        <cache ref="weak" level2="0" />
        <persist serialization="false" />
        <cascade-persist always="auto" on-persist="false" on-commit="true" />
        <dirty-tracking arrays="false" />
    </entities>

    <schema>
    </schema>

    <server>
        <connection port="6136" max="0" />
        <data path="$objectdb/db" />
        <!--
        <replication url="objectdb://localhost/test.odb;user=admin;password=admin" />
        -->
    </server>

  
    
    <ssl enabled="false">
        <server-keystore path="$objectdb/ssl/server-kstore" password="pwd" />
        <client-truststore path="$objectdb/ssl/client-tstore" password="pwd" />
    </ssl>

</objectdb>

One possible cause could be clients that do not close their connections to the server (i.e. closing the EntityManagerFactory instances). However, any attempt to understand the issue without more data is a guesswork. Please take a heap dump of the server when RAM usage is high and report which classes and objects consume unexpected RAM resources.