Issue #580: Sort of very large objects - out of memory

The attached test is failing with an out of memory exception when running with -Xmx512m

It performs the following actions:

1) Create 5 persist threads which create and persist a large object in a loop 100 times each

2) Create 5 get threads which read 100 objects each in a loop based on object creation time and priority

The message size should be around 1mb so I think there should be around 11mb of "live" data in play at any time. The objectdb cache settings are low in the config - database section is below for reference.

<database>
  <size initial="256kb" resize="256kb" page="2kb" />
  <recovery enabled="true" sync="false" path="." max="128mb" />
  <recording enabled="false" sync="false" path="." mode="write" />
  <locking version-check="true" />
  <processing cache="8mb" max-threads="10" />
  <query-cache results="0mb" programs="500" />
  <extensions drop="temp,tmp,odb" />
  <activation code="..." />
</database>

I've uploaded the heap dump to the ftp site - filename is java_pid7384.zip

The zipped archive wouldnt upload due to its filesize so I've uploaded a rar version instead. This is java_pid7384.rar

Your test demonstrates a limitation of the ObjectDB query processing mechanism in sorting very large objects. This requires a fix, but meanwhile, adding a composite index can solve the problem:

@Entity
    @Index(members={"priority", "createTime"})
    public static class Message {

If there is an index that covers all the fields that are in use in the query and it is much smaller than the complete data (since the huge strings are excluded) the index is used by ObjectDB and query execution becomes faster and consumes less memory.

Execution of a query as the following using indexes:

SELECT m FROM Message m ORDER by priority DESC, createTime

could even be faster if both sort keys were either ascending or descending, since then a range query could just pick the first or last entry in the composite index (instead of scanning the entire index).

Composite indexes with different ASC / DESC components will be supported in future versions. Meanwhile, consider reversing the priority values, in a way that lower values are picked first using:

SELECT m FROM Message m ORDER by priority, createTime