This issue was discussed in previous forum posts. We are aware of this problem. ObjectDB includes a mechanism that reduces memory allocation (and uses temporary files instead of RAM) when memory usage is high, but it is far from perfect and can easily fail. It will be addressed with high priority in future versions of ObjectDB.
Note that some operations are more difficult in Java than in other platforms, and recovery from OutOfMemoryError is one of them. It is not always possible to handle this exception and a JVM crash is often the result. So ObjectDB users must use a configuration that will keep sufficient reserve of Java heap space in any scenario.
Regarding a query that returns too many objects, consider limiting the number of result objects.
If you must support very large query result sets, check the temp threshold setting. If you reduce the threshold then ObjectDB will use temporary files instead of RAM more often. Processing complex queries may require using many temporary lists, so unfortunately the value of this parameter is not straightforward and you will have to check its effect on your application.
Reducing the number working threads in the server can also help.
More specific analysis and help will be possible if you can provide a memory heap snapshot.
ObjectDB - Fast Object Database for Java (JPA/JDO)