virtual servers and one file

Hi ObjectDB Team

In the past, we have often found that our customers use virtualized servers where the hard drives are also virtualized. From a certain size of the objectdb file (approx. 1.5 GB), there is a massive drop in performance. We have already seen that queries take more than 20 seconds instead of a few milliseconds.

Copying the database to an explicit server solves the problem, but the desire for virtualization is apparently increasing.

We suspect that Object DB uses random access files access and only part of the virtual file is in the virtual server's cache. Have you ever heard of this? Our only current workaround is to swap out older Entities entirely, but of course that's a crutch. Could there be a way in the future to split the database across multiple files?

Thank you and best regards Arne

Hi,

> We suspect that Object DB uses random access files access and only part of the virtual file is in the virtual server's cache. Have you ever heard of this?

We are not aware of this issue (no previous reports). A drop in performance from a few milliseconds to up to 20 seconds is extreme and requires further investigation. Your suggestion that partial caching of the database file is the reason seems reasonable and maybe this could be solved with a different virtualization configuration. If the physical machine has sufficient RAM for caching the entire database but the virtualization space is limited then obviously this can cause a performance drop.

ObjectDB 2.x does use RandomAccessFile. ObjectDB 3.0, which is currently under development switched to memory mapped files, which are usually more efficient. It is unclear, however, if this will solve the issue, as virtualization might kill the ability to use the full resources of the physical machine either way.

> Could there be a way in the future to split the database across multiple files?

This is indeed under consideration but could you please provide more information regarding how breaking a database into multiple files, for example one file per entity class, may help with this particular issue?

Thanks for quick respond

Actually we rely on customer stories and what we see in remote sessions. As far as I understand it is not the problem of less memory of the physical or virtual machine. It seems to me (but only guesswork) that there is a (maybe configurable) limit per file in the virtualized harddisk.  As of my understanding each virtual file is (depending on the size) not completly loadet in memory and the virtualisation software tries to make a prediction for the next access, which should be easy for linear streaming and difficult for random access files.

> ... but could you please provide more information regarding how breaking a database into multiple files

If there is a limitation per file (also guessed) then splitting in several files might solve that problem, wether in a file per entity or just depending on continous primary key.

I will try to get an access to an virtualized server and setup a testcase. My idea is to workout the treshhold value and then I try to open several databases in parallel which in sum exceeds the treshhold and compare the response time. I don't know the data structure of the objectdb file to force access to distant file areas. Maybe depending on the creation date?

best regards

Arne

Thank you for the information. Further details of your investigation, as well information regarding the specific virtualization software and configuration involved, will be welcome.

> I don't know the data structure of the objectdb file to force access to distant file areas. Maybe depending on the creation date?

It depends on your schema. Assuming you want to simulate random access (rather than a complete scan) you can retrieve objects with random primary keys (you can keep all the primary keys in memory or use a simple test database with a continuous range of simple numeric IDs). If you want to cover indexes as well you will have to repeat this operation with every index (i.e. retrieve object by indexed values), but for a simple test you can also have a database with no indexes.