Performance problem

Hi, we have a strange performance problem with ObjectDB 2.5.3_01 and JBoss 7.1.1.
We have two ObjectDB databases on one ObjectDB server. Production database is running 5-10 times slower than test database. Their size and number of records are almost identical. When running production database in separate test environment its speed is very good.
We did a performance analysis on our 2 linux servers one running JBOSS and other running ObjectDB.
- CPU utilization of JBOSS server is max 5-10% (per core)
- CPU utilization of JBOSS server is 80-150% (this is per core)
Now is the interesting part, when running a query from ObjectDB explorer CPU utilization is minimal ie. 1%
Running a query from ObjectDB explorer on complete database which has 12000 records takes 30ms which we think is very good.
In our web application this behaviour gives aprox 10-13 s of refreshing a data table vs 2 second with test database.

Does anybody have any idea what could be wrong.

The details are unclear. For example:

- CPU utilization of JBOSS server is max 5-10% (per core)
- CPU utilization of JBOSS server is 80-150% (this is per core)

Please provide also details of using:

1. The test database on the production server from the Explorer.
2. The production database on the production server from the Explorer.
3. The test database on the production server from JBoss.
4. The production database on the production server from JBoss.
5. The test database on the production server from a simple console test program.
6. The production database on the production server from a simple console test program.

and if possible, all the above also on the test server.

Hi, we have collected results. Results from console program reveal some other interesting facts

The test database on the production server from the Explorer.

• Result 1: aprox 20000 records in 30ms
• Result 2: 14000 Document records in 30ms

The production database on the production server from the Explorer.

• Result 1: 12000 records in 30ms
• Result 2: 2200 Document records in 11ms

The test database on the production server from JBoss.

• Result - 3 seconds for 50 to 1000 results (this is from JBOSS Vaadin application

The production database on the production server from JBoss

• Result - 13 seconds for 50 to 1000 results (from JBOSS Vaadin application)

CONSOLE TEST PROGRAM

The production database on the production server from a simple console test program.

• Result 1 - Get all Documents 26 sec, 2200
• Result 2 - Get 100 Documents also 26 sec
• Result 3 - Get 60 Documents 26 sec
• Result 3 - Get 50 Documents 58ms

The test database on the production server from a simple console test program.

• Result 1 - Get all Documents 4 sec, 14000 results
• Result 2 - Get 100 Documents 3,4 sec
• Result 3 - Get 60 Documents 3,4 sec
• Result 3 - Get 50 Documents 58 ms

Here we see some very interesting results, it seems that number of query results have large effect on speed, especially if we exceed certain threshold.

Regards.

hi, additional info, 

First version of query gives result in 50ms for first 40 records from database but second version which queries 40+ records gives 19 seconds. We pinpointed that from he 53 record performances degrades significantly. On other queries threshold is different, possible due the size of the results (probably related to the number of related objects)

First version of code:

    EntityManagerFactory emf =
        Persistence.createEntityManagerFactory(
            "objectdb://10.10.10.14/E_POLICIJA.odb;user=admin;password=admin");
                //$NON-NLS-1$

    em = emf.createEntityManager();

    long startTime;
    long endTime;

    startTime = System.currentTimeMillis();

    int i = 0;
    while(i < 40){
        TypedQuery<AktImpl> queryAkt =
            em.createQuery("SELECT e FROM AktImpl e", AktImpl.class);

        queryAkt.setFirstResult(i);
        queryAkt.setMaxResults(20);
        queryAkt.getResultList();
        i += 20;
    }

    endTime = System.currentTimeMillis();
    System.out.println((endTime - startTime));

Second version of code:

    EntityManagerFactory emf =
        Persistence.createEntityManagerFactory(
            "objectdb://10.10.10.14/E_POLICIJA.odb;user=admin;password=admin");
                //$NON-NLS-1$

    em = emf.createEntityManager();

    long startTime;
    long endTime;

    startTime = System.currentTimeMillis();

    int i = 0;
    while(i < 60){
        TypedQuery<AktImpl> queryAkt =
            em.createQuery("SELECT e FROM AktImpl e", AktImpl.class);

        queryAkt.setFirstResult(i);
        queryAkt.setMaxResults(20);
        queryAkt.getResultList();
        i += 20;
    }

    endTime = System.currentTimeMillis();
    System.out.println((endTime - startTime));

We created support ticket and updated our database file.

regards.

Running both queries without your entity classes (which we do not have) results in a quick response time of about 100ms for each of them.

Obviously the problem is not in the execution of that simple query, but probably due to a massive EAGER fetch that follows the query execution. That eager fetch doesn't happen for example when the query is run the Explorer, and in other cases in which the entity class with the EAGER definition is unavailable.

You found that the problem starts with result #53. If you run the query in the Explorer you can easily see that results #1 - #52 are AktImpl instances and that result #53 is PartolniNalogImpl instance.

So apparently loading of that PartolniNalogImpl instance takes too long.

Please check that class. You probably have EAGER definitions that cause a massive load of objects from the database. You may find out that by a recursive EAGER fetch you enforce loading of the entire database.

Thanks very much for your help. We will try to optimize the usage of EAGER parameter in most of our queries, we did work without it in the beginning but application was slower.

In one of our queries EAGER must remain, it is for the Employee query which we make every time we login or need information for a particular user. For example we have a class user which has attributes User name, Password and ID, and that class has relationships with few other class (5-10) that store info about Organization, Employee type, permissions (we use Appache Shiro and permission data are stored in ObjectDB)

We need all this data and running a query for a single user takes 20 seconds, running a query only against class Users for all users in the database (300) takes approx. 20 milliseconds (this obviously the EAGER usage case)

I have attached complete application for you to try.

Last night we also become aware of the "#53 is PartolniNalogImpl instance" issue. We tried earlier to modify this relationship but it breaks the database. We reported this problem as a schema change problem but will try again to fix it.

regards.

Usually when EAGER causes performance problems it is due to its recursive nature.

You wrote that there are 5-10 entities referenced from User but you should check the entire closure, i.e. the complete graph of objects that are reachable from a User instance through relationships that are defined as EAGER. If this graph of objects is too large, you have to make the decision how to reduce its size, considering also your application needs.

HI

only 4 clases are set to EAGER which are in relations to USER class. Our main question after investigating application code is whay does it take 20 seconds for the Entity manager to get all users from the USER class but runing the same query from Explorer takes 20ms. APROX 400 user are in the system.

Could you please check our sample application we sent.

Regards.

The answer to your question is that the Explorer doesn't apply EAGER fetch.

You wrote that there are only 4 EAGER relationships in User. Did you check which of them is related to the problem? This should be a very easy check, just try the same query with different combinations of EAGER / LAZY for these relationships. After finding the problematic relationship you can go ahead and analyze the target class in the same way, recursively, until you find the problematic relationship.

Please understand that we can only offer a limited support in this forum, and unfortunately we cannot get into the fine details of your project. An attempt to compile your project failed because the compiler rejected some of the non English characters. Anyway, after identifying the problem as a massive eager fetch you should be able to find the problematic relationship.

Thank  you very much for your help. We understand that you can provide limited support.

This morning we identified the most probable cause of the performance issue.
The 4 classes we mentioned are not the problem, we have two  relation between class USER and class MESSAGES, one from USER to MESSAGES other from MESSAGES to USER, both are 1 to many. When we removed one of them we got big performance improvements.
We have two related concerns with this
1. we have similar relations through our model and they don’t seem to cause problems yet.
2. We reported a issue in forum “Object Relations Error after Schema Change”, the relations we found are the problem are the same one mentioned in this issue. We solved this problem temporary by migrating all data to new database.
We would appreciate if you could investigate why we have errors after Schema Change. We will provide a detailed testing sample.
Can we assume that ObjectDB by design should be able to handle this kind of relations or there are some guideline on how to do this.

Regards.

and thank you again for your help.

A circular eager relationship can be very problematic.

ObjectDB is designed to be very flexible with schema changes, as explained in this manual page. But this forum thread is about performance, so now that the performance problem was found, this thread should be closed, and schema issues, if any, may be discusses in another forum thread.

Again, thanks you for you help.