Issue #321: Once served to JSF page via @EJB query bean, many list fields are null (but same query ok after fresh persist in @PostConstruct)

I have marked this as CRITICAL because although there probably is no database integrity problem, the inability to robustly query relationship fields in a web application renders the ObjectDB system seemingly useless.

objectdb-2.2.5_10

NetbeansIDE6.9.1

Glassfish3.01

client-server mode.

odb file written to db-files/greendesk/greendesk.odb (see persistence.xml), copy of odb output attached.

Sorry this is a complex one, but it is extremely important. I have taken great trouble to make it simpler to understand with careful output logging to the Glassfish logs, and also with a diagnostic table showing the query result in indexTest.xhtml.

The problems is illustrated by running the attached JavaEE Web App and viewing both the Glassfish output log and the single served JSF page indexTest.xthml. For a number of different List fields, a query performed immediately after creating and persisting many Element objects gives expected results, but the same query performed later form a served JSF page via a querying @EJB gives many fields null (and not even empty, I mean truly null).

In the following explanation I will refer to the (bi-directional) ownedElements List of the Element class, but the same problem happens to  uni-directional @OneToMany fields, and also there seem to be some simple @OneToOne relationship fields that likewise become null:

    private List<Element> ownedElements = new ArrayList<Element>();
    @OneToMany(mappedBy = "owner", cascade = CascadeType.ALL)
    public List<Element> getOwnedElements() {
        return ownedElements;
    }

    private Element owner;

    @ManyToOne(fetch = FetchType.LAZY)
    @JoinTable(name = "Element_owns_Element",
    joinColumns =
    @JoinColumn(name = "owner_ID", referencedColumnName = "ID"),
    inverseJoinColumns =
    @JoinColumn(name = "ownedElement_ID", referencedColumnName = "ID"))
    public Element getOwner() {
        return owner;
    }

(For completeness I've left the old ORM annotations in, but there are other cases without these that also exhibit the problem. Also, I don't think the FetchType is the culprit.)

INSTRUCTIONS TO REPRODUCE PROBLEM

The entities (mostly Element or Element subclass objects) are created in the @PostConstruct createData() method of ConfigBean. We will focus here on the case of one object of class TestDeepValues. On creation the logs show:

INFO: RequestBean []: Created: com.greensoft.entity.test.TestDeepValues [null](TEST: test deep values)
INFO: RequestBean []: Persisted: com.greensoft.entity.test.TestDeepValues [45](TEST: test deep values)
INFO: RequestBean []: Updated: com.greensoft.entity.test.TestDeepValues [45](TEST: test deep values)

Then at the end of createData() I perform an echoed query on all elements and interrogate the ownedElements list, which gives the expected results (corresponds with the database, too, see screenshot attached):

protected void testQuery() {
        String $i = "testQuery";
        TypedQuery<Element> query = em.createQuery("SELECT e FROM Element e", Element.class);
        List<Element> elements = query.getResultList();
        for (Element e : elements) {
            log_info($i,"---");
            log_info($i,"element",e);
            log_info($i,"element.project",e.getProject());
            List<Element> ownedElements = e.getOwnedElements();
            log_info($i,"element.ownedElements",ownedElements);
            if (ownedElements != null) {
                for (Element owned: ownedElements) {
                    log_info($i,"element.ownedElement",owned);
                }
            }
        }
        log_info($i,"---");
    }

INFO: ConfigBean []: testQuery: element([class=TestDeepValues][project=TEST: project 1][id=45][name=TEST: test deep values])
INFO: ConfigBean []: testQuery: element.project([class=Project][id=1][name=TEST: project 1])
INFO: ConfigBean []: testQuery: element.ownedElements([[class=FloatQuantity][project=TEST: project 1][id=46][name=deep float quantity (length)], [class=IntegerQuantity][project=TEST: project 1][id=47][name=deep integer quantity (number of rooms)], [class=BooleanValue][project=TEST: project 1][id=48][name=deep boolean value (no formaldehyde)], [class=StringValue][project=TEST: project 1][id=49][name=deep string value (building name)]])
INFO: ConfigBean []: testQuery: element.ownedElement([class=FloatQuantity][project=TEST: project 1][id=46][name=deep float quantity (length)])
INFO: ConfigBean []: testQuery: element.ownedElement([class=IntegerQuantity][project=TEST: project 1][id=47][name=deep integer quantity (number of rooms)])
INFO: ConfigBean []: testQuery: element.ownedElement([class=BooleanValue][project=TEST: project 1][id=48][name=deep boolean value (no formaldehyde)])
INFO: ConfigBean []: testQuery: element.ownedElement([class=StringValue][project=TEST: project 1][id=49][name=deep string value (building name)])

The following reported failed is the same whether or not I perform the above post-construct query.

When I serve the query via and @EJB TestQuery and @RequestScoped Page JSF to a simple XHTML page the list field values becomes null (in my larger real-world application, lots of other @OneToMany and @OneToOne fields are also null).

In all cases lists that are given in the explored database either empty or populated are served as null (!).

The query is performed by an @EJB:

@Stateless
public class TestQuery {

    @PersistenceContext
    private EntityManager em;

    public List<Element> findElements() {
        String $i = "testQuery";
        TypedQuery<Element> query = em.createQuery("SELECT e FROM Element e", Element.class);
        List<Element> elements = query.getResultList();
        return elements;
    }
}

And the JSF backingbean:

@ManagedBean
@RequestScoped
public class Page extends All_ {

    /** Creates a new instance of Page */
    public Page() {
    }

    @EJB
    private TestQuery testQuery;
    public List<Element> getElements() {
        String $i = "Page.getElements";
        List<Element> elements = testQuery.findElements();
        for (Element e : elements) {
            log_info($i,"---");
            log_info($i,"element",e);
            log_info($i,"element.project",e.getProject());
            List<Element> ownedElements = e.getOwnedElements();
            log_info($i,"element.ownedElements",ownedElements);
            if (ownedElements != null) {
                for (Element owned: ownedElements) {
                    log_info($i,"element.ownedElement",owned);
                }
            }
        }
        log_info($i,"---");

        return elements;
    }

}

Note how I am tediously logging the result to Glassfish log (so can test whether problem is at XHTML level, which it is not). Because it gets pulled more than once during the various JSF phases the output is quite copius, but one can easily extract the relevant object diagnostics thus, which shows the error, that the ownedElements field is now given as null:

INFO: Page []: Page.getElements: element([class=TestDeepValues][project=TEST: project 1][id=45][name=TEST: test deep values])
INFO: Page []: Page.getElements: element.project([class=Project][id=1][name=TEST: project 1])
INFO: Page []: Page.getElements: element.ownedElements(null)

There is also a copy of the indexTest.xhtml output with a table of the query results attached.

Now please note well. I have run this on EclipseLink and it works, completely.

There is either some setting I'm not familiar with in ObjectDB that explains this strange behaviour, or there is another genuine bug.

Either way, this has once again prevented my attempts to complete an evaluation of ObjectDB against my real-world project that works with EclipseLink, and it has once again taken many hours of my time.

Please attend to this as soon as possible and fix this carefully.

I look forward to having this fixed and to having a robust ObjectDB working on my problem, as I am greatly encouraged by experiments last night that lead to me being able to see large portions of my web application suite loaded in the odb Explorer.

regards,

Darren Kelly (Webel IT Australia, Sydney)

In the terms of your information at:

http://www.objectdb.com/java/jpa/persistence/retrieve

EITHER: the transparent activation is failing

AND/OR: entities are becoming detached.

http://www.objectdb.com/java/jpa/persistence/detach

I can't find out how to test where an entity is detached.

http://www.objectdb.com/api/java/jpa/PersistenceUtil

Does not seem to provide a method for testing this.

My feeling (before running your application) is that indeed detachment might be the problem here.

Try setting the fetch type of ownedElements to EAGER and check if it affects.

Transparent activation works only when the entity object is managed - once it becomes detached existing values are not expected to be cleared but then transparent activation is disabled for that entity object.

I think that JPA doesn't provide an API for checking detachment but JDO does:

    log_info(javax.jdo.JDOHelper.getObjectState(entity));

It is also possible to identify the state of an entity object by exploring it in the debugger.

After running your test application and investigating this issue further it seems that there is nothing wrong with ObjectDB behavior in this case.

Your TestQuery class uses a transaction scope EntityManager (the default as explained below), so when the findElements method completes - all the entity objects in the persistence context are detached. Because the ownedElements relationship is defined as LAZY (the default fetch type for to-many) - the entity objects are detached without that field. Transparent activation is not supported for detached objects - so the ownedElements field is unavailable for your JSF.

Please consider one of the following solutions (both tested successfully with your application):

1. Load the ownedElements relationship before detachment

The easiest way is to set the fetch type to EAGER:

@OneToMany(mappedBy = "owner", cascade = CascadeType.ALL, fetch=FetchType.EAGER)
public List<Element> getOwnedElements() {
    return ownedElements;
}

Alternatively you can use JOIN FETCH, accessing the field in your EJB's findElements, etc.

2. Switch to extended scope EntityManager

By doing that you prevent detachment on exit from findElements.

@Stateful // Change 1 - EXTENDED requires a Stateful session bean
public class TestQuery {
    @PersistenceContext(type=PersistenceContextType.EXTENDED) // change 2
    private EntityManager em;
    public List<Element> findElements() {
        String $i = "testQuery";
        TypedQuery<Element> query = em.createQuery(
            "SELECT e FROM Element e", Element.class);
        List<Element> elements = query.getResultList();
        return elements;
    }
}

Setting fetch=FetchType.EAGER on 'ownedElements' does seem to fix it (the lists load non-null, either as empty or correctly populated).

This seems to imply that ObjectDB operation under FetchType.LAZY (which the above would seem to suggest is your default) is broken.

Please investigate the test project to find out why elements become detached (which I assume is breaking the lazy loading).

Darren

PS: We do not want to have to fork our main web application to go through it and tediously change hundreds or even thousands of lists to FetchType.EAGER as a workaround just to perform an evaluation of ObjectDB. Also, FetchType.EAGER is not suitable in production mode for large hierarchies of office building data corresponding to ownedElements-owner.

Please read carefully the explanations at #4 - this works exactly as expected.

I am afraid your understanding of lazy fetch and detachment is wrong. Detachment is done automatically by the application server and lazy references are not expected to be loaded before detachment.

You may ask - why EclipseLink works differently in this case? Good question but you will have to refer it to the EclipseLink guys. ObjectDB doesn't do automatically things that are not required, such as loading lazy relationships (even though it is allowed by JPA - lazy is only a hint),  allocating IDs during persist, initialization of collections, etc. - because extra operations take extra time and ObjectDB is selected by customers because of its superior performance.

My understanding of lazy fetch and detachment is NOT wrong, and your system is not working correctly.

It is not correct that just because a field is set to LAZY LOAD that is should suddenly load as null (instead of empty or populated), and there is no reason why any of these entities involved should suddenly become detached.

The logic of the TestQuery and the Page JSF are no different from that of your own JavaEE tutorial and servlet + query example.

The reason the relationships of the entities should load at all is that loading should be triggered via the JSF EL access to the property, in this case ownedElements.

Your reply makes no sense at all. What is this about "initialization of collections" being "not required". They are initialized and stored into the database, and on retrieval the JPA provider is obliged to correctly populate them eagerly OR to put a placeholder there that will then cause population of the collection once the property/field is access, such as by a JSF page EL expression.

This is a massive problem for me. I should not have to go through 100s or 1000s of relationships and force them to be eager just because your system is not doing what JPA is supposed to. It has nothing to do with irregular behavior in EclipseLink, that is just nonsense.

Darren

https://stackoverflow.com/questions/3642151/detach-jpa-objects-with-lazy-initialized-properties

May I just point out something screamingly obvious.

Your own JavaEE example has a single entity class Guest with just 2 basic fields and no relationships, which is next to useless as a demonstration of the technology, because without any relationships there is no way of demonstrating the distinction between a Guest object and simple relational database table.

I don't mind being wrong if I learn something from it, and I'm not too proud to admit I'm wrong when I am, but I don't think I am.

Would you please demonstrate to me, either in my code or using a better example than Guest, the magic moment when a JSF developer having performed a query in an EJB and having returned that from a JSF backing bean getter is supposed to "reattach", or whatever is you are prescribing, the properties/fields for relationships ?  Then I will eat my hat.

Are you seriously suggested that after getting the query result I have to iterate over it and somehow reattach it to get the relationship data out ? Should I iterate over the result list and then fetch managed entities again from the EntityManager by id ?

EclipseLink, which BTW is the JPA reference implementation, gives entities as the result of queries that are attached; you should too.

Darren

See also:

https://stackoverflow.com/questions/1385840/what-is-the-preferred-design-pattern-for-combining-jpa-ejb-and-jsf-managed-beans

From (admittedly not the spec): http://en.wikibooks.org/wiki/Java_Persistence/Querying

"Normally JPA queries return your persistent Entity objects. The returned objects will be managed by the persistent context (EntityManager) and changes made to the objects will be tracked as part of the current transaction."

My query should likewise return managed entities, and the lazy loading of relationship fields should work.

From Pro  JPA2:

"When prepared within a transaction, the persistence provider has to take steps to convert the results into managed entities"

Note it does not say the application programmer has to do it with his/her hands in tedious code each time.

I agree with everything that is written in #11.

ObjectDB returns managed objects from the query.

But when findElements completes - Glassfish detaches all the entity objects - unless you use extended scope EntityManager (have you read #4?). This is not something that ObjectDB or any JPA provider can change.

> http://stackoverflow.com/questions/1385840/what-is-the-preferred-design-pattern-for-combining-jpa-ejb-and-jsf-managed-beans

Have read: can you please provide (I am currently looking):

- the bit in the JPA spec that says that the results of query need not be managed (that it is a provider variation point).

- the one line of code your approach that I need to introduce to ensure I can lazy load relationship fields.

After this much input and helping you fix bugs all week on my part, you owe my that much, even if it is to cure my delusion.

I repeat: this is a massive problem for me.

Ok, I am looking at #4 again and @Stateful // Change 1 - EXTENDED requires a Stateful session bean

I apologise that our exchange got out of sync, and I was busy looking at the JPA spec.

I will try @Stateful here on my application.

It would mean still changing a lot of classes.

And please note my remarks about your JavaEE example being in desperate need of some relationships !

Darren

More information in Pro JPA 2:

• Pages 139 - 142 (Transaction-Scoped Persistence Contexts / Extended Persistence Contexts)
• Pages 159 - 160  (Detachment)
• Pages 76 - 77 (Lazy Fetching)

I appreciate your help - in addition to finding some critical bugs your session of porting from EclipseLink to ObjectDB this week provides very useful input (maybe you can post a summary to the forum when you done).

Ok, I tried the @Stateful+PersistenceContextType.EXTENDED and agree it works, however it does mean some significant changes in my main project (although these won't hurt switching between ObjectDB and EclipseLink, we want to keep that option open if possible until the evaluation runs and is proven on our main web app, without forking them.)

EAGER is definitely out.

As for:

>I am afraid your understanding of lazy fetch and detachment is wrong.

I certainly don't think my understanding is wrong (and I won't be eating my hat); for the sake of this issue the wider discussion on how ObjectDB and EclipseLink differ (or perhaps interact with the Glsassfish app server differently) under @Stateless is now academic.

Thanks for your input and patience.

There is another problem that arises however for me with ObjectDB when neither LAZY nor EAGER are explicitly given for fetch, namely I get nullpointerexceptions downstream because of some tricky getters we use (which I'll report as a support ticket).

Q: What is supposed to happen in Object DB when neither LAZY nor EAGER are given ?

For I am getting a 3rd result that I do not get with either explicit EAGER or explicit LAZY

PS: You urgently need a good JavaEE example with multiple entities, and at least one variation that demonstrates @Stateful+PersistenceContextType.EXTENDED, because for large real-world systems forcing EAGER everywhere is not an option.

> Q: What is supposed to happen in Object DB when neither LAZY nor EAGER are given ?

JPA defines the default as EAGER, except for collections (to-many) which are LAZY by default.

Regarding tutorials  / examples, thank you for your comments. There is always room for improvements. Since the purpose of the tutorials is to help in starting using ObjectDB in the IDE / Server - I think that they should remain simple as possible, but additional demo applications should be added when time permit.

POSTSCRIPT: I urge everybody involved with ObjectDB and especially with your support to please read the following forum posting and especially the quotes I've found confirming that EclipseLink permits navigation of lazy loaded relationships on detached entities. This can have big impact on adoption of your ObjectDB system by existing EclipseLink users such as me, as it means they may have to perform significant recoding of their existing applications just to try your tool out:

Object DB vs EclipseLink/TopLink: on accessing lazy loaded relationships outside a transaction, after a query, from Glassfish

Darren

A feature request to implement a similar extension in ObjectDB was added to issue tracking.