Issue #1092: Unexpected exception (Error 990) on find

I have a huge amount of inserts/updates and after a about 850.000 Entities I got the following error:

[ObjectDB 2.4.7_16] Unexpected exception (Error 990)

  Generated by Java HotSpot(TM) 64-Bit Server VM 1.6.0_30 (on Windows 7 6.1).
Please report this error on http://www.objectdb.com/database/issue/new
com.objectdb.o.InternalException: null
com.objectdb.o.InternalException
at com.objectdb.o.VUT.k(VUT.java:691)
at com.objectdb.o.VUT.j(VUT.java:360)
at com.objectdb.o.STV.J(STV.java:309)
at com.objectdb.o.PAG.at(PAG.java:991)
at com.objectdb.o.MST.aU(MST.java:483)
at com.objectdb.o.MST.aT(MST.java:434)
at com.objectdb.o.MST.U3(MST.java:407)
at com.objectdb.o.WRA.U3(WRA.java:249)
at com.objectdb.o.LDR.F(LDR.java:532)
at com.objectdb.o.LDR.E(LDR.java:466)
at com.objectdb.o.LDR.z(LDR.java:268)
at com.objectdb.o.OBC.aO(OBC.java:1052)
at com.objectdb.o.OBC.aM(OBC.java:970)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:591)
at com.objectdb.jpa.EMImpl.find(EMImpl.java:514)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
at $Proxy33.find(Unknown Source)
at local.jpa.mapping.xmlparsing.RepoEbene1a.getEntity(RepoEbene1a.java:46)
at local.jpa.mapping.xmlparsing.RepoEbene1a.persist(RepoEbene1a.java:105)
at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at $Proxy34.persist(Unknown Source)

Unfortunately the stack trace doesn't provide sufficient information.

If this is reproducible and you can share your test it would be very helpful (you can upload it in a separate private support ticket, and source code is not essential).

Alternatively, you may check if this problem exists in previous versions (2.4.7, 2.4.6). If it is a new problem, we may be able to track the change the causes it.

Sadly, Im was not able to create a short testcase.

But I have found out that it comes to the exception, even if there are absolutly no relations.

I have a Map inside the entity, and there comes only Exceptions when the values becomes sometimes relativly long (> 256 chars). When I cut all values at a length of 256, there seams to come no exception.

It seams that there had to be some updates (not only inserts) to some entities. But the Exception seams to come always when searching for a item thats not in the db.

The problem is in version 2.4.7. I will try with an older Version on Monday.

The 256 observation is interesting and may be relevant.

The exception is thrown during find. What entity key type is used in that find? Is it String? What is the length of these strings? You mentioned long strings as map values, but in this context, long strings as entity keys could be more relevant.

Please check the database file using the ObjectDB Doctor and report errors.

The keys are also Strings, but I believe they are much shorter. There was no problem with the keys when I test it with shorter values, but I will check on Monday.

Probably its the combination of Key- and Value-Length.

Hi, the ObjectDB Doctor says:

ObjectDB Doctor [version 2.4.7_16]
Copyright (c) 2013, ObjectDB Software. All rights reserved.

Scanning the database file...
.................................................. 100MB
.................................................. 200MB
.................................................. 300MB
.................................................. 400MB
. 403MB (total)

Analyzing database structure...
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Global Value Errors
-------------------
[1] Unexpected total object count: 850986 (expected 850977)
[2] Unexpected total page count: 206592 (expected 206563)

BTree Value Errors
------------------
[1] local.jpa.entity.ebene1a.MyEntity
  - Unexpected object count: 850986 (actual 850977)

Page Relation Errors
--------------------
[1] Page #96658 unexpected exception: null
[2] Page #96658 last key is null, parent page #144908 next key is 'Elementarprodukt:400000100465140:7:1:'
[3] Page #119529 unexpected exception: null
[4] Page #119529 last key is null, parent page #155867 next key is 'Elementarprodukt:400000100437073:6:1:'|1
[5] Page #121854 unexpected exception: null
[6] Page #121854 last key is null, parent page #144906 next key is 'Elementarprodukt:400000100435789:5:1:'|1
[7] Page #162693 unexpected exception: null
[8] Page #162693 last key is null, parent page #69427 next key is 'Elementarprodukt:400000100444732:5:1:'|1
[9] Page #166680 unexpected exception: null
[10] Page #166680 last key is null, parent page #89322 next key is 'Elementarprodukt:400000100458624:5:1:'|1
[11] Page #173823 unexpected exception: null
[12] Page #173823 last key is null, parent page #57101 next key is 'Elementarprodukt:400000100317025:7:1:'|1
[13] Page #184921 unexpected exception: null
[14] Page #184921 last key is null, parent page #114591 next key is 'Elementarprodukt:400000100417179:5:1:'|1
[15] Page #203864 unexpected exception: null
[16] Page #203864 last key is null, parent page #203865 next key is 'Elementarprodukt:400000100422048:13:1:'|1

Page Content Errors
-------------------
[1] Unexpected exception in page #96658 entry #2: null
[2] Page #118249 has unsorted entry TOC (entry 3)
[3] Unexpected exception in page #118249 entry #2: null
[4] Unexpected exception in page #119529 entry #2: null
[5] Unexpected exception in page #121854 entry #2: null
[6] Unexpected exception in page #162693 entry #2: null
[7] Unexpected exception in page #166680 entry #2: null
[8] Unexpected exception in page #173823 entry #2: null
[9] Unexpected exception in page #184921 entry #1: null
[10] Unexpected exception in page #203864 entry #2: null

Index Errors
------------
[1] Index local.jpa.entity.ebene1a.MyEntity[assignment] requires rebuild.
  - has 850977 entries instead of 850986

Large Sections with Errors
--------------------------
Group #1:
Page#70606 3:'Elementarprodukt:400000100317025:7:1:' 1/2 1980+609/2589
Group #2:
Page#138243 3:'Elementarprodukt:400000100417179:5:1:' 1/2 1980+396/2376
Group #3:
Page#165140 3:'Elementarprodukt:400000100422048:13:1:' 1/2 1980+519/2499
Group #4:
Page#74733 3:'Elementarprodukt:400000100435789:5:1:' 1/2 1980+544/2524
Group #5:
Page#50108 3:'Elementarprodukt:400000100437073:6:1:' 1/2 1980+31/2011
Group #6:
Page#44332 3:'Elementarprodukt:400000100444732:5:1:' 1/2 1980+293/2273
Group #7:
Page#63413 3:'Elementarprodukt:400000100450183:5:1:' 1/2 1980+41/2021
Group #8:
Page#108541 3:'Elementarprodukt:400000100458624:5:1:' 1/2 1980+669/2649
Group #9:
Page#89835 3:'Elementarprodukt:400000100465162:5:1:' 1/2 1980+669/2649

Is it a new database, created by this ObjectDB version?

If you run your application again with values larger than 256 characters, will you get these Doctor errors again?

Could you try running the Doctor on a database that you create with values smaller than 256 characters?

When I cut the values in the map to 256 chars, it runs to the end and there are no errors reportet from the ObjectDB Doctor:

ObjectDB Doctor [version 2.4.7_16]
Copyright (c) 2013, ObjectDB Software. All rights reserved.

Scanning the database file...
.................................................. 100MB
.................................................. 200MB
.................................................. 300MB
.................................................. 400MB
.................................................. 500MB
.................................................. 600MB
.................................................. 700MB
.................................................. 800MB
.................................................. 900MB
.................................................. 1000MB
.................................................. 1100MB
.................................................. 1200MB
.................................................. 1300MB
.................................................. 1400MB
.................................................. 1500MB
.................................................. 1600MB
.................................................. 1700MB
.................................................. 1800MB
.................................................. 1900MB
.................................................. 2000MB
.................................................. 2100MB
.................................................. 2200MB
.................................................. 2300MB
.................................................. 2400MB
.................................................. 2500MB
.................................................. 2600MB
.................................................. 2700MB
.................................................. 2800MB
.................................................. 2900MB
.................................................. 3000MB
.................................................. 3100MB
.................................................. 3200MB
.................................................. 3300MB
.................................................. 3400MB
.................................................. 3500MB
...... 3513MB (total)

Analyzing database structure...
10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

-------------------------
No errors have been found
-------------------------

I've also tried with 2.4.6 and 2.4.5 and get errors

I've read in the forum about the key com.objectdb.disable.optimization.small=true

I've tried, but there is no change in the behaviour.

The Doctor results show that the problem is indeed related to the size parameter.

Unfortunately it would be very difficult to locate the source of the problem without a way to reproduce it.

If you cannot provide an application (not necessarily a small test, and possibly without source code), maybe we can build a test that reproduces it, but much more details are needed, it will take a lot of time, and the effort may not succeed.

Please check if value size 256 is indeed the limit, i.e. what is the maximum size that does work. Maybe the problem is related to the size of the values indirectly, since objects that are larger than 2KB need more than one page and are handled differently.

With a limit of 1120 it seams to work, with a limit of 1130 it won't. Propably with the Key and the Overhead of the Map-Entry-Struktur it hits the 2k limit.

There was a problem with using large objects in the past, but it was fixed and we use tests (on every build of ObjectDB) to verify that large objects work well.

Unfortunately it seems that you are experiencing a new problem. Apparently progress in fixing this would be possible only if we will be able to reproduce it, either by a simple test or by a large application. So this issue will have to remain open until a test case is submitted.

I've found something out. When I change my Id to a unique Column it seams to work without an error and the Doctor reports no error.

The searching of Entities will in this case use a query instead of a find.

So the core of my entity look now like this:

@Entity

@Table(schema="ebene1")
public class MyEntity {
protected static final int ATTR_LAENGE = 1000000000; // no limit, but in case of objectdb i think this limit should have no effekt

    // @Id // with this instead of the unique column it won't work
    @Column(unique = true, nullable = false, length = 127)
    private String key;

    @ElementCollection(fetch=FetchType.EAGER)
    @CollectionTable(joinColumns=@JoinColumn(name="FK"), uniqueConstraints=@UniqueConstraint(columnNames={"K", "FK"}))
    @MapKeyColumn(name = "K")
    @Column(name = "V", length = ATTR_LAENGE)
    private Map<String, String> attributes = new HashMap<String, String>();
}

Your last finding indicates that the problem happens in some specific case. However, it is still impossible to detect the problem without either further isolation of the problem and producing a small test case (which is preferred), or by providing a full application that demonstrates the exception.

This report is similar to the report in issue #1977  that was fixed now in version 2.7.1.

The new version fixes a critical bug that in some cases (rare but possible) may cause database corruption.