Failed to resize file - file system limitation error

Hi,  we are having production problems with ObjectDB and the below error in our production ODB service log. 

Can you please let us know the reason this is occuringand how to resolve?

Thank you,

Clinton Moore

813-562-0116

Exception in thread "Thread-13044" com.objectdb.o.UserException: Connection is closed
    at com.objectdb.o.MSG.a(MSG.java:64)
    at com.objectdb.o.NTS.Y5(NTS.java:206)
    at com.objectdb.o.NTS.f(NTS.java:194)
    at com.objectdb.o.STD.j(STD.java:121)
    at com.objectdb.o.STD.k(STD.java:132)
    at com.objectdb.o.DTT.run(DTT.java:85)
    at com.objectdb.o.STD.run(STD.java:75)
com.objectdb.o.UserException: Failed to resize file 'F:\CAP\AdherisHealthSolutions\DatabaseServer\data\patient_history_RiteAid.odb'
    at com.objectdb.o.MSG.a(MSG.java:77)
    at com.objectdb.o.LFL.Zi(LFL.java:1047)
    at com.objectdb.o.DFL.run(DFL.java:1002)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: The requested operation could not be completed due to a file system limitation
    at java.io.RandomAccessFile.setLength(Native Method)
    at com.objectdb.o.LFL.Zi(LFL.java:1044)
    ... 2 more

What is the current database file size and which file system do you use? If the database file size reached the maximum size for your file system you may have to switch to a different file system.

Other possible causes for this exception reported on the Internet (not related directly to ObjectDB) are too many files in a directory, fragmentation and compression, e.g.:

https://forum.mendix.com/link/space/app-development/questions/7625

https://stackoverflow.com/questions/15683417/file-copy-error-the-requested-operation-could-not-be-completed-due-to-a-file-sy

I ran the Database Doctor and it ended up failing with the error below.  Can you please advise on next steps?

Thank you,

Clinton

[12161] Page #26356937 has invalid reference from key 'AEA6C33' to child page #7
8764514.
[12162] Page #27778657 has invalid reference from key 'A3A28F6CD4E1EF0C1D8DCE59E
A9BC9986338B205'|2 to child page #78764515.
[12163] Page #32233516 has invalid reference from key 'A4F58B6' to child page #7
8764512.
[12164] Page #33415486 has invalid reference from key 'FDB6ECE' to child page #7
8764513.
[12165] Page #36685050 has invalid reference from key 'B266C4FE' to child page #
78764525.
[12166] Page #42215999 has invalid reference from key 'A5FFBA3C9958CA09DC8A353B6
32DC615145D3128'|2 to child page #78764517.
[12167] Page #43528578 has invalid reference from key '4036A4B' to child page #7
8764520.
[12168] Page #49318794 has invalid reference from key '5765680F6861CE7BFBFD00E49
6EDB536B07BBF0F'|2 to child page #78764521.
[12169] Page #57737986 has invalid reference from key '65624BDAAF8673559641A7037
F4FE610B90DF6D8'|3 to child page #78764516.
com.objectdb.o.UserException: Failed to read from file 'C:\Users\CMOORE~1\AppDat
a\Local\Temp\objectdb_3126483439234298180\DoctorSection_1022.fls'
        at com.objectdb.o.MSG.a(MSG.java:64)
        at com.objectdb.o.FIR.n(FIR.java:147)
        at com.objectdb.o.BYR.h(BYR.java:652)
        at com.objectdb.o.BYR.r(BYR.java:569)
        at com.objectdb.o.BYR.r(BYR.java:557)
        at com.objectdb.o.RSV.S(RSV.java:141)
        at com.objectdb.o.VUT.b(VUT.java:695)
        at com.objectdb.o.VUT.j(VUT.java:373)
        at com.objectdb.o.VUT.j(VUT.java:359)
        at com.objectdb.o.CMV.<init>(CMV.java:70)
        at com.objectdb.o.VUT.b(VUT.java:675)
        at com.objectdb.o.VUT.j(VUT.java:373)
        at com.objectdb.o.VUT.j(VUT.java:359)
        at com.objectdb.o.VLV$a._c(VLV.java:485)
        at com.objectdb.o.VLV$a.compareTo(VLV.java:466)
        at com.objectdb.o.HEP.f(HEP.java:208)
        at com.objectdb.o.VLV.g(VLV.java:359)
        at com.objectdb.o.VLV.h(VLV.java:321)
        at com.objectdb.o.VLV.e(VLV.java:289)
        at com.objectdb.o.SCV.b(SCV.java:94)
        at com.objectdb.o.DCM.q(DCM.java:540)
        at com.objectdb.o.DCM.p(DCM.java:283)
        at com.objectdb.o.DCM.j(DCM.java:183)
        at com.objectdb.Doctor.main(Doctor.java:22)

[ObjectDB 2.7.6] Failed to read from file 'C:\Users\CMOORE~1\AppData\Local\Temp\
objectdb_3126483439234298180\DoctorSection_1022.fls' (error 117)

F:\CAP\AdherisHealthSolutions\DatabaseServer\bin>

To answer your question, the file size of patient_history_RiteAid.odb is 157.5 GB.   The file system is NTFS.   This maximum file size for NTFS does not sound like it is something we are hitting.   Do you have any other ideas?

Thank you,

Clinton

The next step may be to run the Doctor in repair mode and check whether you still have the errors with the new generated (fixed) database.

Turning off the disk compression did fix the issue.  We no longer have production impact.

Please close the ticket.

Thanks,

Clinton

Thank you for the update, and regarding the Doctor errors, consider: (a) repairing the database and switching to the new generated database, (b) running a newer version of the Doctor from the last ObjectDB version (even if you cannot upgrade to the last ObjectDB version in this production system).

Regards,

Hi, we are experiencing this same issue again, on the same file, and it is impacting production.  I confirmed that disk compression is disabled.  I have shut down the DB server am copying the file to ensure that file fragmentation is not an issue.   If this does not correct the issue do you have any other suggestions?

Thank you,

Clinton

The cause of this exception is unclear, but here are several suggestions:

1. Try to resize a copy of this database without ObjectDB, by a simple Java console application that creates a RandomAccessFile instance for that file and then calls setLength, e.g. with a new size of 200GB. If it works, try the same while the file is locked for reading Try it a again also when the file is locked for reading/writing. These checks might indicate whether the issue is related to ObjectDB or to your system limits.
2. If not set, try to set the objectdb.temp.concurrent-file-read system property to true when running your server (-Dobjectdb.temp.concurrent-file-read=true). If set to true, try false.
3. If not tried already, try to repair the database file using the Doctor and then switch to the new created database file.

Thank you for your response.   We have defragmented the file which originally had 700K+ fragments (on SSD however, which we did not think would be an issue).

I have not been able to run DB Doctor because of the space required for a temp file on C:\.   Is there a way to place this temporary file used during execution of DB Doctor on a different file?

Also, is there any additional debugging that we could enable to help with determining root cause if this occurs again?  If so, how can I enable this?

Thank you,

Clinton

Since it last happened, have you seen the database file becoming larger successfully?

You can change the temporary path in the ObjectDB configuration (the one that is used by the Doctor) or in running Java, but probably it is unlikely that running the Doctor will solve the issue, so it is just something to try.

This is a system exception rather than ObjectDB exception, in a native Java method RandomAccessFile::setLength. The setLength method produces the following error messages if the requested length parameter is invalid (in Java 17):

• If it is negative: "An attempt was made to move the file pointer before the beginning of the file".
• If it exceeds the allowed size: "The parameter is incorrect".

So it does seem as a specific system issue rather than a bug in ObjectDB or a corrupted database file.

Therefore, you might be able to debug it externally to ObjectDB by using suggestion 1 in post #9 above.