database more or less destroyed

again at our major Aerospace customer we are facing the situation that the database became useless after a server shutdown. Attached the log of the doctor run, any idea how such massive problems can occur

The Doctor output indicates index corruption, i.e. index synchronization with the data failed.

As it happened after server shutdown, it is likely that the database file was not closed correctly. Was it controlled exit or server process killed? Even if the process was killed, recovery is expected automatically on the next time the database is opened, unless the recovery file is not available (e.g. the database is first opened after copying to another location without the recovery file). Is there an indication in the log for recovery? More information is needed.

it was an direct shutdown of the server without any shutdown of our app. We did not copy the db file or do anything special, see attached the log after the restart (after server shutdown)

by the way, are the indexes updated/rebuilt by the doctor ? in the log after the doctor run we do not see a rebuild of the indexes

The log indicates that the recovery was executed after restart. Unfortunately the errors later indicate that it failed.

To explore why it failed we need the database and the recovery file before the restart, which may not be available to you now. If you could reproduce the issue in tests, it will be examined.

The Doctor rebuilds all indexes. Run it again (in test mode) to make sure that there are no more issues.

Parallely to exploring this issue, it is recommended to make the arrangements that a server shutdown will include ordinary shutdown of the application and the database.

It is also recommended to check the file system, which might have been damaged during that shut down. If you see errors in the storage device or the file system then the database file may develop additional issues now (and that also may be a cause of recovery failures), so such errors have to be fixed.

the file system and hardware checks have already been made, there was no problem detectable, here the link to the db file before doctor

https://www.dropbox.com/scl/fi/g5bbh0w9432yd5sam2bta/coreSystemDb_.7z?rlkey=lgxc0t1e7a3dtjd5tv1xil6p9&dl=0

do you have any issues running objectdb on virtual environments

There is a report of performance issue, which is still open, as there are no sufficient details and a test to reproduce it.