doctor error message after database getting page errors

again we have at another customer after weeks of no problems these entries in the log (and during that times the system is nearly unusable):

...

[2023-09-01 05:43:19 #4 store] 
RetrievalTask.reportPrimaryResult0: page#18247113, reader.getPos() = 21

[2023-09-01 13:03:07 #5 store] 
RetrievalTask.reportPrimaryResult0: page#18647442, reader.getPos() = 1344

[2023-09-01 13:03:32 #6 store] 
RetrievalTask.reportPrimaryResult0: page#18109299, reader.getPos() = 21

[2023-09-01 13:30:21 #7 store] 
RetrievalTask.reportPrimaryResult0: page#16379024, reader.getPos() = 21

[2023-09-01 13:30:59 #8 store] 
RetrievalTask.reportPrimaryResult0: page#16009697, reader.getPos() = 271

[2023-09-01 13:35:19 #9 store] 
RetrievalTask.reportPrimaryResult0: page#18515423, reader.getPos() = 1371

[2023-09-01 13:44:58 #10 store] 
RetrievalTask.reportPrimaryResult0: page#18009259, reader.getPos() = 495

[2023-09-01 13:53:44 #11 store] 
RetrievalTask.reportPrimaryResult0: page#18077984, reader.getPos() = 21

[2023-09-01 13:55:01 #12 store] 
RetrievalTask.reportPrimaryResult0: page#17736647, reader.getPos() = 932

[2023-09-01 13:55:24 #13 store] 
RetrievalTask.reportPrimaryResult0: page#17500039, reader.getPos() = 631

...

trying to do a doctor on the customer system with around 40 GByte free RAM (database is around 37 GByte) ends up with a heap exception see attachment

we had to transfer the database to an osx pc, doing the doctor shows:

Page Relation Errors

--------------------


[1] Free page list is broken

Here is a link for the corrupt database:

https://www.dropbox.com/scl/fi/6ywgr9yx459c9tskv525t/coreSystemDb.rar?rlkey=cyokdss3zkpwfi66rh88j1scx&dl=0

It is unclear whether the issue is indeed with the database file, as besides broken references (which are application errors) the Doctor only reports a "Free page list is broken" error, which by itself may not be a critical issue (there is a known issue with removing an index in old databases that may cause this error and the result is free space that is not reclaimed but no other known consequences, the space  can be reclaimed by fixing the database with the Doctor).

The OutOfMemoryError crash when running the Doctor is probably due to insufficient heap size. The Doctor seems to run fine with the default JVM heap size on a PC with 16GB RAM.

Did a server restart solve the issue? If it did then the error is probably not with the database file but with some unexpected runtime state. There is a good chance that it will not happen if you switch to the most up to date version, as several critical bugs that may cause various unexpected states have been fixed in the recent years.

When these "RetrievalTask.reportPrimaryResult0' are written to the log, exceptions with more details are also thrown. Have you seen such an exception in the log file (on the server or the client) or thrown back to your application?

some report about the state:

- server restart solved the problem for a while

- we changed this night to 2.8.8_05, performance looks good

in the log we have now these entries:

Large number of query plan combinations (1906624)

is there a config setting that allows us to log the queries that cause such a warning ?

Version 2.8.9 should also log the relevant query string.