Issue #1229: Server crash - Mismatch client-server protocol prefix

We've been having a problem where our ObjectDB server goes unresponsive every 2 days or so.  I've reviewed the server-side logs, and there are no events logged shortly before the crash.  Once the server gets in this state, it is unresponsive to bin/server.sh restart; I have had to reboot the entire server on which it's running.  The only recorded symptom is a client-side error (which fortunately gets auto-mailed to me):

[ObjectDB 2.5.2] javax.persistence.PersistenceException
Mismatch client-server protocol prefix (error 523)
        at com.objectdb.jpa.JpaQuery.getSingleResult(JpaQuery.java:723)
        at org.apache.jsp.status_jsp._jspService(status_jsp.java:101)
        at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:390)
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:334)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at com.fastmodel.fastplan.db.EntityManagerFactory.doFilter(EntityManagerFactory.java:202)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:680)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:999)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:565)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:307)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
Caused by: com.objectdb.o.UserException: Mismatch client-server protocol prefix
        at com.objectdb.o.MSG.d(MSG.java:61)
        at com.objectdb.o.SCO.b(SCO.java:210)
        at com.objectdb.o.NTS.z(NTS.java:287)
        at com.objectdb.o.CLS.S(CLS.java:340)
        at com.objectdb.o.CLS.Q(CLS.java:282)
        at com.objectdb.o.CST.U6(CST.java:501)
        at com.objectdb.o.QRR.g(QRR.java:244)
        at com.objectdb.o.QRR.f(QRR.java:153)
        at com.objectdb.jpa.JpaQuery.getSingleResult(JpaQuery.java:716)
        ... 27 more

I realize this isn't a lot to go on; I can provide a snapshot of our database via private channel if you would like.

Since you are probably using the same ObjectDB version on the client and the server, the stack trace indicates an issue in communication between the client and the server, which could possibly happen due to an unknown server failure.

The database file will not help here, but just verify by running the Doctor that it is healthy.

A thread dump of the server could be useful if you had a way to obtain it.

In addition, you just reported another issue today. Is it just a coincidence that both issues started now? Maybe the other issue is the cause of this server problem? Maybe both are result of a change in your system or upgrading an ObjectDB version?

The connection between the two issues is that I discovered issue #206 while examining the logs, hunting for a clue regarding this issue.  I had upgraded my production server to 2.5.2 on 8/15 (from 2.3.something).  The issue I reported as #206 started within a few days, and occurred in clusters, but its occurrence never immediately preceded a crash.  The crashes didn't occur until 8/29.  Approximate failure times were 8/29 @ 11:54 AM, 8/30 @ 7:41 PM, 9/1 @ 12:15 PM (all times PDT).

Other issues I noted in the log were some out-of-memory errors (although I'm running with -Xmx1024m) and some issues with a particular index, which was also flagged when I ran the database doctor on a backup of the production database.  I will be taking the production site offline tonight so I can run a repair on that issue.

I'm attaching all server logs since the upgrade.

All these errors may be related. Please report again after repairing the index and increasing the heap size (-Xmx1024m may not be sufficient).

Do you still see this error message after eliminating the memory and index issues?

Update: Apparently this exception is the result of an OutOfMemoryError exception, and allocating sufficient RAM solves the problem. We should check how future versions of ObjectDB can at least produce a better error message in this case.