Replication issue, Replayer failing due to NullPointerException.

#1

I'm getting replication errors and upon trying to play the replayer I get a NullPointerException which i'm also seeing in the slave logs when it's trying to replicate...

$ sudo java -cp objectdb.jar com.objectdb.Replayer db.odb
ObjectDB Replayer [version 2.4.2]
Copyright (c) 2012, ObjectDB Software. All rights reserved.
java.lang.NullPointerException
at com.objectdb.o.LFL.X(LFL.java:1154)
at com.objectdb.o.LFL.P(LFL.java:934)
at com.objectdb.o.FIW.af(FIW.java:94)
at com.objectdb.o.FIW.ad(FIW.java:55)
at com.objectdb.o.SSS.aj(SSS.java:907)
at com.objectdb.o.SSS.ag(SSS.java:731)
at com.objectdb.o.SSS.af(SSS.java:707)
at com.objectdb.o.MST.UB(MST.java:1109)
at com.objectdb.o.STC.l(STC.java:324)
at com.objectdb.o.STC.i(STC.java:212)
at com.objectdb.o.STC.h(STC.java:142)
at com.objectdb.o.RPM.t(RPM.java:223)
at com.objectdb.Replayer.main(Replayer.java:62)

Any idea what could be causing this?

#2

I should add this causes the replicated database to be out of sync with the master.

#3

The stack trace indicates that the replicated database is unexpectedly closed.

Maybe there are additional relevant stack traces in the log?

ObjectDB Support
#4

There isn't too much else useful in the log, i've tried deleting all the DB files on both sides, starting a clean master and verifying it works as expected and the DB's are recording from the very first transaction. Now when I go to run the slave server and get it to replicate I still get logs full of:

java.lang.NullPointerException
at com.objectdb.o.SFL.ae(SFL.java:851)
at com.objectdb.o.MST.ae(MST.java:1378)
at com.objectdb.o.MST.Vb(MST.java:1289)
at com.objectdb.o.WRA.Vb(WRA.java:369)
at com.objectdb.o.WSM.Vb(WSM.java:174)
at com.objectdb.o.WRA.Vb(WRA.java:369)
at com.objectdb.o.WSN.Vb(WSN.java:658)
at com.objectdb.o.STC.v(STC.java:511)
at com.objectdb.o.STC.i(STC.java:248)
at com.objectdb.o.STC.h(STC.java:142)
at com.objectdb.o.RPR.run(RPR.java:202)
at java.lang.Thread.run(Thread.java:722)
com.objectdb.o.UserException: Failed to synchronize replicated database
at com.objectdb.o.MSG.d(MSG.java:61)
at com.objectdb.o.RPT.d(RPT.java:97)
at com.objectdb.o.RPT.<init>(RPT.java:65)
at com.objectdb.o.SHN.aw(SHN.java:704)
at com.objectdb.o.SHN.K(SHN.java:203)
at com.objectdb.o.HND.run(HND.java:133)
at java.lang.Thread.run(Thread.java:722)

However I do see all the DB's under "/usr/test/backend/db/$replication/" on the slave; "/usr/test/backend/db/" is the data path and where all the DB's are on the master. I cannot get an EntityManager connection to the slave DB though.

I've tried stopping both servers and copying the db files from the master across to the slave first just in case as per: http://www.objectdb.com/database/forum/346 even though this shouldn't be relevant since I have recording on from the start.

It's worth noting that some of the databases may be totally empty(no objects persisted) when trying to replicate.

I'm going to show you my master and slave config files in case there is something obvious I am doing wrong:

Master

<objectdb>

<general>
  <temp path="$temp/ObjectDB" threshold="64mb" />
  <network inactivity-timeout="0" />
  <url-history size="50" user="true" password="true" />
  <log path="/var/log/test/" max="10mb" stdout="false" stderr="false" />
  <log-archive path="/var/log/test/archive/" retain="90" />
  <logger name="*" level="info" />
</general>

<database>
  <size initial="10mb" resize="10mb" page="2kb" />
  <recovery enabled="true" sync="false" path="." max="512mb" />
  <recording enabled="true" sync="false" path="." mode="write" />
  <locking version-check="true" />
  <processing cache="64mb" max-threads="10" synchronized="false" />
  <query-cache results="32mb" programs="500" />
  <extensions drop="test,testing" />
</database>

<entities>
  <enhancement agent="true" reflection="error" />
  <cache ref="weak" level2="0mb" />
  <persist serialization="false" />
  <cascade-persist always="auto" on-persist="false" on-commit="true" />
  <dirty-tracking arrays="false" />
</entities>

<schema>
</schema>

<server>
  <connection port="2122" max="100" />
  <data path="/usr/test/backend/db/" />
</server>

<users>
  <user username="backend" password="password" admin="true">
    <dir path="/" permissions="access,modify,create,delete" />
  </user>
  <user username="backend-ro" password="password">
    <dir path="/" permissions="access" />
  </user>
</users>

<ssl enabled="false">
  <server-keystore path="$objectdb/ssl/server-kstore" password="pwd" />
  <client-truststore path="$objectdb/ssl/client-tstore" password="pwd" />
</ssl>

</objectdb>

Slave

<objectdb>

<general>
  <temp path="$temp/ObjectDB" threshold="64mb" />
  <network inactivity-timeout="0" />
  <url-history size="50" user="true" password="true" />
  <log path="/var/log/test/" max="10mb" stdout="false" stderr="false" />
  <log-archive path="/var/log/test/archive/" retain="90" />
  <logger name="*" level="info" />
</general>

<database>
  <size initial="10mb" resize="10mb" page="2kb" />
  <recovery enabled="true" sync="false" path="." max="512mb" />
  <recording enabled="true" sync="false" path="." mode="write" />
  <locking version-check="true" />
  <processing cache="64mb" max-threads="10" synchronized="false" />
  <query-cache results="32mb" programs="500" />
  <extensions drop="test,testing" />
</database>

<entities>
  <enhancement agent="true" reflection="error" />
  <cache ref="weak" level2="0mb" />
  <persist serialization="false" />
  <cascade-persist always="auto" on-persist="false" on-commit="true" />
  <dirty-tracking arrays="false" />
</entities>

<schema>
</schema>

<server>
  <connection port="2122" max="100" />
  <data path="/usr/test/backend/db/" />
  <replication url="objectdb://master:2122/w.odb;user=backend-ro;password=password" />
  <replication url="objectdb://master:2122/x.odb;user=backend-ro;password=password" />
  <replication url="objectdb://master:2122/y.odb;user=backend-ro;password=password" />
  <replication url="objectdb://master:2122/z.odb;user=backend-ro;password=password" />
</server>

<users>
  <user username="backend-ro" password="password">
    <dir path="/" permissions="access" />
  </user>
</users>

<ssl enabled="false">
  <server-keystore path="$objectdb/ssl/server-kstore" password="pwd" />
  <client-truststore path="$objectdb/ssl/client-tstore" password="pwd" />
</ssl>

</objectdb>

 

I'm using the following connection String for my slave EntityManagerFactory:

"objectdb://localhost:2122//master:2122/x.odb;user=backend-ro;password=password"

The master is working 100% fine with:

"objectdb://master:2122/x.odb;user=backend;password=password" and the above config.

 

I was expecting I should be able to point that slave config at a master without any existing DB files on the slave and have it replicate without too much pain, assuming recording is enabled from the start.

#5

I managed to fix this by copying the contents of $replication into the directory above it(the "data path"), this seems like a bug.

I was seeing errors along the lines of the user had no permissions to create a database in the path etc. which isn't correct as it was running as root, I just had this suspicious feeling that copying the db contents into the data path would fix it as it did the same thing a while ago and that fixed it.

I am also finding the recording directories growing upon DB restarts even though there was no writing to the DB, they double in size exactly the amount of the resize value. This means everytime I restart the DB my DB is growing by resize value across all DB's, this also seems like a bug. I have recording set to write only.

 

#6

From your description it does seems as a bug, but unfortunately exploring it and finding the cause will require a test case that will demonstrate the problem.

ObjectDB Support

Reply