Failed to synchronize replicated database

Hello, we are trying to enable replication feature,

But when we are trying to startup slave database, we'll get following Exception:

com.objectdb.o.UserException: Failed to synchronize replicated database
    at com.objectdb.o.MSG.a(MSG.java:64)
    at com.objectdb.o.RPT.a(RPT.java:100)
    at com.objectdb.o.RPT.<init>(RPT.java:67)
    at com.objectdb.o.SHN.R(SHN.java:706)
    at com.objectdb.o.SHN.w(SHN.java:204)
    at com.objectdb.o.HND.run(HND.java:134)
    at java.base/java.lang.Thread.run(Thread.java:832)

I've tried to stop master, clear all recording files, switch to all mode:

<recording enabled="true" sync="false" path="." mode="all" />

Then I stopped also the slave
Removed all files from $replication directory.
Started master
Started slave

But the exception is still thrown periodically.
Please can you provide any troubleshooting steps?

For replication to work the slave database must start with a copy of the master database in the replication directory (see Setting Slave Databases). Have you copied the master database to the slave?

Thanks, it helped, now it seems working. I have few followup questions:

The replicated databases on the slave server are automatically generated under a special root directory, $replication, under the server data root directory. Starting a new replication of an existing master database requires copying the existing master database to the slave side.

1. By "requires copying the existing master database to the slave side" I assumed, that ObjectDB server can do it on it's own, when the slave database does not exist in the $replication directory. Can't this feature be added?

2. How big difference can the ObjectDB server synchronize? In the scenario when the master database is started, and there are done some write operations before the slave database is started. Should it work that way? Or are we supposed to start them both before allowing clients to connect the master database.

1. Copying the existing database file automatically is not complex but might be slow if the database is very large, and may cause issues if there is no sufficient space on the disk for another copy of the database. Anyway, we will try to take this comment into account in future development.

2. The replication should catch up if all the transactions are available, i.e. transaction recording of the master is available from the moment of the last slave database update.

1. Thank You
2. Do you have somewhere written, how the recording works in more detail?
From what I read and saw It seems to me, that after first start it will create a copy of current state as some basepoint (which is also taking quite lot of time on bigger files :-)) And then it is recording the transactions which can be replied to the basepoint. But I did not found, that there is any limit, how many transactions should be recorded. How we can properly manage disk space used by recording?
I suppose that we can harmlessly delete all recordings when ODB Server is offline, and it will create new basepoint, but are there any other options? For example, that at some point transactions are written to the basepoint and deleted later to save disk space?
 

The recording files have two main uses:

1. Provide the ability to replay the entire transaction history, since the last breakpoint (i.e. an available db backup), e.g. for debugging.
2. In replication, to help slave databases catch up with the master, mainly if the slave was down.

If there is sufficient storage space then keeping all (or many) recording files could be beneficial. However, for replication you only need the last recording files, containing the transactions that have not been applied yet to the slaves (e.g. if the slave was down).

So there is not currently an API that would allow us to cleanup old transaction files at runtime? We would really love that, because we are using multiple separate ObjectDB Servers. Any server is responsible for managing few DB files, but need to keep them separately.
And I think, that we would need to truncate transaction files time to time. 

Allowing it on runtime by replying transactions to the last basepoint would be welcomed because otherwise we would have increased downtime during copying the database again after clearing transaction files which would be unpleasant.

However, for replication you only need the last recording files

 by this you mean, that we can even remove older transaction files leaving there the new ones and the base file, so we would not been able get to the current state by replying transactions, but the replication would work still? It doesn't seem to me as a good practise, so I would rather delete all transactions and create new base file. But I would like to know if I understood it correctly.

Really Thank You for your time spent answering my questions.

Yes, it seems you understood correctly. There are certainly things that may improve replication, so if you start working now with replication with many databases, you are welcome to post feature requests (as separate threads), which could be considered in due course.