Failover from Master to Slave

Good morning!

I am interested in ObjectDB for a client project I am currently working on. In order to evaluate your tool I would like to know if your product has the ability to failover from a Master Database to a Slave Database and then recover back to the master when the master comes back online.

I don't see any documentation about automatic failover.

Thanks

Rich

Hi!

You can read more about this here: https://www.objectdb.com/java/jpa/tool/replication

The disadvantage is that slaves can be used just for reading.

Hi Z,

Thank you for the response. I did read the article about replication which helps with scaling reads.

My question is really about fault tolerance. I cannot discuss exactly the scenario because the client work I have is confidential, but essentially we are building an e-commerce solution.

So if I am understanding correctly. We could configure ObjectDB with replication, but if the Master database goes down the application will still be able to read from the slaves but writes will not be allowed until the master is restored.

So in our e-commerce solution that would mean if the master goes down customers could continue to browse say the item catalog but would not be able to add an item to the shopping cart because that would require a database write.

If the master is down for even an hour that could mean that could mean thousands of dollars in lost revenue.

-RG

Currently automatic switching from a failed master database to a replicated database is indeed limited by ObjectDB to read only mode.

But there is another option.  Your application can switch to the replicated database by changing connection url, i.e. by connecting to the replicated database using its own url. In that case, the new connection is an ordinary connection that is not limited to read only mode. This requires some operations by your application: identifying a failure, moving to the replicated database url, switching the master and slave roles (in order to replicate changes in the other direction from the slave to the master), and finally switching back to the master url when ready.

For support: do you have an idea how change the urls when in a Java EE JTA environment without restarting the app? (where all data is taken on app deployment from persistence.xml)

Here are relevant discussions and solutions:

https://stackoverflow.com/questions/9957179/jpa-dynamic-persistence-unit-name

https://stackoverflow.com/questions/8784302/dynamic-persistencecontext-unitname-attribute-for-container-based-entitymanager

For support: thank you very much. It seems to be possible, but with some work & exception handling.

The big question is: how do we make sure we don't corrupt the databases? Let's take the following scenario:

1. DB A is master, DB B is A's slave
2. 3 application servers run an application that uses A as master
3. Server of DB A fails.
4. Applications switch to DB B as master
5. Applications are restarted due to update/maintenance. When they start DB A is selected as master again, because they are configured to use DB A if it works.
6. Problem: DB A won't have all the modifications that were added into B while it was down.

One possible solution: when we configure persistence units in persistence.xml, we give 2 urls for each persistence unit. During execution/runtime, ObjectDB client picks the connection that represents the Master DB. (there should be a way in which DB servers know if they're masters or not)

If DB A fails, ObjectDB should automatically switch to the second URL and promote it as master, and DB A should be automatically transformed into slave when it comes online again. In this way, even though applications are restarted, ObjectDB will always connect just to the one that represents the Master, avoiding data corruption.

Does it make sense? Do you see the problem?

The problem is clear. This solution could work with multiple applications only if all the applications always select the correct database, so you must have a way to mark switching from master to slave.

Possible solution is to create a marker file (even empty) on the master server file system and on failure to delete that file and create another file on the slave server. Application will always check that file before selecting which server to connect to.

File management on ObjectDB Servers is supported by the Utilities.getServerFileSystem method.

I really don't think the file marker solution will work at all.

Here is the reason: in some cases failures are not consistent. Let's say the NIC of Database A is failing but the failures are intermittent.

In that case App Server 1 might be communicating with Database A then decide to failover to B due to a timeout.

So A fails over and B is told to take the file marker. How is App Server 1 supposed to remove the "I'm the Master" file marker from Database A if the NIC is not working properly.

Let's say App Server 2 is idle during this whole time but then wakes up during a period of increased load. It tried to connect to A and it does successfully because A is not truly down but the NIC is failing. So it sees A and A still has a master file marker so it decides to write to A.

Well there you go .. you have a high probability of database corruption.

I would suggest going with the Hadoop solution which is to effectively kill a server that is either slow to respond or not working properly. That would prevent this scenario.

This is correct, a file marker alone is not enough. You must make sure that all the applications stop using that server. One way is to kill the server. Another way is to check the marker file routinely by a background process of each application. Write operations to the new master server may have to be delayed a few seconds to let all live applications see the change, but a delay of a few seconds should be acceptable in most applications.

Hello,

ragarding switching from Master to Slave, does it require objectDB restart?

Generally is there a way how to change configuration in runtime?

Thanks

Marta

Any configuration change requires restarting the server.