Overview

A Mirror is a centralized asynchronous replica of data sent by spaces in a cluster. This data is asynchronously batched into the Mirror Service which interacts with its configured data source. The default Mirror Service implementation uses Hibernate (RDBMS) to interact with an external data source. You can implement your own Mirror Service that interacts with any data source of your choice.

The Mirror Service is an implementation of the data-cache persistency modules. The com.gigaspaces.datasource.BulkDataPersister interface is the preferred way to persist asynchronously batched data into a data source. This is done under one transactional context. Other interfaces (see the com.gigaspaces.datasource APIs) persist data and handle each piece of data under its own transactional context. Thus, asynchronous batches maximize the efficiency when interacting with the external data source. This activity is sometimes referred to as a write-behind operation.

By nature, asynchronous replication is not reliable, and data might not be replicated if the source space has terminated unexpectedly. Reliability can be achieved only in the presence of other synchronous replicated spaces. Achieving reliability is later discussed in more detail.

See how failure scenarios are handled by the Mirror Service below.

Mirror Service as Space

The Mirror Service implementation replaces the storage back-end of the space.

The Mirror is not a regular space and should not be interacted with directly. Operations performed directly on the Mirror are unpredictable and might cause inconsistency. Moreover, the GigaSpaces Management Center doesn't show the space's contents or statistics.

Loading the Topology

The Mirror Service is loaded just like any other space, with the additional schema property set with mirror-space-schema.xml as the schema to use.

There are two ways to load a Mirror Service space:

• Using the SpaceFinder API:

  com.j_spaces.core.client.SpaceFinder.find("/./mirror-service?schema=mirror);

• Using the gsInstance CLI:

  gsInstance "/./mirror-service?schema=mirror"

The Mirror Service configuration only requires that other cluster members know that it is enabled. Cluster members do not need to explicitly define the Mirror as a member in their replication group.

There are two ways to enable the Mirror Service:

• Using the mirror URL property - the mirror property should be set for the members that interact with the Mirror Service. You can have some of your cluster members mirror their data, and some not. The ones you want to mirror should have the mirror URL property used when starting the space.
  

  gsInstance "/./mySpace?cluster_schema=primary_backup&total_members=1,1&id=1&mirror"
  gsInstance "/./mySpace?cluster_schema=primary_backup&total_members=1,1&id=1&backup_id=1&mirror"

  
  
  The spaces above are using the default space schema (default-space-schema.xml).
• Enabling the mirror-service.enabled tag in the cluster schema - you can simply enable the Mirror Service for all members of this cluster by setting the mirror-service.enabled tag value to true. With this approach, your URLs can remain unchanged - i.e., without appending the mirror URL property.

  <mirror-service>
  	<enabled>true</enabled>
  	...
  </mirror-service>

Viewing the Topology

The GigaSpaces Management Center displays the Mirror Service space in the Cluster view alongside the replication group's cluster members. The Mirror Service, like any other member, has its status displayed (alive - yellow nodes, and unreachable - green nodes) which allows you to provision and track failures or network disruptions.

Configuration

The Mirror Service can be configured using of the mirror-space-schema.xml. The external-data-source section in this XML specifies the data-source-class and some inherent properties which communicate what is supported by the implementation.

Overriding the Mirror Service Implementation

By default, the data-source-class is configured to use Hibernate (RDBMS) to interact with an external data source. When implementing your own Mirror Service, your should override this tag's value with the full class name, for example:

<external-data-source>
	<!-- default is com.gigaspaces.datasource.hibernate.HibernateDataSource -->
	<data-source-class>com.company.datasource.MyMirrorDataSource</data-source-class>
	...           		
</external-data-source>

The Mirror com.gigaspaces.datasource.BulkDataPersister implementation might be called by multiple threads. Your custom Mirror implementation should be thread safe to handle multiple concurrent invocations. If your implementation accessing a database, makes sure it is using a database connection pool. The recommendation relevant for Mirror (Write behind), Write Through or Read Through scenarios.

Communicating Properties

The Mirror Service properties are derived from those configurable for an external data source, see Settings & Configuration.

Configuring Replication

The mirror-service block in the *-cluster.xml file specifies the interaction between the replication group members and the Mirror Service.

• The enabled element flags the Mirror Service on and off. Once this property value is true all cluster members will be Mirror aware - i.e. will be delegating their activities into the Mirror service.
• The lookup url is used to locate the centralized asynchronous Mirror Service. In case you change the name of the Mirror Service you should modify this parameter value to facilitate the correct Mirror service URL.
• The bulk-size element represents the amount of operations to be transmitted in one bulk (in quantity and not actual memory size) from an active primary to the Mirror Service.
• The transmission trigger – when either the interval-millis times out, or the interval-opers is reached, replication is performed.
• replication reconciliation mode – the repl-original-state should be enabled. This ensues that write/take operations or multiple updates of the same entry will be sent to the mirror and not will be discarded when sent within the same batch.

The default values are as follows:

<mirror-service>
	<enabled>false</enabled>
	<url>jini://*/mirror-service_container/mirror-service</url>
	<bulk-size>100</bulk-size>
	<interval-millis>2000</interval-millis>
	<interval-opers>100</interval-opers>
</mirror-service>

<repl-policy>
	<repl-original-state>true</repl-original-state>
</repl-policy>

The Mirror Service may receive replication events from multiple active primary partitions. Each active partition sends its operations to the Mirror service via dedicated replication channel. The Mirror handles incoming replication requests simultaneously.

Custom Mirror Implementation

See below example (MirrorBench class) for a Custom Mirror implementation. This example calculates the throughput events are sent to the Mirror from the primary spaces. Since the events are sent from the primary spaces in asynchronous periodic manner, the throughput will be oscillating between small and large amount of events sent per second - i.e. there would be durations of time where the throughput will be high and durations of time where the throughput will be low.

Here is illustration of the expected behavior:

The MirrorBench implements the BulkDataPersister and the ManagedDataSource interfaces.
The MirrorBench.executeBulk method counts the incoming BulkItems and print the throughput every 1000 BulkItems arrival.
Since the MirrorBench.executeBulk method might be called by multiple threads (concurrent replication threads - one per partition into the Mirror), the counter implemented using the AtomicInteger.

package com.j_spaces.examples.datasource;
import java.util.List;
import java.util.Properties;
import java.util.concurrent.atomic.AtomicInteger;
import com.gigaspaces.datasource.BulkDataPersister;
import com.gigaspaces.datasource.BulkItem;
import com.gigaspaces.datasource.DataIterator;
import com.gigaspaces.datasource.DataSourceException;
import com.gigaspaces.datasource.ManagedDataSource;

public class MirrorBench implements BulkDataPersister,ManagedDataSource<Person> {
	AtomicInteger count = new AtomicInteger();
	long lastTime = System.currentTimeMillis();
	int lastVal =0;
	public void executeBulk(List<BulkItem> bulk) throws DataSourceException {
		count.getAndAdd(bulk.size());
		int val = count.intValue();
		if (val - lastVal >1000)
		{
			lastVal = val ;
			long dur = System.currentTimeMillis() - lastTime;
			lastTime = System.currentTimeMillis();			
			double tp = 1000 / (double)dur * 1000;
			System.out.println("got:" + val  + " operations"+ " TP[sec]:" + tp);
		}
	}

	public void init(Properties arg) throws DataSourceException {
		System.out.println("init" + arg);
	}

	public DataIterator<Person> initialLoad() throws DataSourceException {
		return null;
	}

	public void shutdown() throws DataSourceException {
	}
}

The MirrorBench would be configured using the following properties (mirror.properties file):

space-config.persistent.enabled=true
space-config.persistent.StorageAdapterClass=com.j_spaces.sadapter.datasource.DataAdapter
space-config.external-data-source.data-source-class=com.j_spaces.examples.datasource.MirrorBench
space-config.engine.cache_policy=0
space-config.external-data-source.init-properties-file=/config/mirror/hibernate.properties

The MirrorBench would be started using:

gsInstance "/./mirror-service?schema=mirror&properties=mirror" "..;..\classes;%JARS%"

The Partitioned Clustered space would be started using:

gsInstance "/./mySpace?cluster_schema=partitioned-sync2backup&total_members=2,1&id=1&mirror"
gsInstance "/./mySpace?cluster_schema=partitioned-sync2backup&total_members=2,1&id=2&mirror"
gsInstance "/./mySpace?cluster_schema=partitioned-sync2backup&total_members=2,1&id=1&backup_id=1&mirror"
gsInstance "/./mySpace?cluster_schema=partitioned-sync2backup&total_members=2,1&id=2&backup_id=1&mirror"

The Client feeder application that performs the write operations will have the following:

IJSpace space = (IJSpace) SpaceFinder.find("jini://*/*/mySpace");
long lastTime = System.currentTimeMillis();

for (int i = 0; i < 100000; i++)
{
	space.write (new Person("first-" + i, "last-" + i, new Integer(i)) , null , Lease.FOREVER);
	if (i % 1000  ==0 )
	{
		long dur = System.currentTimeMillis() - lastTime;
		lastTime = System.currentTimeMillis();
		double tp = 1000 / (double)dur * 1000;
		System.out.println("wrote:" + i + " objects to space TP:" +tp );
	}
}

Usage Scenarios

Writing Asynchronously to Mirror Data Source

The following is a schematic flow of a synchronous replicated cluster with 3 members, which are communicating with a Mirror Service:

The topology was loaded with the following command lines:

gsInstance "/./mirror-service?schema=mirror"
gsInstance "/./mySpace?cluster_schema=sync_replicated&total_members=3&id=1&mirror"
gsInstance "/./mySpace?cluster_schema=sync_replicated&total_members=3&id=2&mirror"
gsInstance "/./mySpace?cluster_schema=sync_replicated&total_members=3&id=3&mirror"

Reading from Mirror Data Source

The Mirror Service space is used to asynchronously persist data into the data source. As noted above, the Mirror is not a regular space and should not be interacted with directly. Thus, data can't be read from the data source using the Mirror Service space. Nonetheless, the data might be read by other spaces which are configured with an external data source.

The default Hibernate (RDBMS) implementation implements all external data source interfaces, including the ones used by the Mirror Service space (i.e. com.gigaspaces.datasource.BulkDataPersister). You might have noticed this, since we didn't change the default data-source-class. The only thing that needs to be altered is the usage property.

By default, the usage property in the persistent-space-schema.xml is set to read-write. This means that the space members can also perform write operations into the data source.

For consistency reasons, it is important that the spaces reading from the external data source are configured to read-only – unless your application logic requires differently.

<external-data-source>
	...
	<!-- data source usage mode - options - read-write,read-only -->
	<!-- default is read-write-->
	<usage>read-only</usage>
	...
</external-data-source>

The cluster schema needs to be configured to use an external data source which, when dealing with a Mirror, is central to the cluster.

<cache-loader>
  <external-data-source>true</external-data-source> 
  <central-data-source>true</central-data-source>
</cache-loader>

Here is a schematic flow of how a Mirror Service space asynchronously receives data to persist into an external data source, while the cluster is synchronously reading data directly from it.

The topology was loaded with the following command lines:

gsInstance "/./mirror-service?schema=mirror"
gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=sync_replicated&total_members=3&id=1&mirror"
gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=sync_replicated&total_members=3&id=2&mirror"
gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=sync_replicated&total_members=3&id=3&mirror"

The spaces above are using the persistent space schema (persistent-space-schema.xml), which needs to be configured with read-only and overridden by the \config\datasource.properties file. The properties file defines the external data source used by the persistent space. Also, the sync_replicated-cluster-schema.xsl has been configured with external-data-source and central-data-source.

Partitioning Over Central Mirror Data Source

When partitioning data, each partition asynchronously replicates data into the Mirror Service. Each partition can read back data that belongs to it (according to the load-balancing policy defined).

Here is a schematic flow of how two partitions (each a primary-backup pair) asynchronously interact with an external data source:

The topology was loaded with the following command lines:

gsInstance "/./mirror-service?schema=mirror"

gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=partitioned-sync2backup&total_members=2,1&id=1&mirror"
gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=partitioned-sync2backup&total_members=2,1&id=1&backup_id=1&mirror"

gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=partitioned-sync2backup&total_members=2,1&id=2&mirror"
gsInstance "/./mySpace?schema=persistent&properties=datasource&cluster_schema=partitioned-sync2backup&total_members=2,1&id=2&backup_id=1&mirror"

The spaces are using the persistent space schema (persistent-space-schema.xml) which needs to be configured with read-only and overridden by the \config\datasource.properties file. The properties file defines the external data source used by the persistent space. Also, the partitioned-sync2backup-cluster-schema.xsl has been configured with external-data-source and central-data-source.

Achieving Reliability

Since asynchronous replication is not reliable by definition, achieving reliability requires at least one reliable member to sync with. Reliability is compromised when data is not asynchronously replicated due to an unexpected termination of a replicating source member. Therefore, reliability can be achieved only in the presence of other synchronous replicated spaces. As long as there is a synchronous member around, it asynchronously replicates data (and losses) into the Mirror Service space.

It is recommended to use reliable asynchronous replication with GigaSpaces 6.0.1 and onwards, since this version includes many optimizations and performance enhancements.

Configuration

All synchronous cluster schemas contain a reliable tag as part of their async-replication block, which is set to true.

The default value is false - which has been applied in all asynchronous cluster schemas.

<repl-policy>
	<replication-mode>sync</replication-mode>
	<recovery>true</recovery>
	...
	<async-replication>
		...
		<reliable>true</reliable>
	</async-replication>
</repl-policy>

Members of a replication group recover from each other when they load. This is true for all-in-cache members or persistent members with no data. Thus, to ensure recovery, the recovery tag must be set to true.

For ensuring reliable asynchronous replication replication-mode tag must be sync at all cluster schemas.

Recovery from Data Loss

When joining a Mirror Service to a synchronous cluster, e.g., primary_backup-cluster-schema.xml, the backups serve as the reliable counterpart from which a Mirror Service space can obtain data in the absence of the initial primary space.

If the reliable tag is set to false, and the primary is terminated prior to the replication of data into the Mirror Service space, data loss is evident. The new elected primary (a former backup) does not replicate data received via replication channels.

A rule of thumb - only a member which is elected as primary transmits data into the Mirror Service space.

Reliability ensures that the backup (once it becomes primary) retransmits replicated data into the Mirror Service space. This 'playback' consists of replicated data for which the backup did not receive acknowledgments. As long as the topology is up and running, acknowledgments are sent from the Mirror Service space to the primary, which forwards them to the backup. If at any time, the primary terminates, the backup continues to replicate, beginning from the last acknowledgment sent, thus ensuring reliability.

Usage Examples

Mirror Service examples are part of the external data source examples.

In addition, specific Mirror Service usage scenarios are presented as part of the Mirror Service examples section.

Considerations

External data source considerations also apply to the Mirror Service.

• The Mirror Service is a single space which joins a replication group. The Mirror Service is not a clustered space or part of the replication group declaration.
• When the Mirror Service is loaded, it does not perform memory recovery. See the reliability section for more details.
• Transient Entries are not persisted into the data source - just like in any other persistent mode.
• Unlike other external data source interfaces, the com.gigaspaces.datasource.BulkDataPersister interface doesn't differentiate between Space API and Map API operations. The call to com.gigaspaces.datasource.BulkItem.getItem() returns either the user entity when using the Space API, or the java.util.Map.Entry when using the Map API.

Known Issues

External data source known issues also apply to the Mirror Service.

• The partitioned-sync2backup-cluster-schema.xsl should be used instead of the partitioned-cluster-schema.xsl. However, if your partition has no backups, you must remove the <fail-over-policy> or have the failover <policy-type> defined as fail-to-backup. Not having these settings results in an incorrect data load from the database, and incorrect data in the partitions.
• Your Entry must have getters and setters for all public fields.

Troubleshooting

Log Messages

The external data source logging level can be modified as part of the <GigaSpaces Root>\config\gs_logging.properties file. By default, it is set to java.util.logging.Level.INFO:

com.gigaspaces.persistent.level = INFO

Logging is divided according to java.util.logging.Level as follows:

Configuration messages when loading the default Mirror Service at a CONFIG level:

> gsInstance "/./mirror-service?schema=mirror&groups=mygroup"
...

CONFIG [com.gigaspaces.persistent]:
schema-xml configuration:
<external-data-source>
         <data-source-class>com.gigaspaces.datasource.hibernate.HibernateDataSource</data-source-class>
         <data-class>class java.lang.Object</data-class>
         <supports-inheritance>true</supports-inheritance>
         <supports-version>false</supports-version>
         <usage>read-write</usage>
</external-data-source>
cluster-xml configuration:
<cache-loader>
         <external-data-source>false</external-data-source>
         <central-data-source>false</central-data-source>
</cache-loader>

INFO messages displayed when loading one of the replication group members connected to a Mirror Service:

> gsInstance "/./mySpace?cluster_schema=sync_replicated&total_members=2&id=1&mirror&groups=mygroup"
...

INFO [com.gigaspaces.core.cluster.replication]:
        Mirror Service Connector : Started
                Source space     : mySpace_container1:mySpace
                Mirror URL       : jini://*/mirror-service_container/mirror-service?groups=mygroup
                BulkSize         : 100
                IntervalMillis   : 2000
                IntervalOpers    : 100
...

Replicator: Connection established with target space
[  source space: mySpace_container1:mySpace  ]
[  target space: mirror-service_container:mirror-service ; target
space url: jini://*/mirror-service_container/mirror-service?groups=mygroup&timeout=5000&state=started  ]
...

INFO messages displayed when the Mirror Service establishes a connection with a replication group member:

INFO [com.gigaspaces.core.cluster.replication]:
Joined new [mySpace_container2:mySpace] member to the mirror-service_container:mirror-service Mirror Service.

FINER messages displayed when the Mirror Service receives an asynchronous bulk to persist:

FINER [com.gigaspaces.persistent]:  ENTRY [
BulkDataItem<Op: WRITE, IGSEntry<com.j_spaces.examples.datasource.Person,
    UID: -1989577544^39^0^0^0, Fields: firstName: first-0, id: 0, lastName: last-0, >>,
BulkDataItem<Op: WRITE, IGSEntry<com.j_spaces.examples.datasource.Person,
    UID: -1989577544^39^1^0^0, Fields: firstName: first-1, id: 1, lastName: last-1, >>,
...
BulkDataItem<Op: UPDATE, IGSEntry<com.j_spaces.examples.datasource.Person,
    UID: -1989577544^39^7^0^0, Fields: firstName: first-7 update, id: 7, lastName: last-7 update, >>,
BulkDataItem<Op: REMOVE, IGSEntry<com.j_spaces.examples.datasource.Person,
    UID: -1989577544^39^0^0^0, Fields: firstName: null, id: 0, lastName: null, >>
...
]

Handling Failover

This section describes how GigaSpaces mirror service handles different failure scenarios. The following table lists the services involved, and how the failure is handled in the cluster.

Active services are green, while failed services are red.

Unlikely Failure Scenarios

The following failure scenarios are highly unlikely. However, it might be useful to understand how such scenarios are handled by GigaSpaces. This is detailed in the table below.

Active services are green, while failed services are red.