 | Summary: Data-Grid Clustering between physically remote distributed sites.
Author: Shay Hassidim, Deputy CTO, GigaSpaces
Recently tested with GigaSpaces version: XAP 7.1
|
| Your Rating: |
Thanks for voting! |
|
|
Please Wait |
Results: |
|
12 |
rates |
Overview
The GigaSpaces WAN Gateway, a solution for synchronizing multiple clusters over the WAN can be found at Multi-Site Replication over the WAN.
Enterprise wide applications require the sharing of data across remote geographical sites. Several instances of the application need to share data, while the communication bandwidth and speed between the different sites is relatively slow. In many cases, due to security and limited bandwidth, each site requires only a portion of the local site data to be replicated to other remote sites. In such cases, data should be filtered before being replicated to other remote sites based on some custom business logic.
Data-Grid synchronization over the WAN could be between:
- A Data-Grid running in one site of the Enterprise and another Data-Grid running on another remote site of the Enterprise.
- Local Data-Grid running within the Enterprise and another Data-Grid running over the Cloud.
- Data-Grid running over Cloud A and another Data-Grid running on Cloud B.
The above is relevant for disaster recovery requirements and for active-active scenarios.
Support for Wide-Area Network (WAN) Data-Grid synchronization is addressed by the following components:
- Asynchronous Replication – GigaSpaces allows batching Space operations and replicating them based on time intervals or amount of operations. The space includes a redo log mechanism that logs operations and entries in an efficient manner and performs a handshake in the source and target space for every replication event. Since the replication is conducted in an asynchronous manner, clients response time is not affected.
- Processing Unit – You may deploy your business logic into GigaSpaces, having the business logic and the Space collocated within the same memory address. The business logic may react when a matching event occurs within the space. This allows you to build a specific business logic that interacts with remote sites, and synchronize activities across the different sites.
- Multiple Unicast-Based Service Lookups – by default, GigaSpaces distributed Lookup Service discovered using multicast without the need to specify a specific host machine. In WAN based environments, multicast might be blocked across the different sites. To resolve this problem, GigaSpaces allows defining multiple machines that host the Lookup Service (aka LOOKUPLOCATORS), and the discovery of these lookups is done via the unicast protocol.
- Mirror Service – the GigaSpaces Mirror Service allows local site nodes to replicate their operations in a reliable asynchronous manner into a central service, whose sole responsibility is to act as the gateway to remote sites. With the Mirror Service, users may perform optimizations, data conversions and transformations before writing the data to the remote sites.
- Data Replication Granularity – WAN environments create the need to define fine-grained replication – sometimes at the object level. GigaSpaces supports space, class and object level (content-based) replication granularity that is decoupled from the actual application business logic and done via external configuration.
- Data Bulk Load – in WAN environments there is a need to pre-load the different nodes prior to their availability for public use. GigaSpaces include a built-in mechanism to pre-load spaces before they are accessible to clients from external resources.
- Master-Local Space or Local View - (Hub & Spoke Architecture) – In WAN environments, you might want to form smart dynamic caches on the client-side that are decoupled from the backend spaces, to store only data relevant for the application instance. This client-cache runs in the client memory address and is updated using streaming techniques with a push/pull model. The local cache can be loaded on-demand or pre-loaded using standard SQL.
- Automatic Data Recovery – GigaSpaces provides a data and event notification registration recovery mechanism, allowing restarted spaces to pre-load data from a partner space.
- Custom Space UID Generation – many systems have their own unique ID generated internally by an application-level ID generator. You may use this application-level ID as the basis for GigaSpaces Object UID, and manipulate the data using this UID.
- Loosely-Coupled Architecture – because data is stored in memory within logical units (spaces) you can construct loosely-coupled architectures, having data distributed without physical relationships in terms of data complexity, machine host names, Network IP or actual space location.
- System-Level Events and Fault-Detection – GigaSpaces "nervous system" – the Service Grid – provides system-level events, allowing external applications and internal components to be aware of the status of the various distributed components, such as their deployment, active mode or failure – and act upon receiving this information. These events can be accessed via JMX API.
- Intuitive Provisioning via a GUI – GigaSpaces provides an extensive enterprise-level monitoring, provisioning, management and administration GUI tool. Using one management console, the user can view and query the status of the deployed services, the In-Memory Data Grid, and space in real time.
This section illustrates several patterns you can apply to construct a WAN-based cluster.
There are three main options:
- Classic Synchronous Replication Synchronization
- Treat all sites as one big site.
- Direct Asynchronous Synchronization
- Viable solution for simple cases.
- Becomes VERY complex to handle with large scale applications.
- Mirror Gateway Synchronization
- Each site running its own data-grid cluster.
- Good for large scale applications.
- Based on the Mirror functionality.
- The recommend approach in most cases.
Classic Synchronous Replication Synchronization
This approach is relevant when the latency between the sites is relativly low where the application interaction with the data-grid is read mostly.
Having a local cache/local view running as part of the client application may improve the read performance avoiding read operation to access data-grid partitions over the WAN.
- Pros
- One cluster - Simple to deploy and manage.
- Local Cache/View improve read activities performance.
- Avoid reading/writing data over the WAN.
- Cons
- Primary and backups spread across both sites.
- No local site HA.
- Synchronous replication over the WAN - replication speed depends on network latency - Impacts Application performance.
- Client communicates with space instance over the WAN - depends on network latency.
Direct Asynchronous Synchronization
Direct Asynchronous Replication - One Partition
This is the simplest topology where you deploy asynchronously replicated spaces, each node located in a different site, serving the local clients. This architecture may be useful when having limited amount of clients and a limited amount of data that can be stored within a single JVM. With a 64 bit JVM you may store relatively large amount of data (10G-50G) without having the JVM garbage collection affecting the Data-Grid behavior.
- Pros
- Clients interacts with local site space member.
- Very simple to Manage.
- Cons
- No local site HA - Recovery from remote site.
- Limited amount of data capacity (one JVM capacity).
- Not scalable. The Data-Grid has a limit for the amount of clients it can serve.
Direct Asynchronous Replication - Multiple Partitions
Replication Over WAN is achieved by using one cluster configuration.
Each partition directly replicates its data and operations in asynchronously manner into its partner.
This approach using two load-balancing groups, and one replication group using the replication matrix or replication filter. With this topology, each partition at each site has a replica space at the other site.
- Pros
- One cluster - no need to manage multiple clusters , or multiple configurations.
- Cons
- No out of the box support.
- Hard to manage. Too many moving parts.
- Each sub-cluster should be exact mirror of the other one.
- Many connections over the WAN.
- Manual static configuration. There is no way to deploy such a cluster via the regular deploy commands.
- No built-in SLA support.
- Requires implementation of a special replication filter.
Mirror Gateway Synchronization
Mirror Gateway Synchronization used to synchronize operations conducted at one local site with operations conducted in a remote site. In this way, you may have different clusters with a different number of partitions at each site. Every site runs a Mirror Gateway that is notified with any changes occurring within the local site Data-Grid.
Mirror Gateway - One way
- Pros
- Simple to Manage.
- Two independent separate clusters.
- Automatic deployment. No static deployment or manual handling.
- Built on top of rock solid Mirror Service (used also for persistency).
- Client interact with local site space members.
- Cons
- Need a custom Mirror and a Dispatcher to be implemented.
- Multiple (but small) amount of connections over the WAN.
- No Active-Active support. Only hot stand-by model.
| An example Mirror Gateway - One Way implementation can be found here |
Mirror Gateway - Two way
- Pros
- Support full Active-Active.
- Two independent separate clusters.
- Automatic deployment. No static deployment or manual handling.
- Built on top of rock solid Mirror Service (used also for persistency).
- Cons
- No conflict resolution support (concurrent updates)
- Need a custom Mirror and a Dispatcher to be implemented.
- Multiple (but small) amount of connections over the WAN.
Mirror Gateway/Dispatcher - Two way
- Pros
- Support full Active-Active.
- Two independent separate clusters.
- Automatic deployment. No static deployment or manual handling.
- Built on top of rock solid Mirror Service (used also for persistency).
- Better control on data flow over the WAN.
- Small amount of connections over the WAN.
- Basic conflict resolution support (concurrent updates)
- Cons
- Need a custom Mirror and a Dispatcher to be implemented.
- Dispatcher should use persistent/clustered dedicated space to survive failure.
Mirror Gateway/Dispatcher - Two way - Architecture
The basic architecture uses a dedicated data-grid to synchronize the different sites. The input into this data-grid comes from a Mirror Service running at each site. The Mirror service running an embedded space acting as a Gateway unit. The interaction between the Mirror and its embedded space (which got a replica at the other site) done in-memory. This provides fast performance and eliminates bottlenecks.
With this approach, the mirror writes its incoming data from the local site spaces into an embedded space. The embedded space is part of a different cluster and has a peer in each remote site. In this way, data is replicated in an asynchronous manner across the WAN. If preferred, the embedded space can use the persistent schema running in LRU cache policy mode , allowing the embedded space to evict data from its cache, and allowing an unlimited amount of data storage without exposing the VM to an out-of-memory problem.
– Two remote sites are shown below, A and B using a 2-way cluster mirror gateway.
– With this approach there are no proxy operations conducted directly against the remote site – i.e. there is no entity at site A that is using site B's clustered proxy to interact directly with spaces at site B over the WAN.
– Once the data has been replicated from Site A's gateway cluster node to Site B's , the gateway cluster node at site B fans it out into the spaces at Site B via its output replication filter. The output replication filter uses Cluster B's clustered proxy – which provides transparent use of Cluster B's load-balancing and failover policies.
– With this approach, there is one replication channel for all site nodes – you can have multiple partitions at site A or B – all using the same replication channel. Since we have a dedicated cluster acting as the replication channel, we can control it very easily using the asynchronous replication settings. This capability is very important in WAN-based environments, because the network utilization might be expensive.
Recursive Synchronization
One important issue to deal with when implementing the Mirror Gateway is recursive synchronization. This occurs when:
- Clients at Site A write data into site A Data-Grid.
- The Mirror Gateway at site A writes data to the gateway node at site B.
- The gateway node at site B writes the data to site B Data-Grid.
- Site B's Data-Grid replicate the data to Site B Mirror gateway.
- Site B's Mirror gateway replicates the data to the gateway node at site A.
- The gateway node at site A writes the data back to site A Data-Grid.
(Site Data-Grid A -> Mirror gateway A -> -> Site B Data-Grid -> Mirror gateway B --> Site A Data-Grid, etc.)
To prevent a recursive synchronization, you should filter out data originating from a remote site when the mirror gateway writes its data into the remote site – i.e. Space Objects originating at site A and replicated from Site A to Site B would not be replicated back from Site B to Site A.
By default, the Space Object UID includes the original container and space name – so you can use the Space Object UID to perform the filtering or use some user defined field to indicate the space object origination.
Mirror Gateway Deployed as a Processing Unit
The Service Grid can monitor the mirror (and the embedded gateway when used) life cycle. It can deploy these and restart these once there is a failure where their activity. To accomplish this, you should to deploy the Mirror Gateway as a Processing Unit. Once the Mirror gateway is started, it constructs the embedded space gateway node as part of its initialization methods.
If the mirror fails, the source space accumulates the data as part of its redo log to be sent to the mirror once will be restarted. There is no need to provide high-availability for the mirror. There is a handshake mechanism between the source space and the mirror. If the mirror implementation BulkDataPersister.executeBulk() fails due to a mirror JVM failure, the source space resends the last batch to the mirror again when the mirror is restarted. Additionally, if the mirror fails, the Service Grid restarts it on some other available GSC – this allows the source space(s) to resend the last replication packet to the mirror (and through it, to the remote site via the embedded gateway cluster space).
Recovery from a Complete Shutdown
If there is a complete shutdown of all spaces, data must be pushed from the relevant data source (database, remote site) into the local site, using a helper service or a program that retrieve the data and writes it into the restarted site. One simple way to do this is via the ManagedDataSource interface implementation, that is activated on startup of the spaces – data loaded into the space via the ManagedDataSource.initialLoad() would not be sent into the mirror. In this case, recursive synchronization would not have occurred.
Mirror Gateway/Dispatcher - Two way - Using a Content Router Device
With this pattern you use a Content Router device (Solace , Cisco , Caringo) to implement the Mirror Gateway described above.
With this pattern the Content Router device is responsible for the following:
- Data Security over the WAN.
- Data Encryption over the WAN.
- Data Compression over the WAN.
- Data Persistency over the WAN (data is stored on the device before it is sent over the network).
- Data Reliability over the WAN.
- Avoid recursive synchronization.
- One to many cluster synchronization with data filtering.
