Skip to content
GigaSpaces Logo GigaSpaces Logo
  • Products, Solutions & Roles
    • Products
      • InsightEdge Portfolio
        • Smart Cache
        • Smart ODS
        • Smart Augmented Transactions
        • Compare InsightEdge Products
      • GigaSpaces Cloud
    • Solutions
      • Industry
        • Financial Services
        • Insurance
        • Retail and eCommerce
        • Telecommunications
        • Transportations
      • Technical
        • Operational BI
        • Mainframe & AS/400 Modernization
        • In Memory Data Grid
        • Transactional and Analytical Processing (HTAP)
        • Hybrid Cloud Data Fabric
        • Multi-Tiered Storage
        • Kubernetes Deployment
        • Streaming Analytics for Stateful Apps
    • Roles
      • Architects
      • CXOs
      • Product Team
    • Building a Successful Hybrid and Multicloud Strategy
      vid-icon Guide

      Learn how to build and deploy a successful hybrid and multicloud strategy to achieve: agility and scalability, faster go-to-market, application acceleration, legacy modernization, and more.

      DOWNLOAD
    • Contact us
    • Try Free
  • Resources
    • Resource Hub
      • Webinars
      • Demos
      • Solution Briefs & Whitepapers
      • Case Studies
      • Benchmarks
      • ROI Calculators
      • Analyst Reports
      • eBooks
    • col2
      • Featured Case Studies
        • Mainframe Offload with Groupe PSA
        • Digital Transformation with Avanza Bank
        • High Peak Handling with PriceRunner
        • Optimizing Business Communications with Avaya
    • col3
      • Blog
      • Technical Documentation
    • Live Webinar: Enable Digital Transformation With a High Performing Data Platform
      article-icon Live Webinar | March 11 - 9 AM EST/3 PM CET

      Join Capgemini and GigaSpaces for a discussion on the latest modernization trends for enterprises that are embarking on digital and business transformation

      REGISTER NOW
    • Contact Us
    • Try Free
  • Company
    • Col1
      • About
      • Customers
      • Management
      • Board Members
      • Investors
      • Events
      • News
      • Careers
    • col2
      • Partners
      • OEM Partners
      • System Integrators
      • Technology Partners
      • Value Added Resellers
    • col3
      • Support & Services
      • University
      • Services
      • Support
    • GigaSpaces is Headed to CDAO 2021
      calendar-icon Event | March 2-4

      Join us at CDAO 2021, the premier Virtual Summit for Data and Analytics Leaders. We'll be moderating "Transforming Financial Services to a Customer-Centric Business", alongside USAA Bank, Regions Bank, and Capital Group.

      SIGN UP NOW
    • Contact Us
    • Try Free
  • Contact Us
  • Try Free
  • Products, Solutions & Roles
    • Products
      • InsightEdge Portfolio
        • Smart Cache
        • Smart ODS
        • Smart Augmented Transactions
        • Compare InsightEdge Products
      • GigaSpaces Cloud
    • Solutions
      • Industry
        • Financial Services
        • Insurance
        • Retail and eCommerce
        • Telecommunications
        • Transportations
      • Technical
        • Operational BI
        • Mainframe & AS/400 Modernization
        • In Memory Data Grid
        • Transactional and Analytical Processing (HTAP)
        • Hybrid Cloud Data Fabric
        • Multi-Tiered Storage
        • Kubernetes Deployment
        • Streaming Analytics for Stateful Apps
    • Roles
      • Architects
      • CXOs
      • Product Teams
    • Contact Us
    • Try Free
  • Resources
    • Resource Hub
      • Webinars
      • Demos
      • Solution Briefs & Whitepapers
      • Case Studies
      • Benchmarks
      • ROI Calculators
      • Analyst Reports
      • eBooks
    • Featured Case Studies
      • Mainframe Offload with Groupe PSA
      • Digital Transformation with Avanza Bank
      • High Peak Handling with PriceRunner
      • Optimizing Business Communications with Avaya
    • Blog
    • Technical Documentation
    • Contact Us
    • Try Free
  • Company
    • About
    • Management
    • Customers
    • Board Members
    • Investors
    • Events
    • News
    • Careers
    • Partners
      • OEM Partners
      • System Integrators
      • Technology Partners
      • Value Added Resellers
    • Support & Services
      • University
      • Services
      • Support
  • Contact Us
  • Try Free

NoSQLNoCompromises

Subscribe to our blog!

Subscribe for Updates
Close
Back

NoSQLNoCompromises

GigaSpaces March 26, 2014
7 minutes read

NoSQL databases provide vast storage and high availability, but at the cost of losing transactions, relational integrity, consistency, and read performance. This post presents an architecture that combines and in-memory datagrid as a high performance transactional layer to a NoSQL database, providing a complete application platform with the high scalability of NoSQL, with the transactionality, real time event processing, distributed execution, and ultra-low latencies of the XAP data grid.

===================================================================
Why NoSQL?
Business have been driven by to NoSQL data stores because of several virtues, including economics, very large storage capacity, elasticity, high availability, reliability, write scaling, and a flexible data model. All of these virtues are purchased at a cost, which to a large degree can be tied to moving to a distributed architecture.

The Compromises
In order to scale horizontally on cheap hardware (a key NoSQL attribute), several compromises had to be made, including abandoning some highly valuable characteristics of relational databases. Among them, read consistency, transactions (and transactional isolation), stored procedure, triggers, and robust security. Since data it typically replicated across multiple cluster nodes, write operations cannot be accommodated across all replicas with acceptable latency. Comparable latencies in XAP are minimal because synchronous writes are to RAM only. This means that “eventual consistency” approaches are employed to synchronize data nodes after a write has completed. In practical terms it means that a reader is not guaranteed to get the most recent version of data. The lack of transactions means that multiple arbitrary data elements can not be modified or created atomically. This opens the door to readers seeing partial writes. The lack of stored procedures makes anything but the most simplistic data manipulation inefficient, and the lack of triggers make even simplistic event processing impossible.

The main consequence of the “compromises” is that use cases are limited. Highly concurrent transactional applications need not apply, whereas the class of massive, unstructured “information gathering” applications is well served. A typical approach to these compromises is to simply partition the system architecture into transactional and NoSQL divisions, with perhaps a data scrubbing/enrichment connection between them. For example, raw data from user web interactions is written to NoSQL, and a batch process performs ETL on it to make it suitable for relational (and/or OLAP) clients. This can manifest itself as two completely separate stacks, or a hybrid.

The problem with these solutions is they are complex, typically multi-vendor, difficult to scale, not elastic, and inefficient. Complex because of many products, multiple interfaces to manage, multiple domains of expertise, multiple upgrade cycles etc.. It is difficult to scale the entire stack because each tier scales independently in different ways. Since each part of the architecture is not elastic in it’s own right, the whole is also inelastic. Inefficiency in processing is caused by multiple physical tiers and boundaries that messages must be marshalled and transmitted across.

But what if we could access NoSQL data transactionally, retaining its scalable, elastic and highly available nature, while providing stored procedures and triggers? Delivering all of this, along with in-memory speed, is what can be achieved by integrating the Gigaspaces XAP in-memory application platform with a NoSQL database.

What Does No Compromises NoSQL Look Like?
By combining the virtues of the XAP in-memory platform with NoSQL storage, a system (not just a data store) of high scalability and low latency emerges. The attributes of the “no compromises” architecture:
ACID Transactions
Fully consistent reads
Horizontal scalability
Collocated, native language business logic (procs)
Real time eventing (“triggers”/”continuous query”)
Complex event processing
SQL Queries
Elastic & self healing
Reads in the low 10s microsecond range
Redundant writes in the low 100s microsecond range
Role based security for both data and management

The idea is simple. A two product layered architecture with data flowing through the in-memory cluster. As data flows through the in-memory subsystem, a portion is retained (typically in LRU fashion, but not necessarily) in memory. This retained data serves as a very low latency distributed cache for reads. Writers will experience data grid latencies across the distributed in memory data set, as writes forwarded to the NoSQL store are asynchronous. Clients accessing the grid for transactional processing will experience full ACID support. Data flowing through the in-memory subsystem can trigger internal distributed or external event handlers for complex event processing/enrichment, and distributed RPCs on the in-memory cluster can provide efficient business process level services (equivalent to distributed stored procedures) to clients.

The two technologies are a natural complement to each other, with XAP providing in-memory processing, transactions, elasticity, and high availability, and NoSQL providing the economical storage of vast amounts (100s of terabytes) of data. The architecture is greatly simplified and capable compared to the partitioned architecture “workarounds” mentioned earlier, with only two products necessary (as XAP even provides a JEE web container).

NoSQL Compronmises

How XAP Makes It Possible
XAP is a distributed in-memory storage and processing platform. XAP partitions data and processing across multiple physical or virtual nodes to provide the highest operational performance possible. Data is stored redundantly in memory for safety and availability, and is fully transactional (including XA support). From the perspective of a system client, the memory across all machines appears as a continuous large memory space, and supports distributed RPC, distributed task execution, messaging, and event driven programming paradigms. These combined features lowers system complexity be reducing the number of physical tier and system interfaces. XAP supports runtime resizing of the cluster (elasticity), strongly typed data (objects), and weakly typed data (documents).

XAP also supports persistence to external data stores, and this includes conventional relational databases as well as NoSQL stores. Modifications to in-memory storage are queued for asynchronous write-behind behavior, and cache misses are forwarded to the underlying store. XAP also integrates dynamic type introduction and index creation with underlying stores for a seamless integration.

In addition to NoSQL persistence, XAP can perform off-heap persistence to flash memory. This allows XAP to present a potentially vast (100s TB) flash backed store, in combination with conventional RAM and NoSQL. This capability enables a 3 tier memory architecture that can be tuned to application memory cost vs capacity/latency requirements. The flash integration will be available in the upcoming XAP 10 release.
Conclusion
The combination of XAP and NoSQL is powerful because of their similarities and their differences. From the availability, scalability, and elasticity perspective, the technologies have similar virtues, with NoSQL providing the long term and very large scale persistence. From the user use case perspective, XAP fills a large gap by providing conventional support for transactions, effectively creating a bridge between transactional clients and NoSQL data, and in many cases eliminating the need for a relational database at all. From the performance perspective, XAP provides in-memory performance for both reads and writes, as well as real time event processing and distributed colocated logic. From the system management and complexity perspective, only two products are required to provide a complete application stack.

The value of a two tier, ultra-high performance, highly available, self healing, and horizontally scalable system is enormous, not only on the technical level, but also on the business TCO level, especially when large relational databases and related software can be reduced or eliminated. This capability is available and in production today, with current official support for Cassandra and MongoDB, and more on the way.

CATEGORIES

  • Application Architecture
  • Application Performance
  • application scalability
  • architecture
  • Caching
  • distributed computing
  • GigaSpaces
  • High Availability
  • Java
  • low latency
  • MySQL
  • NoSQL
  • Scalability
  • Scalable Architecture
  • XAP
GigaSpaces

All Posts (23)

YOU MAY ALSO LIKE

December 19, 2007

Get some “space” over the…
2 minutes read

May 25, 2011

GigaSpaces Citrix integration on top…
3 minutes read

March 27, 2008

Amazon gets more elastic
1 minutes read
  • Copied to clipboard

PRODUCTS, SOLUTIONS & ROLES

  • Products
  • InsightEdge Portfolio
    • Smart Cache
    • Smart ODS
    • Smart Augmented Transactions
    • Compare InsightEdge Products
  • GigaSpaces Cloud
  • Roles
  • Architects
  • CXOs
  • Product Teams
  • Solutions
  • Industry
    • Financial Services
    • Insurance
    • Retail and eCommerce
    • Telecommunications
    • Transportation
  • Technical
    • Operational BI
    • Mainframe & AS/400 Modernization
    • In Memory Data Grid
    • HTAP
    • Hybrid Cloud Data Fabric
    • Multi-Tiered Storage
    • Kubernetes Deployment
    • Streaming Analytics for Stateful Apps

RESOURCES

  • Resource Hub
  • Webinars
  • Blogs
  • Demos
  • Solution Briefs & Whitepapers
  • Case Studies
  • Benchmarks
  • ROI Calculators
  • Analyst Reports
  • eBooks
  • Technical Documentation
  • Featured Case Studies
  • Mainframe Offload with Groupe PSA
  • Digital Transformation with Avanza Bank
  • High Peak Handling with PriceRunner
  • Optimizing Business Communications with Avaya

COMPANY

  • About
  • Customers
  • Management
  • Board Members
  • Investors
  • News
  • Events
  • Careers
  • Contact Us
  • Book A Demo
  • Try GigaSpaces For Free
  • Partners
  • OEM Partners
  • System Integrators
  • Value Added Resellers
  • Technology Partners
  • Support & Services
  • University
  • Services
  • Support
Copyright © GigaSpaces 2021 All rights reserved | Privacy Policy
LinkedInTwitterFacebookYouTube

Contact Us