Contents

1. Overview

2. Caching: What, Why, and Where

3. Caching Pattern Types

4. Eviction Policies

5. Caching Solution Types

6. Why Consider Database Caching

7. Utilizing Distributed Caching

8. Summary of Caching Benefits

9. Caching Case Studies

10. Looking Ahead

1. Overview

Enterprises are grappling with escalating data volumes as they strive to deliver innovative apps and services. This surge in data is exacerbating performance issues in traditional data management infrastructures.

To address these challenges, caching emerges as a critical tool to accelerate data access, enhance application responsiveness, and optimize costs. This post delves into caching concepts, use cases, and popular tools to empower data architects and engineers in building high-performance data solutions.

2. Caching: What, Why, and Where

Caching is a technique used to manage frequently accessed data in-memory. The method facilitates access to data while also reducing the workload on the main data stores. Not only can caching enhance performance; it can also improve the scalability and availability of an application.

Caching has a proven track record of helping companies of all sizes improve application performance. And in fact, advanced caching services provide much more than just a quick key-value cache. But in order to understand what caching is all about and where it is headed, let’s first review the basics.

The three primary types of caching are: 

• Key-value caching: the key is a unique identifier for the data, and the value is the data itself, see Key-Value Caching below
• Object caching: stores entire objects instead of just key-value pairs, and is usually employed for storing complex data structures
• Distributed caching: stores data across multiple servers, offering enhanced scalability and redundancy

3. Caching Pattern Types

Different caching solutions support different implementations, each with its pros and cons. Data architects and software engineers must therefore consider which solution supports these out of the box and the design that best suits the team’s needs prior to implementation.

Read-Through (Lazy Loading)

Applications using the read-through caching pattern essentially treat the cache as the main database. When looking for a specific ID, the application will hit the cache first; if the data is missing, then the cache will be responsible for pulling the data from the main database and updating it.

How applications use read-through caching pattern (aka lazy loading)

Write-Through

The write-through caching pattern is similar to the read-through in that instead of writing directly to the main database, the application writes to the cache as if it were the main database. The cache then writes to the main database. This write to the cache is done synchronously and in turn, impacts the write latency.

Write-through caching pattern

Write-Behind

Write-behind caching takes a similar approach to write-through, except that writing to the main database is done asynchronously. This means write latency is only limited to the cache latency. Because it is volatile, however, there is a risk of data loss if the cache is down. That being said, some advanced caching solutions provide both performance and no data loss in case of failure.

Write-behind caching pattern

4. Eviction Policies

A cache behaves differently than a database which stores all the data ever entered. This is due to the balance of cost versus performance, as the fastest data tier (RAM) is more expensive than slower tiers (SSDs and hard disks). Instead, in such cases, an eviction policy can be used to prevent the cache size from going over the maximum limit. The cache achieves this by removing elements according to the rules set by the eviction policy. There are several methods to implement, depending on what the data architect believes will best serve applications.

These methods include:

• Least recently used (LRU): Evicts the data that was least recently used, first.
• Least frequently used (LFU): Instead of removing the data that was least recently used (as with LRU), it evicts data that was least frequently used. Meaning that the data that is set to be removed might have been used recently, but wasn’t used as often as the rest of the data.
• Most recently used (MRU): In contrast to LRU, MRU removes data that was mostly recently used.
• First in first out (FIFO): This eviction policy removes data that was added first, or least recently. It is unrelated to usage and is based solely on when the data was entered into the cache.

Each approach has its advantages and disadvantages. Ultimately, the goal of all of these policies is to only keep the data that is most often accessed in the fastest tier. But these policies are not always sufficient. To this end, some more advanced in-memory cache providers offer an additional option:

• Business-Policy Driven: This allows data architects to set an app’s eviction policy to a more nuanced set of rules- and logic-based approach as opposed to a straight forward LRU or FIFO one. This grants full control over which data is saved in cache and when it should be evicted to SSD or disk according to the logic of the business needs, and allows for SLA-driven cost and performance optimization.

5. Caching Solution Types

Caching solutions can take on many different formats. While data architects might think first of the key-value cache, this is not the only option; some cache systems provide developers with additional capabilities.

Key-Value Caching

Some caches are based on a key-value store. This essentially means the data store has a unique key and it will provide a corresponding value back. Because this is all done in-memory, they are commonly used by developers as caching layers. While some key-value stores are limited in terms of the data types they can store, there are some versions that allow for multiple data types, such as lists, arrays, and sets.

Generally speaking, key-value caching layers tend to be the first technology developers turn to, but these are not always enough. For simple cases, the key-value solution is often fast and efficient. However as data grows and user queries increase, architects are faced with new challenges, such as the inability to index on more than one criteria (adding secondary indexes).

In such cases, key-value stores generally offer the option to duplicate the entire data store so it can be indexed according to the new key. But this process is hardware-consuming (more expensive) and inflexible. Further, more complex relations (e.g., hierarchical and many-to-many) cannot be implemented resulting in the solution being deadlocked.

Fully Indexed In-Memory Caching

Data architects are not limited to caching in key-value caching systems. They can also take advantage of in-memory computing solutions that offer SQL support. This includes fully advanced indexing capabilities, such as having multiple secondary indexes, as well as index by exact match, ranges, compound, texts and geo-spatial data fields.

Additionally, some caching solutions support full SQL APIs, which enable a more flexible slice and dice of the data and easy migration from the primary data store.

They also provide both advanced indexing and SQL support, allowing for easy on-demand analytics and BI reporting. Without SQL support or advanced indexing, aggregations must be predefined for satisfactory performance, which involves a great deal of coding.

Comparing Advanced Caching to Simple Caching Solutions

6. Why Consider Database Caching

No matter the design pattern or database, implementing a caching tier may take time and add some complexity to the overall design.

If this is the case, why implement it at all?

Data architecture requires data caching because having data stored locally in memory can help reduce issues such as long latency times between requests and high concurrency of users. In-memory caching can also help reduce run times of requests and batch jobs. Jobs that previously took hours can be reduced to minutes, and jobs that took minutes cut to seconds. In this modern world, where 53% of users are likely to abandon a website that takes more than 3 seconds to load, speed is critical.

Further, some caching systems that are provided as services can not only reduce latency but also scale elastically. As usage volume increases or an unexpected usage peak occurs, performance is not impacted. Dynamic scaling also prevents overprovisioning of private or public clouds.

7. Utilizing Distributed Caching

Distributed caching is an effective solution to meet demand for faster data access and processing in large-scale, high-load applications, offering:

• Improved Performance: By storing frequently used data in memory and close to the application layer, distributed caching reduces the need for expensive database calls, which speeds up application response times
• Scaling: In a distributed cache, data is partitioned across several nodes, enabling the addition of more nodes to the cache cluster which increases the cache’s capacity 
• High Availability and Fault Tolerance: Distributed caches usually replicate data across multiple nodes, therefore if one node fails, requests can be served by other nodes in the system, assuring high availability of data and removing the ‘single point of failure’
• Load Distribution: Since read operations can be offloaded to the cache, resources are freed up on the main data store 

8. Summary of Caching Benefits

The right caching platform can help organizations:

• Accelerate digital applications
• Support multiple concurrent clients
• Handle peak loads
• Shorten the batch reporting window
• Accelerate BI and generate BI reports on fresh data

9. Caching Case Studies

Now that we know what caching is, let’s explore some real-life examples of how in-memory caching—and in this case smart in-memory caches—can improve business processes.

Real-Time Trading Risk Reporting System

Many banking and financial institutions still rely on overnight calculations to provide them with risk models, credit valuations, and potential market exposure. They often have hundreds of applications that must constantly be synced with each other to support their analysts, traders, and general employees.

But slow databases and data warehouses can bog down the data access, applications, and models running on these systems.

This is especially relevant for this use case, where a bank was struggling to manage all of its data needs. The bank needed to handle all its modern models, which were connected to hundreds of data sources—that were often impacted by changes happening every minute—in order to accurately manage risk. Updating systems nightly because their data warehouses couldn’t handle the influx of data therefore posed a serious risk.

To mitigate this risk, the bank opted for an in-memory cache that kept the last 24 hours of multiple sources of data loaded to the cache and that offered advanced indexing and full SQL support. This enabled the clients to perform real-time risk analysis calculation on live data.

This model offers significant benefits for financial institutions in particular, and has already been adopted by most of the major banks.

Batch Acceleration for a Leading Airline

Company mergers and partnerships frequently require integrating systems. Often, this process is managed by batch jobs that help manage data flows. However, with large companies in particular, the merging and integration of data can be slow, harming productivity and customer experience.

Two airlines recently merged and needed to integrate their crew management systems, which were responsible for handling crew members’ shifts and which flights they would be on. Adding to the complexity, airline union policies allowed crew members to swap flights.

Due to the many possible risks and stress that could be caused by crew members not knowing whether their requests to switch flights would be approved, the airline was contractually mandated to finish this batch process within four hours.

This was a problem: The batch jobs took nine hours on standard SQL databases and at best half that on NoSQL databases, thus failing to meet the four-hour SLA.

To address this, the companies moved to an in-memory caching system that offered a distributed in-memory grid platform. This platform turned their nine-hour batch job into a three-hour batch job, thus meeting their SLA and improving overall user experience.

The PSA Group CO2 Emissions Calculator: Legacy Modernization 

Following the 2015 “dieselgate scandal,” the EU was forced to replace its outdated CO2 emissions test. With the new regulations in place, Groupe PSA, a member of the Stellantis European automobile manufacturing corporation, was forced to provide a CO2 emissions report for each car.

To help manage this, the manufacturer created an online CO2 emissions calculator for their vehicles. The calculator ran on a production mainframe that could handle up to 200 requests per second. The demand, however, was expected to reach 3,000 queries per second.

They initially attempted to address this through the use of a key-value data store, but with all the different possible configurations permitted by the manufacturer, this resulted in billions of custom configurations. This proved difficult for the key-value store, because the key-value store was optimized for a single criteria query, but the custom configurations required the ability for multi-criteria queries. This made the key-value store inefficient and unusable, and their efforts failed.

The car manufacturer therefore switched to an in-memory grid solution that allowed them to limit requests from their mainframe by having an advanced indexing layer of all possible configurations. This enabled more efficient management of all possible data that could be cached in memory. Moreover, by collecting data and business logic in the same memory space, they were able to dramatically decrease data network movement and overhead while enhancing performance.

10. Looking Ahead 

As data management evolves to adapt to new technologies, two topics are at the forefront,  AI Driven Caching and Cloud Native Caching. 

AI-Driven Caching 

AI caching is a dynamic approach that leverages ML algorithms to optimize data storage and retrieval processes. This innovative method is transforming how organizations handle vast datasets, accelerate application performance, and reduce operational costs. At the core of AI caching lies the ability to predict data access patterns. By analyzing historical usage data, ML models can identify data that is frequently accessed, infrequently used, or even obsolete, enabling caching systems to make informed decisions about which data to store in high-speed memory to optimize cache performance. 

Tip: When selecting what to cache for your ML models, you should cache results that are expensive to compute, frequently accessed or reused, and stable or deterministic.  

How AI Caching Works

AI Caching using machine learning algorithms to predict which data will be needed next and preemptively caching this data, significantly boosting efficiency and performance.

• Data Collection: AI-driven caching systems start by collecting data on access patterns, request frequencies, response times, and cache hit rates and use this data to train ML models.
• Pattern Recognition: ML algorithms analyze the collected data to identify patterns and correlations, such as detecting that certain data sets are frequently accessed together, or at specific times.
• Prediction and Pre-caching: Based on the recognized patterns, the AI system predicts future data access needs. It then preemptively caches the predicted data, ensuring that it is readily available when needed.
• Dynamic Adaptation: AI caching systems continuously learn and adapt to changing access patterns. This dynamic adjustment helps maintain optimal performance even as usage patterns evolve. 

Benefits of AI Caching

AI models can continuously learn and adjust caching strategies accordingly. This self-tuning capability ensures that the cache remains optimized for performance, even in dynamic environments.

• Improved Performance: By predicting and pre-caching data, AI caching reduces the time required to access frequently used data, leading to faster application response times.
• Resource Optimization: AI caching optimizes the use of storage resources by ensuring that only the most relevant data is cached, reducing unnecessary data storage.
• Cost Efficiency: Efficient caching reduces the load on primary storage systems, potentially lowering operational costs associated with data retrieval and storage

By optimizing data storage and retrieval, AI can accelerate application performance, reduce latency, and improve user experience. Moreover, it can help organizations reduce hardware and energy costs by minimizing data movement.  

Cloud-Native Caching

Cloud-native caching leverages the cloud’s inherent capabilities to optimize data storage and retrieval, ensuring high performance, scalability, and resilience. It involves deploying caching solutions as cloud services, often managed by the cloud provider itself. This approach offers several advantages over traditional caching methods:

• Scalability: Cloud-native caching solutions can easily scale up or down to accommodate fluctuating workloads. This is crucial for cloud-native applications that often experience unpredictable traffic patterns.
• High Availability: Distributed caching architectures, commonly used in cloud-native environments, provide redundancy and fault tolerance, ensuring continuous availability of cached data.
• Integration with Cloud Services: Cloud-native caching solutions seamlessly integrate with other cloud services, such as databases, message queues, and load balancers, creating a cohesive and efficient architecture.

Key components of cloud-native caching include:

• Distributed Cache: A distributed cache is spread across multiple nodes, enhancing performance and reliability, and can handle large datasets and high traffic volumes.
• In-Memory Cache: For extremely low latency requirements, in-memory caches store data directly in the server’s memory, providing rapid access 
• Cache Invalidation: Effective cache invalidation strategies are essential to maintain data consistency. Cloud-native caching solutions often provide advanced mechanisms for cache invalidation, such as tagging, events, and time-to-live (TTL) settings
• Serverless caching: Offers scalable and cost-effective caching solutions without the need to manage infrastructure

By adopting cloud-native caching, organizations can significantly improve the performance and scalability of their applications. It enables faster response times, reduced load on backend systems, and enhanced user experience.

11. Last Words

Advanced caching solutions have proven a useful tool for developing systems that need to meet tight SLAs. By strategically caching data in RAM, applications can deliver rapid responses, enhance scalability, reduce operational costs, and ensure data availability. Distributed, in-memory caching significantly improves the delivery of the responsive, high-performance applications that users demand.