To deliver innovative digital services that provide responsive, enjoyable customer experiences, organizations need an agile, modern data environment. Since the data that these services require often resides in legacy infrastructure, organizations search for solutions that can adapt their existing IT architecture to a service-oriented environment that can manage huge volumes of concurrent API requests. This post discusses how Digital Integration Hub architecture (DIH) helps organizations modernize their data stacks, and how this Hub stacks up to alternative technological approaches. Some of the forces driving this transformation include:

Cloud Adoption

Cloud computing offers scalability, cost-effectiveness, and on-demand resources, driving wide-scale adoption of cloud platforms.

Event-Driven Architectures

Businesses are increasingly reliant on real-time data to make informed decisions, and need instant access to data to identify trends, respond to customer behavior, and optimize operations. Traditional data warehousing, with its periodic updates and batch processes is no longer sufficient. Businesses are required to react to real-time events, triggering immediate data flows and fostering a more responsive environment, and have adopted event driven architectures to drive real-time decisioning.  

Real-Time Streaming

Real-time streaming technologies enable continuous data ingestion and processing. This allows organizations to analyze data as it’s generated, unlocking valuable insights and enabling immediate actions based on the latest information.

Digital integration hub architecture: powering the data-first approach 

Digital integration hub architecture emerges as a powerful solution to address these

challenges and empower organizations to thrive in the modern data-driven world. 

GigaSpaces Smart DIH platform, a digital integration hub that is pre-integrated and available out-of-the-box, consists of four conceptual layers:

• Data integration layer: Offers event-based data ingestion from source data stores; creates and manages data pipelines, applies data cleansing and validation policies 
• Hosting layer: Provides partition management, high availability, and scales to meet changing demand volumes. Storage tiering management automatically places data in the appropriate storage tier (e.g., RAM vs. SSD) and offers data indexing
• Digitization layer: Offers access data via APIs, microservice lifecycle management, and data enrichment
• Management & Control: Provides deployment, orchestration, monitoring, configuration, and control 

GigaSpaces Smart DIH offers a comprehensive solution that:

• Decouples the core systems from the digital services into an ultra-low-latency, high-performance data fabric 
• Replicates appropriate System of Record (SoR) data, reducing the latency of the API services layer
• Enables an end-to-end microservices approach, reducing the risk of introducing new services
• Combines data integration and data management technologies, data and analytics governance 
• Enables the development of digital services in parallel to the transition of some of the data sources to modern and advanced systems, in the cloud or on-premises

Benefits of digital integration hub architecture

A digital integration hub such as Smart DIH offers the following advantages: 

• Dramatically increasing throughput and scalability, improving API performance and enabling high availability
• Improved TCO: reduce the high workloads generated in SORs by digital application requests
• Facilitate an agile hybrid IT environment: by enabling data to be stored locally, it can lower the number of API calls to cloud and SaaS providers
• Creates a framework for delivering a data-rich user experience by ensuring fast real-time responses based on multiple data sources, which leads to improved customer satisfaction
• Simplifies data and application integration and enables agility by creating an up-to-date picture of fast-changing data needed to support a comprehensive set of APIs and events. 
• Greater flexibility in developing new offerings, by enabling teams to access data without involving SoR teams

DIH: The Competitive Landscape

Some vendors that have implemented certain aspects of DIH architecture include Hazelcast, MiaPlatform, IBM, Oracle and SAP. However, to our understanding, compared to the integrated off the shelf Smart DIH approach offered by GigaSpaces, what these solutions offer is more of a ‘do it yourself’ approach. Many businesses build an Operational Data Store (ODS) as a do-it-yourself, which evolves into a maintain-it-yourself effort. Other organizations choose a solution based on a mix of several key components, many or all of which use source components, such as CDCs or an ETL layer, databases, caching and service-development products, and more. These organizations often find themselves submerged in large and complex projects that incorporate convoluted architecture with integration, caching and other middleware components that integrate relational and non-relational data sources.

In these environments, data is accessed directly from the business applications within the ODS or open source components, resulting in high latency, governance challenges and cluster sprawl. Although the aim of a digital transformation journey is to improve the real-time user experience of the digital applications and the time to market of new digital services, embarking on long complex projects may be counter-intuitive to these goals. Additional challenges of the Do it Yourself (DIY) approach include operational issues, such as ensuring agility, and obtaining the required resources to design, implement and deploy these systems. As various diverse components are added, the cost and complexity of these operations tend to grow. 

A number of differentiators highlight the Smart DIH solution from DIY approaches and other solutions on the market, as summarized in this table:

DIY solutions allow a wider selection of components depending on the requirements. In this solution, there is usually no vendor lock-in, and the customer is responsible for all components integrated into the solution and the application.

However, here too, since a DIY solution is based on open source software uses several products, most of which are open source and some of which are commercial, each of them may have its own vendor lock-in, and there will not always be a suitable alternative to the chosen architecture relative to the other components. Also, after completion of development and integration between the various components, removal of a key component may be complex and problematic.

Last Words

Digital Integration Hub architecture, as exemplified by Smart DIH, is agnostic to the installation environment, and can be installed entirely on-premises, or entirely in the cloud, and in hybrid and multi-cloud environments. The installation is based on Kubernetes, and is not tied to any specific vendor so that users can easily switch to a different cloud vendor. Smart DIH powers optimized performance for real-time APIs, providing high concurrency support, in addition to high availability, and ensures that the integrity of the data is maintained at all layers of the database.