Developing front-end services is lengthy and complex: frequently app developers lack familiarity and intimacy with the data they need and must rely on backend teams for data APIs. This slows down app development, and time to value. At the same time, the data teams responsible for delivering data to these teams are often tied down by traditional data architectures: Backend systems are vulnerable to failure resulting from excessive data calls and heavy workloads; Data is locked down and cannot be easily delivered to front-end apps; Lack of scalability of backend systems may inhibit concurrency and throughput at peak times – just when you need it most. 

Digital Integration Hubs, like GigaSpaces’s SmartDIH, are designed specifically to help overcome these issues. SmartDIH offers low-latency, scalable 24/7 data access to front-end developers, while reducing workloads on backend systems. It achieves this by consolidating multiple disparate data sources into a data layer that synchronizes with the applications via event-driven patterns. 

As many experienced data engineers know, building batch and real-time data pipelines from multiple sources and ensuring data availability and consistency in real-time is no mean feat. The latest features in SmartDIH make this job much easier by offering data engineers a platform for building real-time data pipelines, carrying out data transformations and getting the most of event-driven architecture and ETL to deliver diverse data to front-end apps.

New Data Integration Capabilities

Let’s take a look at a few of the data integration capabilities just released in the latest version of SmartDIH.

• Data transformations and enrichment during data load: This feature offers the ability to carry out data transformations on the fly which allows data engineers to speed up data preparation and build data pipelines faster in the SmartDIH platform using no code / low code. With SmartDIH’s enhanced enrichment and transformation capabilities, data can be cleansed and transformed facilitating data preparation, quality improvement and ensuring accuracy.  
• Broader pipeline creation options: SmartDIH gives data engineers broader options and greater flexibility for creating near real-time data pipelines. Existing event-driven CDC pipelines are critical for real-time apps and services that require continuous, always fresh data. There are situations where CDC is unavailable or may be less appropriate – for example, when you want to take full advantage of the performance and scalability of SmartDIH but data doesn’t necessarily need to be updated in real-time. For these situations, SmartDIH offers the ability to enable JDBC-based data batch loads. Full batch load can now be performed as a “pull” functionality, giving data engineers optimal control over the frequency of data updates. 
• Multiversion Concurrency Control (MVCC) – a method that delivers a concurrent and persistent view of distributed transactions across SmartDIH partitions. SmartDIH keeps multiple versions of modified entries to ensure a persistent snapshot of the data.

The MVCC mechanism provides an elegant and efficient solution, allowing massive updates via the data integration module while maintaining consistency with the systems of record (SoR). In this manner, the consistency and integrity of the data before and after each bulk update is ensured, while maintaining full system availability. 

SmartDIH gives data engineers the ability to manage their data pipelines in a unified real-time data platform. By isolating underlying data sources from front-end applications, SmartDIH enables the development of data APIs directly over a unified data layer, eliminating repetitive one-off integrations, and protecting backend systems from overload and excessive calls – while ensuring data freshness, readiness and availability at all times. 

The result: streamlined and efficient data pipeline creation, simplified data-application integrations, and real-time data availability – with minimal risk to backend systems.