Introduction: What is open platform design in data architectures?
Open platform design is an approach to software development that emphasizes flexibility, interoperability, and ease of integration with other systems and applications. Gartner defines open architecture as “a technology infrastructure with specifications that are public, as opposed to proprietary. This includes officially approved standards as well as privately designed architectures, the specifications of which are made public by their designers.” In other words, an open architecture is based on open standards, which makes it much easier to exchange data and perform operations between different applications and services.
When zooming in on data management technologies and open standards, there are a few that stand out:
- SQL (Structured Query Language) is a standardized programming language that is used to manage relational databases and perform various operations on the data in them. It is by far the most common data language in use today.
- As noted on the Library of Congress website, JSON is a lightweight, text-based, language-independent data exchange format. JSON defines a small set of structuring rules for the portable representation of structured data. JSON is quickly gaining popularity over the legacy XML format.
- The OpenAPI Specification, based on the Swagger Specification, is a machine-readable interface definition language for describing, producing, consuming and visualizing web services in a vendor neutral description format. Open API standardizes a “contract” that allows applications to consume services.
- Apache Kafka is an open-source system developed by the Apache Software Foundation written in Java and Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds. Apache is a different animal from the standards mentioned above as it’s not a public standard. Nevertheless, it is the most popular open-source distributed streaming system currently in use.
Open Platform Design is driving Data Access Services
So how does Smart DIH, embrace open platform concepts? At its core, Smart DIH is guided by open platform design philosophy, and is built on open standards:
- Full support for SQL
- Pluggable Connector framework which plugs natively into Kafka and can consume any JSONdocument regardless of its content
- Restful microservices which publish OpenAPI contracts
Before we do a deep dive into how open platform design is implemented in Smart DIH, we’ll present a high-level overview of the Smart DIH platform.
Smart DIH is a data integration hub solution designed to deliver the ‘always fresh – always on’ data that modern applications rely on. It aggregates multiple back-end Systems of Record (SoR) into a low-latency, scalable, high performance data layer exposing APIs and events. By decoupling systems of record from digital applications, Smart DIH enables enterprises to drastically shorten the development and deployment cycle for new digital services, and rapidly scale to serve millions of concurrent users – no matter which IT infrastructure or cloud topologies they rely on – cloud, on-prem or hybrid.
Smart DIH consists of three functional layers: Digitization layer, hosting layer and a data integration layer.
This first blog in the Smart DIH Open Platform series focuses on how open platform architecture is reflected in the Digitization Layer.
How Open Platform Architecture is reflected in the Smart DIH Digitization Data Layer
The digitization layer connects to multiple and diverse digital channels. Data is served to digital applications over a unified data model that supports a mix of structured and semistructured data through RESTful APIs and events, using SpaceDeck’s low-code service creator. In addition, distributed SQL queries may be executed from third-party applications, using clientless JDBC/ODBC connectors.
The digitization layer is a critical component of the Smart DIH. It provides a standardized interface for accessing and exchanging data that is held in the platform. Essentially, the layer acts as a middleman between the underlying systems and applications and the data. This can be particularly important in complex technology ecosystems where multiple systems and applications need to work together seamlessly and are dependent on multiple data sources. Additionally, the digitization layer can help to promote scalability, maintainability, and flexibility by abstracting the underlying complexities of the systems and applications and providing a high-level, uniform interface that is easy to use and understand.
Lastly, the digitization layer provides a high degree of agility:With its microservices pattern and low code interface, organizations can rapidly prototype and deploy new data services, test and iterate them quickly, and scale them up or down as required. By providing a faster, more flexible approach to application development, low code enables organizations to respond quickly and nimbly to market and business demands.
Technical overview
The data hosted in the Smart DIH is exposed to applications through data access services – from a single endpoint low code data microservice, to a multiple endpoint JAR service. These data services support the OpenAPI specification, using the Swagger framework and Swagger tools.
The data access services are registered through the API manager, so they can be used by application developers. Alternatively, in certain topologies or in larger organizations, they need to be registered through an API Gateway to allow access to them. An API Gateway is a server component that acts as an intermediary between backend services and client applications. Its primary role is to provide a unified interface for clients to access multiple backend services through a single endpoint, while also offloading common tasks such as authentication, authorization, and rate limiting.
Most API Gateway providers support OpenAPI, which is now a widely adopted standard. Examples of API Gateway providers that support OpenAPI include Amazon API Gateway, Apigee, Kong, Azure API management, Google Cloud Endpoint and more.
How open platform characteristics are reflected in the Smart DIH Digitization Layer: API gateway agnostic
- Through its support of open standards and open source – specifically by supporting the OpenAPI specification – Smart DIH enables interoperability with all API gateways and prevents vendor lock-in.
- Smart DIH is an API-driven platform. In this regard it provides APIs that enable developers to quite easily integrate their applications and services with the platform. The APIs are based on widely adopted standards, such as REST and JSON, which ensure interoperability and ease of use.
- Extensibility – The Smart DIH digitization layer is designed to be extensible, with APIs that allow developers to add custom functionality
Benefits: fit into any IT environment
The architecture of the Smart DIH digitization layer offers several potential benefits.
By being agnostic to numerous API gateways, you can easily switch between different gateways without having to carry out any changes. This provides greater flexibility and allows you to choose the gateway that best meets your needs at any given time.
Being agnostic to a specific API gateway also makes your services more portable. This means that it can run on different platforms and infrastructures without having to be modified to work with a specific gateway.
In addition, by not being tied to a specific API gateway vendor, you are less likely to become locked into their ecosystem. This can help you avoid potential vendor lock-in issues and gives you greater control over your infrastructure.
Another benefit is that your service is more compatible with different systems and services. This makes it easier to integrate with other applications and services, which can be particularly useful in large-scale distributed systems.
Overall, being agnostic to API gateways can help promote partnerships by reducing friction, increasing interoperability, and providing more options for collaboration.
In addition, the Swagger spec provides automatically generated documentation of the API, making it usable and accessible to developers.
An example of how these benefits translate into real-life situations is evident in cloud topology. Organizations that operate in multi-cloud environments can benefit from being agnostic to API gateways, as it allows them to use a dedicated gateway in each of the clouds without being limited to a specific gateway. Smart DIH allows data professionals to develop a data access service once and deploy it across multiple API gateways. In this way, data access services exposed via Smart DIH can be published to multiple API gateways in a repeatable, consistent manner. This enables organizations to choose the best cloud service for a specific use case, and avoid vendor lock-in, while executing the same services in all environments.
Conclusion
The Smart DIH Digitization Layer supports the OpenAPI specification, which is a widely adopted standard for describing RESTful APIs. This support enables developers to easily create, deploy, and manage APIs that are compliant with the OpenAPI standard.
By supporting OpenAPI, Smart DIH facilitates interoperability and ease of use, as developers can use familiar tools and practices to work with the platform. It also promotes openness and collaboration, as developers can easily share and reuse API definitions and build on each other’s work.
In addition to OpenAPI, Smart DIH incorporates other open data hub architecture characteristics, such as modularity, extensibility, and community-driven development. These characteristics enable Smart DIH to provide a flexible, scalable, and customizable platform for data integration and digital transformation.
Overall, Smart DIH’s support for OpenAPI reflects its commitment to openness, interoperability, and ease of use. By embracing industry standards and best practices, Smart DIH enables citizen developers to focus on building innovative applications and services, without having to worry about the underlying complexities of data integration and management.
The next blog in our Smart DIH Open Platform series will discuss how Smart DIH embraces a ‘bring your own SQL tool’ philosophy, enabling data professionals to leverage the tools they are already familiar with when creating data pipelines that drive real-time data delivery to digital applications.
Learn more about Smart DIH Open API Support: Watch a demo now!
