We recently hosted a webinar together with Matt Aslett, Research Director for the Data Platform and Analytics Channel with 451 Research to discuss HOAP for the Insight-Driven Enterprise: The Emergence of Hybrid Operational and Analytic Data Processing. It was so successful that we decided to share some of the highlights with you in this blog series. This is the first post based on Matt’s insights.
What is HOAP? There are different phrases used by different organizations in the industry to talk about this concept. Most widely used is HTAP, which stands for Hybrid Transactional and Analytical Processing. Whereas 451 Research prefers to call it HOAP – Hybrid Operational and Analytic Data Processing – because its their belief that while all transactional workloads are operational, actually not all operational workloads are transactional.
Digital Transformation is Real and Happening Now
As organizations move to become more data-driven, we see that most enterprises out there are increasing their investment in data processing and analytics and machine learning software. The data and the rapid processing of data is a key driver in enabling companies to grasp the opportunities that are presented by digital transformation efforts in an attempt to increase competitive advantage.
Let’s unpack these buzzwords for a second. When we talk about digital transformation, it’s important to note that it’s actually something real, it’s happening. According to 451 Research surveys conducted with the 451 Alliance, 47% of organizations say they have a formal strategy and they’re actively in the process of going through a digital transformation project. Another 25% have started, another 21% are in the planning stage.
The key message here is if you are the 7% that has no ongoing digital transformation strategy, you are probably in some trouble.
You could actually argue anybody who’s in that 53%, who don’t actually have a formal strategy and is active in the process of transforming digitally, are potentially in some trouble in terms of getting disrupted by emerging startups or the existing rival providers who are already on the process of transforming themselves.
Leaders are Investing in AI, ML and Personalization
So what do we mean when we talk about digital transformation and its various aspects? 451 Research has identified that a lot of focus is being put into artificial intelligence, machine learning, personalization and more intelligent applications.
If we look at the ways in which leaders (the 47% with a strategy in place) and laggards (the 7% with no ongoing digital transformation strategy) are approaching this, the leaders are already adopting AI and machine learning for intelligent business applications. We also see this playing out in terms of the adoption of cloud infrastructure, in terms of intelligent personalization, and actually shifting applications into the cloud.
It’s easy to think of digital transformation as just the next buzzword, but the more you drill into it, there are very different levels of adoption between those that are leaders and laggards and there is this growing gap between the two.
The Shift from Systems of Records to Systems of Engagement
What we see from an application perspective can be summed up as a shift from Systems of Records to Systems of Engagement. While Systems of Record were previously all about internal, transactional data, mature applications and process efficiency, today we see a shift towards Systems of Engagement, which are more customer facing, intelligent, interactive, personalized applications which are driving disruption within the industry.
Systems of Engagement: Retail Example
To give an example of that, 451 Research has taken note that most companies are increasing their investment in data processing, analytics and machine learning. A key driver to that is to deliver more engaged applications.
Let’s take a look at Retail for example. Say you go into a store and you’re looking to make a purchase. Your engagement is with, or has traditionally been with a person. They can make recommendations, give you advice and point you in the right direction for whatever it is you’re looking for. The transaction, assuming you end up buying something, only actually enters the system of record when you make the purchase. From there, that data can be made available in the systems of analysis, traditionally a data warehouse, for analysis by data analysts, IT professionals and decision makers.
Various shifts are happening across this landscape. Analytics has been brought closer to the systems of analysis, resulting in an increasing amount of analytics and data science happening in-database. The biggest change, in some ways, is actually in terms of the Systems of Engagement.
Increasingly, the Systems of Engagement are being automated and the role of engaging with the customer is being delivered by software. It’s clearly happening online, but it’s also happening in a Brick and Mortar environment. You can go into a physical retail store and you’ll see shoppers engaging with the retailer via a mobile device. It could be their own device or one which is actually supplied in the store to enable them to check stock levels and get advice and recommendations. They might also receive personalized offers directly on their phones as they enter the store; in essence, their Systems of Engagement are completely changing.
Systems of Engagement have to be enabled by what 451 Research refers to as Systems of Intelligence. This includes the rules engines, the decisioning systems, the recommendation engines, natural language processing, image recognition and other forms of AI and machine learning and deep learning; all happening behind the scenes to empower things like chatbots and digital assistants and help with the delivery of those personalized offers.
Systems of Intelligence enable the enterprise to engage intelligently with the customer through operational applications and other Systems of Engagement. But in order to function properly, the Systems of Intelligence have to operate in real time. It’s simply not enough to be generating intelligence via traditional business intelligence applications and reports. There’s still a role for those with human-based data-driven decision making, but Systems of Intelligence actually have to automate the data-driven decisions and deliver that via those operational Systems of Engagement applications at the speed of business.
Data Processing Implications
Data processing implications are multiple. On the one hand, it has to be real-time. Customers expect a response in real time as they are interacting with that chatbot or personal assistant. It’s not just about delivering static content to an individual customer, it needs to be responsive, it needs to be contextual depending on the device you’re using, the kind of offers, previous purchasing, their physical location and other similar inputs. Moreover, as the terms of the engagement channel and the specific requirements change, the delivery of data needs to be responsive to that as well.
Data Processing Requirements
- In-Memory: In-memory is unarguably the only way to actually deliver that required level of responsiveness and performance that’s desired. It has also become and continues to become, increasingly affordable.
- Stream Processing: If we look at stream processing for real-time data processing, the focus is on the continuous processing of events and an event-driven architecture which organizations are looking to take advantage of in order to deliver the kind of applications that we’ve been talking about.
- Hybrid: When we think about hybrid we think on-premise and the cloud, relational and nonrelational data, but it also includes, this hybrid combination of operational and analytic processing.
Hybrid Operational and Analytic Processing
In the traditional data processing landscape, historically you have OLTP and OLAP database systems and ETL processing to move data in between the two.
With HOAP, one can provide a way for organizations to deliver analysis and intelligence from their operational data without the need to necessarily move it via a traditional or ETL processes.
The Combined Database Market
According to 451 Research, in the combined database and analytic and operational data processing $50 billion market, as it exists today, HOAP is actually playing a very small part the overall database market (1.5%).
This makes sense when you realize that existing applications and existing data processing deployments weren’t designed to deliver hybrid operational and analytic processing. They were designed to deliver OLAP and OLTP applications. So this figure is a bit misleading because it includes maintenance revenue for all those existing deployments which were never able to deliver HOAP workloads.
If we actually exclude the incumbent revenue and focus on new revenue, we’re looking at about $4.6 billion for example, in 2016. In this case, HOAP already has 10% of new revenue emerging in that space. From this point of view, it is a large and significant part of the market and even more so if you look at how we expect this to evolve.
By 2021, HOAP will become a significant part of the market for new database projects with over 30% of the total market by that point.
Another important point is that HOAP is part of the operational database market. Although we’re talking about analytic workloads and interactive analytics, it’s being performed on operational data. HOAP is not a replacement for traditional data warehousing, for standalone analytic processing. Just like the retail example, however, if you’re developing machine learning models and recommendation engines, that needs to be done on historical data in a specialist analytic database or Hadoop, but you then apply the model on the operational data as it’s being generated.
The data warehouse originated, arguably, as a workaround for the performance limitations of operational data processing. HOAP does not eradicate the need/desire for separate data analytics. However it removes the cost and complexity of ETL for real-time data processing and more frequent query requirements.
HOAP Data Processing is Just the Foundation
The potential of hybrid operational and analytic processing is great. Its foundations are ready, but on top of it you need the applications which take advantage of that data processing capability, such as recommendations, personalized content, personalized offers, as well as real-time fraud analysis and dynamic pricing. It all boils down to real-time business processing and real-time decisions based on analyzing operational data.
Primary HOAP Use Cases
Systems of record. Traditional enterprise operational applications: ERP, CRM, SCM, HRM. For processing operational data
Systems of engagement. Emerging enterprise operational applications: Recommendations,
Personalized content, Personalized offers, Real-time fraud analysis, Dynamic pricing, Real-time business process optimization. For analyzing operational data
Systems of analysis. Traditional enterprise analytic applications: Data warehousing, Data marts, BI and reporting, Data science. For analyzing historical data.
Hybrid operational and analytic processing database workloads are set to grow significantly for new database deployments, driven by the delivery of automated systems of engagement and the underlying systems of intelligence, in part to support improved customer engagement.
Data processing requirements include:
- In-memory – arguably the only to deliver the required level of responsiveness
- Continuous processing of event streams
- Hybrid: On-premises/cloud, relational/non-relational, operational and analytic
The development of new enterprise operational applications reliant on recommendations, personalized content, personalized offers and real-time fraud analysis, for example, will drive the adoption and use of in-memory, hybrid operational and analytic processing between now and 2021.