AWS re:Invent – A Quick Roundup For the AI and ML Community

With an estimated 50 thousand attendees gathered in Las Vegas last week, Amazon rolled out dozens of new features, upgrades, and new products at AWS re:Invent. Here’s a quick roundup of news out of the annual conference that may matter to members of the AI and ML community.
As every year on AWS:reInvent, AWS announced its new services and updates to existing services at re:Invent 2018, among others:

• AWS announced Outposts, a hybrid-cloud offering that uses the same hardware AWS uses in its cloud
• AWS launched a managed streaming Service for Kafka, a fully managed, highly available service available now on AWS as a public preview
• S3 Intelligent-Tiering, the first cloud object storage class that delivers automatic cost savings by moving data between two access tiers
• Machine Learning Marketplace for sellers, essentially a data catalog that enables finding, training and deploying ML solutions

AWS outposts

AWS announced Outposts targeted to help customers build and support their hybrid clouds, while seamlessly connecting to the rest of AWS’s broad array of services in the cloud.

AWS Outposts will be your on-prem extension of your Amazon VPC.  AWS Outposts allows you to use the same exact APIs and control plane you use in the AWS cloud, but on-premise.  At this stage the service allows provisioning of  EC2 instances (C5, M5, R5), and storage options – EBS volumes, local instance storage and local disk options.

In the upcoming months, additional services will be added, like Amazon Relational Database (RDS), Kubernetes Management Service (EKS), Amazon SageMaker and Amazon EMR, which enable running other popular distributed frameworks such as Apache Spark, Hadoop, Presto, and Flink.

From the GigaSpaces perspective…
GigaSpaces customers can now easily deploy XAP and InsightEdge analytics platform on their Kubernetes cluster in a hybrid on-prem and AWS cloud deployment and additionally deploy machine learning models trained on SageMaker in production.

Streaming

AWS just launched a managed Kafka service called MSK.  AWS is making it seamless for users to set up and manage Apache Kafka cluster and nodes, an open-source stream-processing software platform.

This new announcement is surprising due to the fact that AWS already offers a similar service, called Amazon Kinesis, used for ingesting streaming data, including loading and analyzing data with Kinesis Data Firehose and Kinesis Data Analytics.

Kafka is superior in terms of throughput and latency, as well as it supports an Exactly-once semantics, which guarantees each message being delivered exactly once to the end consumer.

From the GigaSpaces perspective…
The new MKS service allows GigasSpaces customers to stream data for real-time analytics and build their architecture for both hybrid and multi cloud deployment without being locked into a specific cloud provider.

Intelligent Tiering

AWS also announced S3 intelligent-tiering, a new storage class designed to optimize storage costs automatically when data access patterns change, moving data between two access tiers — frequent access and infrequent access.

From the GigaSpaces perspective…
GigaSpaces MemoryXtend module allows you to leverage business-driven policy to automatically auto-tier hot and warm data between RAM, Persistent Memory, Storage Class Memory and SSD, as well as automatically move data for cold storage and archiving in Amazon S3.  By utilizing both mechanisms for intelligent tiering you can efficiently store the right data at the right storage layer based on performance, while optimizing your infrastructure costs across the entire solution and data lifecycle.

A New Machine Learning Marketplace

AWS also announced a new marketplace for Machine Learning solutions, which provides distributed algorithms and data science distribution, like H20.ai and Anaconda, as well as algorithms for text semantic analysis, object detection and more. This catalog enables you to test the ML solutions with SageMaker and then operational your ML models in production over InsightEdge.

My Top  Big Data and Machine Learning Reads

Here are my top choices for further reading concerning AI, ML and other interesting infrastructure services:

• Managed Stream for Kafka:  Fully managed and highly available Apache Kafka service.
• Amazon DynamoDB Transactions: Transactions support in DynamoDB.
• AWS Marketplace for Machine Learning: 150 ML algorithms and models that can be used with SageMaker.
• AWS Outposts: Run AWS Native infrastructure on-premise in your datacenter.  AWS Outposts bring native AWS services, infrastructure, and operating models to virtually any data center, co-location space, or on-premises facility.
• AWS S3 Intelligent-Tiering: a new Amazon S3 storage class designed to optimize storage costs automatically when data access patterns change.
• Amazon Timestream: Fully managed time-series database. Timestream also automates rollups, retention, tiering, and compression of data.
• Amazon Textract: OCR service to easily extract text data from documents and forms. It will automate your document workflows, including capturing,  searching and labeling for processing of loan applications, medical claims and others.
• Amazon Personalize: Fully managed real-time personalization and recommendations service. You provide an activity stream from your application, like page views, downloads, clicks, purchases, etc. as well as as an inventory of the various items, such as: articles/videos/products. Amazon Personalize will customize a personalized recommendations engine to fit your data.
• Amazon Forecast: Accurate time-series forecasting service. Feed historical data and related fresh data and expose forecasts for time-series data that change over time; like price, discount or web traffic.

Inspired?  Learn how to deploy GigaSpaces with Amazon Elastic Kubernetes Service (EKS).