Skip to content
GigaSpaces Logo GigaSpaces Logo
  • Products
    • InsightEdge Portfolio
      • Smart Cache
      • Smart ODS
      • Smart Augmented Transactions
    • GigaSpaces Cloud
  • Roles
    • Architects
    • CXOs
    • Product Teams
  • Solutions
    • Industry Solutions
      • Financial Services
      • Insurance
      • Retail and eCommerce
      • Telecommunications
      • Transportations
    • Technical Solutions
      • Operational BI
      • Mainframe & AS/400 Modernization
      • In Memory Data Grid
      • Transactional and Analytical Processing (HTAP)
      • Hybrid Cloud Data Fabric
      • Multi-Tiered Storage
      • Kubernetes Deployment
      • Streaming Analytics for Stateful Apps
  • Customers
  • Company
    • About GigaSpaces
    • Customers
    • Partners
    • Support & Services
      • University
      • Services
      • Support
    • News
    • Contact Us
    • Careers
  • Resources
    • Webinars
    • Blog
    • Demos
    • Solution Briefs & Whitepapers
    • Case Studies
    • Benchmarks
    • ROI Calculators
    • Analyst Reports
    • eBooks
    • Technical Documentation
  • Contact Us
  • Try Free

A GigaSpaces solution to the Poison Message problem

Subscribe to our blog!

Subscribe for Updates
Close
Back

A GigaSpaces solution to the Poison Message problem

Jeroen Remmerswaal August 17, 2009
4 minutes read

GigaSpaces uses the notion of Event-containers for matching and handling data in the Space as events. This can be done in either Polling mode (the container polls the Space for data matching the criteria), or Notify (the Space pushes matching data as events to the container).

In case of a polling-container, the object should be changed and returned to the Space, to ensure it no longer matches the criteria. Otherwise the event would be re-triggered.

Now, what would happen in case the event-handler encounters a (permanent or temporary) failure during processing, such as a down-stream system is not available?
Well, clearly, the object is not handled, transactions will be rolled-back, etc, but most importantly: the object is returned to the Space in its original state. This means the object will still match the polling-criteria. This can cause endless loops in your polling container, unnecessarily consuming resources, and the problem will only disappear when the underlying problem is resolved.

This problem is known as the Poison Message problem (see for example this link).

A solution is to add try-catch blocks in your event-handler, and handle the problematic events appropriately to avoid loops. Simply catching and handling the exception has an issue though: It does not take in account that the problem may only be temporary. Therefore adding a retry-mechanism may make sense in your application.

However, adding a retryCount and maxRetries to the problematic object will not work: We cannot modify the erroneous object and write it back to the space, as we must revert the transaction under which it failed – there might have been very viable reasons for the exception in the first place!

For this reason we should use a separate object, which I call a Hospital object. This Hospital object holds a uid-reference to the erroneous object, and can include fields like retryCount and maxRetries.
Here’s a snippet of what this Hospital object might possibly look like:

public class Hospital implements Serializable {
    private String id;
    private String poisonObjectId;
    private Integer retryCount;
    private Integer maxRetries;
    private Integer routingId;

public Hospital() {
}

// Getters and setters beyond this line
}

 

Although a try-catch block in your event-containers will work, I have used an Aspect that intercepts any exception from a (polling) event-container:


@Pointcut("@annotation(org.openspaces.events.adapter.SpaceDataEvent)")
private void spaceDataEvent() { }

@Around(“spaceDataEvent()”)
public Object exceptionHandling(ProceedingJoinPoint pjp)
throws EventContainerRetryException {
Object o = null ;
try {
o = pjp.proceed();
} catch (Throwable e) {
eventContainerExceptionHandler.handleException(
(BaseObject)pjp.getArgs()[0],
e
);
}

return o ;
}

In the aspect we use an exception handler, that is invoked whenever any exception occurs. This handler takes care of finding and writing the hospital object, incrementing the retry-count, etc:


public void handleException(BaseObject baseObject, Throwable e)
   throws EventContainerRetryException
{
 Hospital template = new Hospital();
 template.setPoisonObjectId(baseObject.getId());
 Hospital hospital = hospitalGigaSpace.read(template);

if ( hospital == null ) {
hospital = new Hospital();
hospital.setPoisonObjectId(baseObject.getId());
// Set other fields
}

if ( hospital.getMaxRetries().equals(hospital.getRetryCount()) ) {
// Maybe you want to do something generic here to avoid the event
// from being triggered again, for example setting a status flag.
} else {
// Increase the retry count on the hospital object
hospital.setRetryCount(hospital.getRetryCount() + 1) ;
// Write the hospital object back, if you want with a lease-time
hospitalGigaSpace.write(hospital, 120000);
// Rethrow the initial exception caused in the event container
// This will cause its transaction to be rolled back

. throw new EventContainerRetryException(e) ; } }

What is important to realize is that the operations on the hospitalGigaSpace bean should work under a different transactional context from the event container: the polling container transaction must be rolled back while the operations on the hospital object must succeed.

BTW, in a next release of GigaSpaces a generic exception handler will be implemented, which can be used instead of try-catch blocks or AOP.

CATEGORIES

  • GigaSpaces
  • syndicated
Jeroen Remmerswaal

All Posts (4)

YOU MAY ALSO LIKE

January 5, 2012

Terabyte Elastic Cache clusters on…
8 minutes read

July 30, 2014

Using SSD As A Foundation…
8 minutes read

July 29, 2008

GigaSpaces and the Economics of…
14 minutes read
  • Copied to clipboard

PRODUCTS, SOLUTIONS & ROLES

  • Products
  • InsightEdge Portfolio
    • Smart Cache
    • Smart ODS
    • Smart Augmented Transactions
    • Compare InsightEdge Products
  • GigaSpaces Cloud
  • Roles
  • Architects
  • CXOs
  • Product Teams
  • Solutions
  • Industry
    • Financial Services
    • Insurance
    • Retail and eCommerce
    • Telecommunications
    • Transportation
  • Technical
    • Operational BI
    • Mainframe & AS/400 Modernization
    • In Memory Data Grid
    • HTAP
    • Hybrid Cloud Data Fabric
    • Multi-Tiered Storage
    • Kubernetes Deployment
    • Streaming Analytics for Stateful Apps

RESOURCES

  • Resource Hub
  • Webinars
  • Blogs
  • Demos
  • Solution Briefs & Whitepapers
  • Case Studies
  • Benchmarks
  • ROI Calculators
  • Analyst Reports
  • eBooks
  • Technical Documentation
  • Featured Case Studies
  • Mainframe Offload with Groupe PSA
  • Digital Transformation with Avanza Bank
  • High Peak Handling with PriceRunner
  • Optimizing Business Communications with Avaya

COMPANY

  • About
  • Customers
  • Management
  • Board Members
  • Investors
  • News
  • Events
  • Careers
  • Contact Us
  • Book A Demo
  • Try GigaSpaces For Free
  • Partners
  • OEM Partners
  • System Integrators
  • Value Added Resellers
  • Technology Partners
  • Support & Services
  • University
  • Services
  • Support
Copyright © GigaSpaces 2021 All rights reserved | Privacy Policy
LinkedInTwitterFacebookYouTube

Contact Us