Elastic Distributed Risk Analysis Engine

Financial services applications produce reports constantly. Some of them produce reports where the data required for the report is generated over night  via batch processing, and some other type of reports are produced instantly upon user request. This type of processing activity requires fast access to the raw data and the ability to utilize distributed resources available on the local environment or on the cloud.

The Elastic Risk Analysis Engine illustrates the following:
1. Calculating Net present value (NPV) for large amounts of trades in real time using an In-Memory Data Grid.
2. Intelligent Map-Reduce directing calculations into distributed nodes, lowering the network traffic and lowering the load on each calculation node.
3. Simulating lazy data load in a batch mode optimizing database access in case of a cache miss.
4. While the calculation is going on, dynamically scaling up and down the compute/data grid. This will increase the capacity of the compute/data grid and will allow it to utilize additional CPU resources to speed up the calculation time.

The Distributed Risk Analysis Engine performs Net Present Value calculations where the Trades used for the calculation divided into several Books. These books could represent different types of Trades, different markets, different customers , etc.

The Calculating Flow

The Calculating Flow includes the following:
– A client, splitting a list of Trade IDs into multiple batches. Each Batch is sent into the calculation node (space partition) via a The AnalysisTask that implements the Task interface. Each calculation node stores a subset of the Trade data.
– The AnalysisTask is executed. Once completed, an intermediate result is sent back to the client. If the requested Trade cannot be found within the space, it is loaded from the database.
–  The client aggregating the results retrieved from all the calculations nodes and reducing it to four numbers. These four numbers represent books.
 

For long calculations that consume relatively large amount of CPU time, the recommended approach to implement distributed calculations is the Master-Worker Pattern. The approach suggested with the Elastic Distributed Risk Analysis Engine should be used when the calculation time is very short where the data access should be very fast to minimize the data access overhead.

You can find a fully running Implementation here.

Shay Hassidim