Make XAP Work for You: How to Use Remote Read Eval Print Loop (REPL) to Control XAP

We’re excited to introduce a new blog series, covering all things XAP, including insider tips and tricks from our team of architects and developers working around the clock to make XAP work for you. The GigaSpaces blog has evolved substantially, and our goal is to ensure you are kept in the loop with product updates, in-depth tutorials, and how-tos in order to help you up your professional game and offer real value to your team. In 2017, the blog will grow to provide even more content, so be sure to check back regularly as we have heaps of new information on the way. This is the second post in the series. 

Tutorial: How to Use Remote Read Eval Print Loop (REPL) to Control XAP

With XAP 12.1, we’ve developed an admin REST API that makes it possible to perform administration action on the grid. In this tutorial, I will show how to install XAP 12.1 and run the new REST API. I will describe the REST functionality and will show how to use a Scala REPL which has embedded client to this rest server to automate tasks on the grid remotely.

Getting Started with XAP 12.1 REPL

First, you’ll need to download XAP 12.1 in order to have the latest version with the `REST Cloud-Native Orchestration.` Once you’ve downloaded XAP 12.1, go ahead and unpack the zip.
Next, you will have to get the license from the GigaSpaces site. Once you have it, add it to the file `xap-license.txt` in the dir that was created by the zip.
Edit the file `setenv-overrides.sh`  
Add the line `export XAP_MANAGER_SERVERS=your-host-name`
Replace `your-host-name` with your real hostname
Now you are ready to run the system in local mode. To do so, type ‘./gs-agent.sh –manager-local’ from a terminal that has opened in XAP the `bin` directory. If you browse to `http://localhost:8090/v1/index.html` you will see a nice swagger page describing the admin REST API. Give it a try.
For example, go to Hosts view, type “GET /hosts” and get all the hosts in the system. At this time, there should be only one host with no containers.

To create a new container, open the containers section and click on the first “POST /containers” (where the description is “creates a new container”). Fill in the host parameter as shown in the image and click the `Try It Out` button.

You can verify that a container was created by getting all hosts and seeing that your host has a container.

Using this REST API, it is possible to manage the grid from anywhere using the language of your choice.

It is possible to upload jars to the grid and then deploy processes using those jars. You can also choose to deploy an instance of spaces on a node in the grid.
One specific client we provide is the Admin REPL. It is a Scala-based REPL that has the REST embedded as primitives commands.
Our REPL has the following features:

• Supports autocomplete
• It is asynchronous by nature
• Provides Scala as a glue language for the REST primitives
• The command is implemented in a way which makes it easy to compose commands together
• It enable running batch scripts that are doing work against the system

Let’s start up the REPL and see how it works.

Unzip the file: demo-repl.zip and go into the directory where the file a`admin-repl.sh` is found and define the system environment:
restUrl export restUrl=<YOUR_REST_URL>.  In my case – export restUrl=http://your-host-name:8090/v1.  
Make sure to replace `your-host-name` with your real hostname.
The REPL should start with the URL and end with @ as a prompt
Type hosts as the first rest command:

http://barakbo-pcu:8090/v1@ hosts
res2: Future[List[Host]] = Promise@147296487(state=Transforming(List(),Promise@464349589(state=Transforming(List(<function1>),Promise@783042794(state=Transforming(List(<function1>),Promise@279707594(state=Transforming(List(<function1>),Future@2132041506(depth=2,parent=Promise@450962180(state=Transforming(List(<function1>, <function1>, <function1>),Promise@993116346(state=Transforming(List(<function1>),Future@1881654105(depth=1,parent=Promise@1545356473(state=Transforming(List(<function1>, <function1>),Promise@1845290909(state=Transforming(List(<function1>),Future@254677784(depth=1,parent=Promise@1587539421(state=Transforming(List(<function1>, <function1>),Future@1729841184(depth=1,parent=Promise@1688239445(state=Transforming(List(<function1>, <function1>),Future@984348334(depth=1,parent=Promise@1739251556(state=Transforming(List(<function1>, <function1>),Future@1951979433(depth=1,parent=Promise@1633309010(state=Transforming(List(<function1>, <function1>),Promise@1393373445(state=Transforming(List(<function1>),Promise@922136563(state=Transforming(List(<function1>),Promise@194100457(state=Interruptible(List(<function1>),<function1>))))))))))))))))))))))))))))))))))))

Ooh, what happened there? since every REST command is asynchronous in the REPL, it returns a Scala future. We can wait on the Future or add a Listener that will be called upon when the future will be resolved.
Let’s begin with the option of waiting on the Future:

http://barakbo-pcu:8090/v1@ hosts get
res3: List[Host] = List(Host("barakbo-pcu", "192.168.33.172", List()))

This is much better, now we can work with it.

res4 foreach println
Host(barakbo-pcu,192.168.33.172,List())

Let’s try to create two JVM (GSC) on our machine.

http://barakbo-pcu:8090/v1@ containers get
res6: List[DeploymentInstanceContainer] = List()
http://barakbo-pcu:8090/v1@ startContainer("barakbo-pcu") get
res7: DeploymentInstanceContainer = DeploymentInstanceContainer("barakbo-pcu~15240", 15240, List(), List())
http://barakbo-pcu:8090/v1@ containers get
res8: List[DeploymentInstanceContainer] = List(DeploymentInstanceContainer("barakbo-pcu~15240", 15240, List(), List()))
http://barakbo-pcu:8090/v1@ startContainer("barakbo-pcu") get
res9: DeploymentInstanceContainer = DeploymentInstanceContainer("barakbo-pcu~15479", 15479, List(), List())
http://barakbo-pcu:8090/v1@ containers get
res10: List[DeploymentInstanceContainer] = List(DeploymentInstanceContainer("barakbo-pcu~15479", 15479, List(), List()), DeploymentInstanceContainer("barakbo-pcu~15240", 15240, List(), List()))

We have two containers so we can deploy space on those containers.

http://barakbo-pcu:8090/v1@ deploySpace("aSpace", 2, false) get
res11: Space = Space("aSpace", "aSpace", DeploymentTopology(Some("partitioned"), None, Some(2), Some(0)), List("aSpace~2", "aSpace~1"))

Note that deploySpace returns a future that contains space which will be resolved when the space deployment either fails or succeeds. While on the REST API, you will only get the request submitted code, therefore you have to do the polling yourself to get results.

http://barakbo-pcu:8090/v1@ spaces get
res12: List[Space] = List(Space("aSpace", "aSpace", DeploymentTopology(Some("partitioned"), None, Some(2), Some(0)), List("aSpace~2", "aSpace~1")))
http://barakbo-pcu:8090/v1@ rmDeployment("aSpace")
res15: Future[Deployment] = Promise@541397596(state=Transforming(List(),Promise@1579097076(state=Interruptible(List(<function1>),<function1>))))

You can type `help` to see all the REPL REST commands.
A Scala program that uses REST commands can be run as a batch using the command admin-repl.sh program.sc 
Now let’s start writing something more interesting using the REPL script.
Put the following Scala code into the file deploy-example.sc 

@main
def main(args: String*) = {
  println(args)
}

You can run this code (and passing arguments to it) using the command `./admin-repl.sh deploy-example.sc  foo bar`
The output of the println should be ArrayBuffer(foo, bar)  
Let’s add some REPL commands to the script.

@main
def main(args: String*) = {
  println(space.get)
}

You should get the empty list when running this command.
Now let’s try to create three JVMs concurrently and wait for them to start

def killAllContainers = containers.get.map(c => killContainer(c.id).get)
@main
def main(args: String*) = {
  println("removing all containers")
  killAllContainers
  println("starting containers")
  val c = (Future.collect((0 to 3).map(_ => startContainer("barakbo-pcu"))).get)
  println("starting containers done " + c.map(_.id))
}

This script first removes all containers on the system and then starts three containers, waits for them to be ready and then prints their ids. The output should be:

barakbo-pcu:repl $ ./admin-repl.sh deploy-example.sc
restUrl=http://barakbo-pcu:8090/v1
(deploy-example.sc,0)
removing all containers
Apr 03, 2017 5:58:33 PM com.twitter.finagle.Init$$anonfun$4 apply$mcV$sp
INFO: Finagle version 6.42.0 (rev=f48520b6809792d8cb87c5d81a13075fd01c051d) built at 20170203-165908
starting containers
starting containers done ArraySeq(barakbo-pcu~22820, barakbo-pcu~22858, barakbo-pcu~22819, barakbo-pcu~22837)

Similarly, we can write a script that uploads the processing unit sources and at the same time start containers. When all the downloads and containers are ready, it will deploy all of the processing units concurrently and wait for them to finish.

We can combine REST commands in the following ways:

1. Sequentially one after another using flat and flat map for example. For instance, given two container ids which we’d wish to restart one after another. We can write the following code:

/* given the REPL primitive: restartContainer(id: String):  Future[com.gigaspaces.rest.finagle.model.DeploymentInstanceContainer]
*/
def restart2(id1: String, id2: String): Future[(String, String)] =
 restartContainer(id1).
    flatMap(deployment1 => restartContainer(id2).
                 map(deployment2 => (deployment1.id, deployment2.id)))

We can now call this function with ids of two containers to restart since it returns future with a tuple of the container ids we can wait on the result with get. Notice that we will get two new ids because the container id contains the process id and the process is new.

restart2("barakbo-pcu~15380", "barakbo-pcu~15464").get
// > res6: (String, String) = ("barakbo-pcu~15932", "barakbo-pcu~16024")

We can write restart2 using Scala for comprehension.

def restart2(id1: String, id2: String): Future[(String, String)] = for {
   deployment1 <- restartContainer(id1)
   deployment2 <- restartContainer(id2)
}yield(deployment1.id, deployment2.id)

2. Or we can send all the action concurrently (since the containers are not dependent on each other) and wait for the results:

def restart2(id1: String, id2: String): Future[(String, String)] = {
   val f1 = restartContainer(id1)
   val f2 = restartContainer(id2)
   f1.flatMap(deployment1 => f2.map(deployment2 => (deployment1.id, deployment2.id)))
}

This code should run at half the time. Give it a go.
Again we can use Scala for comprehension:

def restart2(id1: String, id2: String): Future[(String, String)] = {
    val f1 = restartContainer(id1)
    val f2 = restartContainer(id2)
    For{
       deployment1 <- f1
       deployment2 <- f2
    }yield(deployment1.id, deployment2.id)
}

You must be thinking, OK this is nice but I don’t have just two containers, I have many of them. Can I compose a list of containers either sequentially or in parallel? As it turns out, you can and it’s super easy. Here’s how:

import com.gigaspaces.rest.finagle.model
def restart(ids: Seq[String]): Future[Seq[String]] = {
   val res : Future[Seq[DeploymentInstanceContainer]]  =
             Future.traverseSequentially(ids)(id => restartContainer(id))
   res.map(seq => seq.map(_.id))
}
restart(List("barakbo-pcu~17042", "barakbo-pcu~17122", "barakbo-pcu~15932", "barakbo-pcu~15539")).get
// > res24: Seq[String] = Vector("barakbo-pcu~21669", "barakbo-pcu~21756", "barakbo-pcu~21845", "barakbo-pcu~21933")

And now do it concurrency, it will be faster.

import com.gigaspaces.rest.finagle.model
def restart(ids: Seq[String]): Future[Seq[String]] = {
    val res : Future[Seq[DeploymentInstanceContainer]] =
              Future.collect(ids.map(restartContainer _))
    res.map(seq => seq.map(_.id))
}

We can now restart all existing containers concurrently using:

containers.flatMap(deployments => restart(deployments.map(_.id))) get

containers return Future[List[DeploymentInstanceContainer]] we flat map on the future to the list of deployments and then extract the ids and send them to restart.

containers.flatMap(deployments => restart(deployments.map(_.id))) get
> res2: Seq[String] = ArraySeq("barakbo-pcu~27453", "barakbo-pcu~27418", "barakbo-pcu~27586", "barakbo-pcu~27409")

Final Thoughts

The shell we are using here is based on Ammonite, which means it is a real shell and not just a REPL. For example, C-C does not exist, it just stops the current execution. You will need to hit the C-D to exit the shell. One more feature of the Ammonite shell is that it can be used to interact with the system. For example, ls.! will return a Seq of the files in the current directory. This means that you can combine XAP admin commands with system commands to write even more meaningful scripts.