Command line services

We investigate whether it’s possible to build micro services from the command line

In my last post Transformative Moments we investigated the Dataweave CLI which we used to map and transform data from files. We discussed the notion of information pipelines and retrieving data from internet servers, but we didn’t show any examples of how you might do that. With some creative scripting, you should be able apply what you learn to constructing viable script based micro services.

For our services we’ll be using the API resources from {JSON} Placeholder. If you have an interest in creating your own Mock services without having to code, check out my article Zero Code REST with JSON Server.

For our first example we’ll map and transform ToDo data, which will filter to select a subset of ToDo’s. If you haven’t installed DW CLI already, be sure to read how in Transformative Moments. I’m going to use the Bash shell from Windows 10 and make some adjustments to the environment setting used in the earlier post. If you’re using Linux you can use the earlier environment settings.

Initial configuration

# Configure Bash env setting for Windows 10 file locations
# add DW CLI to PATH
export PATH=/c/Tools/dw/bin

# Adjust this location to where you want your scripts to go
export DW_HOME=/c/Home/dev/Mule/dw

# download my github scripts if you haven't already
dw --add-wizard MitchDresdner

# if you need to refresh the grimoires to download new or updated code
dw --update-grimoires

# static script locations
export JSON_DATA=$DW_HOME/grimoires/MitchDresdner-data-weave-grimoire/JSON

We’ll review both the Dataweave in-line method and incanting a grimoire for completeness.

Map and Filter ToDo – inline

# Fetch data from ToDo API
$ curl -s http://jsonplaceholder.typicode.com/todos | dw 'output application/json --- payload filter ($.id > 40 and $.id < 43)'

[
  {
    "userId": 3,
    "id": 41,
    "title": "aliquid amet impedit consequatur aspernatur placeat eaque fugiat suscipit",
    "completed": false
  },
  {
    "userId": 3,
    "id": 42,
    "title": "rerum perferendis error quia ut eveniet",
    "completed": false
  }
]

# same idea but create some new mappings
$ curl -s http://jsonplaceholder.typicode.com/todos | dw 'output application/json --- payload filter ($.id > 40 and $.id < 43) map (todo) -> { id: todo.id, task: todo.title, done: todo.completed }'
[
  {
    "id": 41,
    "task": "aliquid amet impedit consequatur aspernatur placeat eaque fugiat suscipit",
    "done": false
  },
  {
    "id": 42,
    "task": "rerum perferendis error quia ut eveniet",
    "done": false
  }
]

Map and Filter ToDo – grimoire

# Uses a spell and static data
dw -i payload $JSON_DATA/ToDo-4.json --spell MitchDresdner/FilterToDo
[
  {
    "myId": 199,
    "task": "numquam repellendus a magnam",
    "done": true
  },
  {
    "myId": 200,
    "task": "ipsam aperiam voluptates qui",
    "done": false
  }
]

# same idea but pulls data from an API
$ curl -s http://jsonplaceholder.typicode.com/todos | dw --spell MitchDresdner/FilterToDo
[
  {
    "myId": 199,
    "task": "numquam repellendus a magnam",
    "done": true
  },
  {
    "myId": 200,
    "task": "ipsam aperiam voluptates qui",
    "done": false
  }
]

Both are pretty good approaches, the inline approach is nice, but may quickly get out of hand as your script becomes more complex. I prefer the spell format for readability. You can use the Dataweave Playground to get it working and add the final script to your grimoire folder.

Shout out to Mariano De Achaval – For creating DW Playground as a GraalVM container, written in Dataweave. This is totally awesome, the Playground can now run from an executable, without a Docker dependency.

# Run the Playground
#   http://localhost:8082
$ dw --eval --spell Playground

We can extend our data pipeline now to do the following:

• Fetch a ToDo using a GET request
• Map the results to satisfy the posts API
• POST the results using CURL

$ curl -s http://jsonplaceholder.typicode.com/todos | dw 'output application/json --- payload filter ($.id == 42) map (todo) -> { userId: todo.userId, title: todo.title, body: "Task completed = " ++ todo.completed }' | dw 'output application/json --- payload[0]' |  curl -H "Content-Type: application/json" -s -X POST http://jsonplaceholder.typicode.com/posts
{
  "id": 101
}

The additional invocation of DW CLI in the pipeline extract the object from the array. The JSON Placeholder won’t preserve our POST so don’t bother looking for it. In conclusion, it appears that we can implement a scripted approach to integrated API systems.

Just because we can doesn’t mean that we should. Error handling, fault tolerance, security and scalability are the hallmarks of a real framework. The DW CLI brings powerful new capabilities to command line scripting not found in other Linux tools.

As I discover other practical uses i’ll add them here.