Tutorial: Web Services & Network Sniffing

A step by step tutorial for capturing live web service traffic using Datumize Data Collector (DDC) and Datumize Data Dumper (DDD), then process each web service call to extract metrics, and produce a file with persisted data to be visualized. This tutorial provides the basic concepts to understand how to using Datumize products to utilize network traffic by doing network sniffing and deep packet inspection.
Overview

This tutorial currently is only supported in Java version 11.

The Sniffing Web Services Sandbox contains just one source that will be the files generated from the traffic capture. This traffic capture will contain Web Service traffic on it generated by a client-server API. Take in to account that this is a traffic generator and not real webservices. From this traffic file the DDC will be then be able to parse the traffic into something understandable for human beings.

Objectives

In this tutorial you will learn: 

• Use Datumize Data Dumper (DDD) to capture network traffic
• Design a pipeline that captures network traffic in PCAP format.
• How to parse data from Web Services.
• Sink the data to a file into a CSV format. 

Requirements

Make sure to comply with the following requirements.

Check Web Service traffic generation

Check the traffic that we have in this machine. In this scenario we simulate real Web Service traffic from one port in the localhost to another port in localhost. To check the current traffic we execute tcpdump:

sudo tcpdump -i any -A -vvv | grep POST

The output of this will be something like this:

.	...	..POST /api/call HTTP/1.1
.	.v.	.vPOST /api/call HTTP/1.1
.	...	..POST /api/call HTTP/1.1
.	.v.	.vPOST /api/call HTTP/1.1
.	...	..POST /api/call HTTP/1.1

This shows us that we have traffic in the machine and you will be able to capture it by using the DDD and process it by using the DDC.

Configure and Deploy DDD

In order to configure a DDD you must first create a machine.

First of all, create a new Deployment plan and then go to Infrastructure tab and click on Add machine. Once your machine is created and bootstrapped you can proceed with configuring the DDD.

Bootstrap the machine as shown in the Bootstrap Datumize software installation.

Click the Add instance button:

Select DDD and give your DDD instance a name.

Other fields can be left with the default values.

The Deploy machine button should now be activated.

Once deployed and after Apply Changes your DDD instance should look like this:

Check output of DDD

The datumize main folder is always /opt/datumize/, so all can be found under that one. Check the generated pcaps:

ls -alt /opt/datumize/pcap

-rw-r--r-- 1 datumize datumize  778070 May 22 15:18 2020-05-22_15-18-08.pcap
-rw-r--r-- 1 datumize datumize  811149 May 22 15:18 2020-05-22_15-17-48.pcap
-rw-r--r-- 1 datumize datumize  766058 May 22 15:17 2020-05-22_15-17-28.pcap
-rw-r--r-- 1 datumize datumize  812000 May 22 15:17 2020-05-22_15-17-08.pcap
-rw-r--r-- 1 datumize datumize  774565 May 22 15:17 2020-05-22_15-16-48.pcap
-rw-r--r-- 1 datumize datumize 1042549 May 22 15:16 2020-05-22_15-16-28.pcap

If your DDC pipeline is not deployed and running your DDD will fill the output folder with pcaps, you can stop the DDD instance and restart it later.

If you want to fully understand how the traffic generator works, please expand this section.

Build the Web Services Pipeline

If you feel lost about using Datumize Zentral, please refer to the available resources in our Getting Started section, including manuals and videos.

The table below summarizes the components used in the pipeline.

Drag the required components from the Pallet to the Workbench and join with a Single Memory Stream.

The table below summarizes the properties to configure the components.

FilePcapSource component:

HTTPDialogGroupProcessor component:

HTTPAssemblerProcessor component:

WSParserProcessor component:

CookProcessor component:

Please note the code in the cook processor is very important for correctly enabling this tutorial.

if (input.containsKey('response')) {
    rq = input.get('request')
    ops = rq.get('operations').asList()
    output.count = ops.size()

    if (ops.size() > 0 && ops.get(1) != null && ops.get(1).get('operation') != null){
            output.firstop = ops.get(1).get('operation').get('func').value()
    }

    rs = input.get('response')
    rsops = rs.get('operations').asList()
    if (rsops.get(1) != null && rsops.get(1).get('operation') != null && rsops.get(1).get('operation').get('result') && rsops.get(1).get('operation').get('result').get('value') != null ){
        output.firstresult = rsops.get(1).get('operation').get('result').get('value').value()
    }
}

SerializerProcessor component:

ComposerProcessor component:

FileSink component:

Test the Pipeline

To test the pipeline you have created please see the guide for Testing a Pipeline for a more in depth overview of the steps necessary to test your pipeline.

For this tutorial, in the pipeline editor page you can select Run from: Beginning. Your Input and Output should show your Input, and Output results if successful. 

Deploy the Pipeline to DDC Instance

In Zentral, you will only need to have one machine, one instance and one Pipeline. For more on using the Infrastructure management tool please see the Zentral guide to Infrastructure Deployment

This pipeline will be deployed with the default  DDC Runtime Policy.

Check the Expected Output

1. As long  as the output is configured as a csv, you will have to check the output folder where it contains the resulting csv's:

   ls -alt /opt/datumize/resources/

   CODE

   # ls -alt /opt/datumize/resources/csv/20200206/1127/
   total 28
   drwxr-xr-x 4 root root  4096 Feb  6 11:29 ..
   drwxr-xr-x 2 root root  4096 Feb  6 11:28 .
   -rw-r--r-- 1 root root   234 Feb  6 11:28 73d96893-7915-49c9-836d-2cf1198d0c30
   -rw-r--r-- 1 root root 15426 Feb  6 11:27 26d430d9-be5a-4b3d-8f95-602fe9f9bfa0

   CODE

2. Check inside the csv by using less:

   # less /opt/datumize/resources/csv/20200206/1127/73d96893-7915-49c9-836d-2cf1198d0c3

   CODE

   The file contains the following that is the result of the webservices dialogs captured by the DDD and processed by the DDC:

   100,sub,21985.0
   100,div,0.9670221843003413
   100,sub,10669.0
   100,div,0.7495874063729846
   100,add,54708.0
   100,div,4.801460823373174
   100,sub,34699.0
   100,div,0.7584804068202213
   100,div,0.39869337979094077
   100,add,58887.0
   100,multi,234696.0

   CODE

Stop All

Automatically

1. By using Zentral you can decide to stop the process at any time just by 

Manually

1. Now, to finish, you can stop all the services that have ran in the machine. First stop the DDD and then the DDC:

   /opt/datumize/dtzdump/dtzdump.sh stop sandbox
   
   /opt/datumize/ddc/bin/ddc stop ws

   CODE

2. To stop the traffic generator, just execute this in the its folder:

   bash stop-traffic-generator.sh

   CODE