This tutorial uses the Composer Processor to process streaming data in windows (sliding, tumbling and session). In this tutorial you will learn how to apply these streaming windows to our sandbox templates.


Tumbling Windows

The templates for this tutorial can be downloaded here

Tumbling windows are used to segment a data stream into distinct time segments. Each segment corresponds to the real time.

Using the Hello source we will create a simple pipeline which writes Hello World and the current timestamp to a CSV file.

We will use a tumbling window with interval of 10 seconds. 

Rules option must be set to rule append.

We want to write one record per second so set the Hello Component Policy as below threads 1 and batch 1.


Check the output, each window is 10 seconds long and follows on from the previous window. 




Session Windows

Session windows are grouped by the data arrival time, the session window will keep increasing until there is a period of inactivity > the interval.

In this example if no data is received for 10 seconds the data will be written to a CSV file.

We are using the same pipeline as the Tumbling window example.

To simulate no data for 10 seconds we have added some simple javascript code to the cook processor which does not output data if the current second > 48.

var d = new Date();
var n = d.getSeconds();
if (n < 49) output.message = input + " - " + (d);
CODE

We want to write one record per second so set the Hello Component Policy as below threads 1 and batch 1.

Check the output, you will notice each window is 50 seconds long although the window interval is only 10 seconds.

Please be aware that if the period of inactivity is not reached the data will continue to be written to memory.


Sliding Windows

Sliding windows are not linked to the current time, the window will start when data is received.

To simulate a break in data greater than the window size we add some simple javascript to the cook processor.

var d = new Date();
var n = d.getSeconds();
if (n <=  30 || n >= 45) output.message = input + " - " + (d);
CODE

We will use a sliding window with interval of 10 seconds. 

Rules option must be set to rule append.


Check the output, You will notice that the third window does not follow on from the previous window.