Skip to main content

V1 - Fluency Processing Language Concepts

Fluency Processing Language (FPL) - River Analytics (Ra) is a programming language designed to analyze and manipulate big data records.

function example()
    search {from="-8h"} sContent("@tags","fpl-example-data")
    let {id, isx5, isprime, odd, even, divisors} = f("@fields")
    aggregate v=values(id) by divisors
    let num_of_ints = listcount(v)
end

stream demo_table=example()

Basics​

Terms​

Task​

A 'task' is the largest unit of execution within FPL. it is comprised of one or more ‘stream’ blocks, which are themselves collections of ‘pipes’, each on a separate line. In concept, this is analogous to a 'program' or 'script'.

Each run or execution of the FPL is considered a new 'task'. The task history is recorded in the Fluency system.

Stream​

A 'stream' block or 'function' is a smaller unit within FPL. It is denoted by the following (equivalent) syntax:

function example()
    // pipes
end

or

function example(){
    // pipes
}

The stream block is self-contained. At the end of execution, the stream block will return the results of the inner FPL pipe commands in a 'table' of columns.

To execute a stream block, use the following syntax:

stream demo_table=example()

This will execute the example stream block, and store the resulting table in "demo_table".

The stream block can also accept arguments, as seen the following example:

function example(from_date, to_date, magic_str)
    search {from=from_date, to=to_date} sContains("",magic_str)
    let event_source = f("@source")
    aggregate total_by_src=count() by event_source
end

stream demo_table1=example("20220501", "20220601", "test")

Pipe​

The smallest unit of execution is a 'pipe'. In FPL, each pipe is defined on their own separate line.

A collection of 'pipe' commands make up a 'stream' block, but 'pipe' commands can also be used outside of a block, within the FPL task itself.

The input / output of a pipe can vary greatly, depending on the type and function.

The details of each available pipe of command is found in the rest of this documnet.

Please visit the relevant sub-section(s) for detailed information on a particular command of the Fluency Processing Language.

Logic Flow​

The goal of any FPL is to transform a collection of log / resource data into a table format. During the transformation process, additional actions can be performed to process the data in various, and potentially more meaningful ways.

Data Selection​

The FPL task begins with data selection. The selection of data is based on an expression that searches the entire dataset. Often, it is done with the search command as the first line. Search selects all the data matching the 'query' criteria within a specified 'time' range.

Data Extraction​

The data extraction step chooses and 'extracts' the desired data field from the selected data. Normally this is done via the let assignment command, used in onjunction with the field-extraction function f(). Notice that f() has parentheses. It is a function, not a command. It returns an object of the selected field(s).

Data extraction transforms the fields of the data object into "variables" or "columns", in preparation for the table data transformation.

Data Processing​

The data processing portion encompasses a wide range of actions that can manipulate / process the raw data to more useful forms. This is also the main difference between FPL and a simple query expression. An analytic capability here allows the user to performs complex calculations over the dataset.

With the data in a "columns", following the data extraction step, aggregation functions, such as sum, min/max, distinct, and stdev can be performed. Advanced functions, such as table join are also available.

Additional utility functions, such as toString, len, and regexp can also be used.

Results and Graphs​

Running an FPL task produce one or more results tables. These tables can use used as-in, and exported as CSV files. Or they can be enhanced into graphical elements, such as Charts and Graphs, and even combined into published Reports.

The default number of rows is set to ten (10). This can be changed with the sort command:

search { from="-7d<d", to=">d" }
assign source=f("@source")

aggregate total=count() by source
sort 5 total

Charts/Graphs and Reports​

After a task is run and the result tabled obtained, the next step in the FPL workflow is to present the results.

The Fluency interface provides a means to produce and publish graphical reports using data tables from FPL tasks. Additionally, once published as a report, the FPL task can be schedule to run a regular intervals. The result of the most recent report execution will be shown as a Dashboard.

Summary​

FPL functional programming language that is designed to work on big data. The basic analytic expression is classified in three categories, data selection, extraction, and processing.

    • The selection timeframe is defined by a form and to value. These are relatively defined by time periods of minute, hour, day, week, and month. 
    • The extraction (assignment) command defines the columns and labels them with the variable name.
    • The processing (aggregat)e variables are the results of function for the dataset in the timeframe. 
    • A sort command determines the order of listing and number of maximum rows.

Page last updated: 2023 Mar 21