The Streaming Expression language includes a powerful statistical programing syntax with many of the features of a functional programming language.
The syntax includes variables, data structures and a growing set of mathematical functions.
Using the statistical programing syntax, Solr’s powerful data retrieval capabilities can be combined with indepth statistical analysis.
The data retrieval methods include:

SQL

time series aggregation

random sampling

faceted aggregation

KNearest Neighbor (KNN) searches

topic
message queues 
MapReduce (parallel relational algebra)

JDBC calls to outside databases

Graph Expressions
Once the data is retrieved, the statistical programming syntax can be used to create arrays from the data so it can be manipulated, transformed and analyzed.
The statistical function library includes functions that perform:

Correlation

Crosscorrelation

Covariance

Moving averages

Percentiles

Simple regression and prediction

Analysis of covariance (ANOVA)

Histograms

Convolution

Euclidean distance

Descriptive statistics

Rank transformation

Normalization transformation

Sequences

Array manipulation functions (creation, copying, length, scaling, reverse, etc.)
The statistical function library is backed by Apache Commons Math library. A full discussion of many of the math functions available to streaming expressions is available in the section Stream Evaluator Reference.
This document provides an overview of the how to apply the variables, data structures and mathematical functions.
Like all streaming expressions, the statistical functions are run by Solr’s /stream handler. For an overview of this handler, see the section Streaming Expressions.

Math Functions
Streaming expressions contain a suite of mathematical functions which can be called on their own or as part of a larger expression.
Solr’s /stream
handler evaluates the mathematical expression and returns a result.
For example, if you send the following expression to the /stream
handler:
add(1, 1)
You get the following response:
{
"resultset": {
"docs": [
{
"returnvalue": 2
},
{
"EOF": true,
"RESPONSE_TIME": 2
}
]
}
}
You can nest math functions within each other. For example:
pow(10, add(1,1))
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": 100
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
You can also perform math on a stream of Tuples. For example:
select(search(collection2, q="*:*", fl="price_f", sort="price_f desc", rows="3"),
price_f,
mult(price_f, 10) as newPrice)
Returns the following response:
{
"resultset": {
"docs": [
{
"price_f": 0.99999994,
"newPrice": 9.9999994
},
{
"price_f": 0.99999994,
"newPrice": 9.9999994
},
{
"price_f": 0.9999992,
"newPrice": 9.999992
},
{
"EOF": true,
"RESPONSE_TIME": 3
}
]
}
}
Data Structures
Several types of data can be manipulated with the statistical programming syntax. The following sections explore arrays, tuples, and lists.
Arrays
The first data structure we’ll explore is the array.
We can create an array with the array
function:
For example:
array(1, 2, 3)
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": [
1,
2,
3
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
We can nest arrays within arrays to form a matrix:
array(array(1, 2, 3),
array(4, 5, 6))
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": [
[
1,
2,
3
],
[
4,
5,
6
]
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
We can manipulate arrays with functions. For example, we can reverse an array with the rev
function:
rev(array(1, 2, 3))
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": [
3,
2,
1
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Arrays can also be built and returned by functions. For example, the sequence
function:
sequence(5,0,1)
This returns an array of size 5
starting from 0
with a stride of 1
.
{
"resultset": {
"docs": [
{
"returnvalue": [
0,
1,
2,
3,
4
]
},
{
"EOF": true,
"RESPONSE_TIME": 4
}
]
}
}
We can perform math on an array. For example, we can scale an array with the scale
function:
scale(10, sequence(5,0,1))
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": [
0,
10,
20,
30,
40
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
We can perform statistical analysis on arrays For example, we can correlate two sequences with the corr
function:
corr(sequence(5,1,1), sequence(5,10,10))
Returns the following response:
{
"resultset": {
"docs": [
{
"returnvalue": 1
},
{
"EOF": true,
"RESPONSE_TIME": 1
}
]
}
}
Tuples
The tuple is the next data structure we’ll explore.
The tuple
function returns a map of name/value pairs. A tuple is a very flexible data structure that can hold values that are strings, numerics, arrays and lists of tuples.
A tuple can be used to return a complex result from a statistical expression.
Here is an example:
tuple(title="hello world",
array1=array(1,2,3,4),
array2=array(4,5,6,7))
Returns the following response:
{
"resultset": {
"docs": [
{
"title": "hello world",
"array1": [
1,
2,
3,
4
],
"array2": [
4,
5,
6,
7
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Lists
Next we have the list data structure.
The list
function is a data structure that wraps streaming expressions and emits all the tuples from the wrapped expressions as a single concatenated stream.
Below is an example of a list of tuples:
list(tuple(id=1, data=array(1, 2, 3)),
tuple(id=2, data=array(10, 12, 14)))
Returns the following response:
{
"resultset": {
"docs": [
{
"id": "1",
"data": [
1,
2,
3
]
},
{
"id": "2",
"data": [
10,
12,
14
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}
Setting Variables with let
The let
function sets variables and runs a streaming expression that references the variables. The let
function can be used to
write small statistical programs.
A variable can be set to the output of any streaming expression. Here is a very simple example:
let(a=random(collection2, q="*:*", rows="3", fl="price_f"),
b=random(collection2, q="*:*", rows="3", fl="price_f"),
tuple(sample1=a, sample2=b))
The let
expression above is setting variables a
and b
to random
samples taken from collection2.
The let
function then executes the tuple
streaming expression
which references the two variables.
Here is the output:
{
"resultset": {
"docs": [
{
"sample1": [
{
"price_f": 0.39729273
},
{
"price_f": 0.063344836
},
{
"price_f": 0.42020327
}
],
"sample2": [
{
"price_f": 0.659244
},
{
"price_f": 0.58797807
},
{
"price_f": 0.57520163
}
]
},
{
"EOF": true,
"RESPONSE_TIME": 20
}
]
}
}
Creating Arrays with col Function
The col
function is used to move a column of numbers from a list of tuples into an array
.
This is an important function because streaming expressions such as sql
, random
and timeseries
return tuples, but the statistical functions operate on arrays.
Below is an example of the col
function:
let(a=random(collection2, q="*:*", rows="3", fl="price_f"),
b=random(collection2, q="*:*", rows="3", fl="price_f"),
c=col(a, price_f),
d=col(b, price_f),
tuple(sample1=c, sample2=d))
The example above is using the col
function to create arrays from the tuples stored in variables a
and b
.
Variable c
contains an array of values from the price_f
field,
taken from the tuples stored in variable a
.
Variable d
contains an array of values from the price_f
field,
taken from the tuples stored in variable b
.
Also notice inn that the response tuple
executed by let
is pointing to the arrays in variables c
and d
.
The response shows the arrays:
{
"resultset": {
"docs": [
{
"sample1": [
0.06490427,
0.6751543,
0.07063508
],
"sample2": [
0.8884564,
0.8878821,
0.3504665
]
},
{
"EOF": true,
"RESPONSE_TIME": 17
}
]
}
}
Statistical Programming Example
We’ve covered how the data structures, variables and a few statistical functions work. Let’s dive into an example that puts these tools to use.
Use Case
We have an existing hotel in cityA that is very profitable. We are contemplating opening up a new hotel in a different city. We’re considering 4 different cities: cityB, cityC, cityD, cityE. We’d like to open a hotel in a city that has similar room rates to cityA.
How do we determine which of the 4 cities we’re considering has room rates which are most similar to cityA?
The Data
We have a data set of unaggregated hotel bookings. Each booking record has a rate and city.
Can We Simply Aggregate?
One approach would be to aggregate the data from each city and compare the mean room rates. This approach will give us some useful information, but the mean is a summary statistic which loses a significant amount of information about the data. For example, we don’t have an understanding of how the distribution of room rates is impacting the mean.
The median room rate provides another interesting data point but it’s still not the entire picture. It’s sill just one point of reference.
Is there a way that we can compare the markets without losing valuable information in the data?
KNearest Neighbor
The use case we’re reasoning about can often be approached using a KNearest Neighbor (knn) algorithm.
With knn we use a distance measure to compare vectors of data to find the k nearest neighbors to a specific vector.
Euclidean Distance
The streaming expression statistical function library has a function called distance
. The distance
function
computes the Euclidean distance between two vectors. This looks promising for comparing vectors of room rates.
Vectors
But how to create the vectors from a our data set? Remember we have unaggregated room rates from each of the cities.
How can we vectorize the data so it can be compared using the distance
function.
We have a streaming expression that can retrieve a random sample from each of the cities. The name of this
expression is random
. So we could take a random sample of 1000 room rates from each of the five cities.
But random vectors of room rates are not comparable because the distance algorithm compares values at each index in the vector. How can make these vectors comparable?
We can make them comparable by sorting them. Then as the distance algorithm moves along the vectors it will be comparing room rates from lowest to highest in both cities.
The Code
let(cityA=sort(random(bookings, q="city:cityA", rows="1000", fl="rate_d"), by="rate_d asc"),
cityB=sort(random(bookings, q="city:cityB", rows="1000", fl="rate_d"), by="rate_d asc"),
cityC=sort(random(bookings, q="city:cityC", rows="1000", fl="rate_d"), by="rate_d asc"),
cityD=sort(random(bookings, q="city:cityD", rows="1000", fl="rate_d"), by="rate_d asc"),
cityE=sort(random(bookings, q="city:cityE", rows="1000", fl="rate_d"), by="rate_d asc"),
ratesA=col(cityA, rate_d),
ratesB=col(cityB, rate_d),
ratesC=col(cityC, rate_d),
ratesD=col(cityD, rate_d),
ratesE=col(cityE, rate_d),
top(n=1,
sort="distance asc",
list(tuple(city=B, distance=distance(ratesA, ratesB)),
tuple(city=C, distance=distance(ratesA, ratesC)),
tuple(city=D, distance=distance(ratesA, ratesD)),
tuple(city=E, distance=distance(ratesA, ratesE)))))
The Code Explained
The let
expression sets variables first.
The first 5 variables (cityA, cityB, cityC, cityD, cityE), contain the random samples from the bookings
collection.
The random
function is pulling 1000 random samples from each city and including the rate_d
field in the
tuples that are returned.
The random
function is wrapped by a sort
function which is sorting the tuples in
ascending order based on the rate_d
field.
The next five variables (ratesA, ratesB, ratesC, ratesD, ratesE) contain the arrays of room rates for each
city. The col
function is used to move the rate_d
field from the random sample tuples
into an array for each city.
Now we have five sorted vectors of room rates that we can compare with our distance
function.
After the variables are set the let
expression runs the top
expression.
The top
expression is wrapping a list
of tuples
. Inside each tuple the distance
function is used to compare
the rateA vector with one of the other cities. The output of the distance function is stored in the distance field
in the tuple.
The list
function emits each tuple
and the top
function returns only the tuple with the lowest distance.
We welcome feedback on Solr documentation. However, we cannot provide application support via comments. If you need help, please send a message to the Solr User mailing list.