Statistics
This section of the user guide covers the core statistical functions available in math expressions.
Descriptive Statistics
The describe function returns descriptive statistics for a numeric array.
The describe function returns a single tuple with name/value pairs containing the descriptive statistics.
Below is a simple example that selects a random sample of documents from the logs collection,
vectorizes the response_d field in the result set and uses the describe function to return descriptive statistics about the vector.
let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
b=col(a, response_d),
c=describe(b))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"sumsq": 36674200601.78738,
"max": 1068.854686837548,
"var": 1957.9752647562789,
"geometricMean": 854.1445499569674,
"sum": 42764648.83319176,
"kurtosis": 0.013189848821424377,
"N": 50000,
"min": 656.023249311864,
"mean": 855.2929766638425,
"popVar": 1957.936105250984,
"skewness": 0.0014560741802307174,
"stdev": 44.24901428005237
},
{
"EOF": true,
"RESPONSE_TIME": 430
}
]
}
}Notice that the random sample contains 50,000 records and the response time is only 430 milliseconds. Samples of this size can be used to reliably estimate the statistics for very large underlying data sets with sub-second performance.
The describe function can also be visualized in a table with Zeppelin-Solr:
Histograms and Frequency Tables
Histograms and frequency tables are tools for visualizing the distribution of a random variable.
The hist function creates a histogram designed for usage with continuous data.
The freqTable function creates a frequency table for use with discrete data.
histograms
In the example below a histogram is used to visualize a random sample of response times from the logs collection.
The example retrieves the random sample with the random function and creates a vector from the response_d field in the result set.
Then the hist function is applied to the vector to return a histogram with 22 bins.
The hist function returns a list of tuples with summary statistics for each bin.
let(a=random(logs, q="*:*", fl="response_d", rows="50000"),
b=col(a, response_d),
c=hist(b, 22))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"prob": 0.00004896007228311655,
"min": 675.573084576817,
"max": 688.3309631697003,
"mean": 683.805542728906,
"var": 50.9974629924082,
"cumProb": 0.000030022417162809913,
"sum": 2051.416628186718,
"stdev": 7.141250800273591,
"N": 3
},
{
"prob": 0.00029607514624062624,
"min": 696.2875238591652,
"max": 707.9706315779541,
"mean": 702.1110569558929,
"var": 14.136444379466969,
"cumProb": 0.00022705264963879807,
"sum": 11233.776911294284,
"stdev": 3.759846323916307,
"N": 16
},
{
"prob": 0.0011491235433157194,
"min": 709.1574910598678,
"max": 724.9027194369135,
"mean": 717.8554290699951,
"var": 20.6935845290122,
"cumProb": 0.0009858515418689757,
"sum": 41635.61488605971,
"stdev": 4.549020172412098,
"N": 58
},
...
]}}With Zeppelin-Solr the histogram can be first visualized as a table:
Then the histogram can be visualized with an area chart by plotting the mean of the bins on the x-axis and the prob (probability) on the y-axis:
The cumulative probability can be plotted by switching the y-axis to the cumProb column:
Custom Histograms
Custom histograms can be defined and visualized by combining the output from multiple stats functions into a single histogram.
Instead of automatically binning a numeric field the custom histogram allows for comparison of bins based on queries.
The stats function is first discussed in the Searching, Sampling and Aggregation section of the user guide.
|
A simple example will illustrate how to define and visualize a custom histogram.
In this example, three stats functions are wrapped in a plist function.
The plist (parallel list) function executes each of its internal functions in parallel and concatenates the results into a single stream.
plist also maintains the order of the outputs from each of the sub-functions.
In this example each stats function computes the count of documents that match a specific query.
In this case they count the number of documents that contain the terms copper, gold and silver.
The list of tuples with the counts is then stored in variable a.
Then an array of labels is created and set to variable l.
Finally the zplot function is used to plot the labels vector and the count(*) column.
Notice the col function is used inside of the zplot function to extract the counts from the stats results.
Frequency Tables
The freqTable function returns a frequency distribution for a discrete data set.
The freqTable function doesn’t create bins like the histogram.
Instead it counts the occurrence of each discrete data value and returns a list of tuples with the frequency statistics for each value.
Below is an example of a frequency table built from a result set of rounded differences in daily opening stock prices for the stock ticker amzn.
This example is interesting because it shows a multi-step process to arrive at the result.
The first step is to search for records in the stocks collection with a ticker of amzn.
Notice that the result set is sorted by date ascending and it returns the open_d field which is the opening price for the day.
The open_d field is then vectorized and set to variable b, which now contains a vector of opening prices ordered by date ascending.
The diff function is then used to calculate the first difference for the vector of opening prices.
The first difference simply subtracts the previous value from each value in the array.
This will provide an array of price differences for each day which will show daily change in opening price.
Then the round function is used to round the price differences to the nearest integer to create a vector of discrete values.
The round function in this example is effectively binning continuous data at integer boundaries.
Finally the freqTable function is run on the discrete values to calculate the frequency table.
let(a=search(stocks,
q="ticker_s:amzn",
fl="open_d, date_dt",
sort="date_dt asc",
rows=25000),
b=col(a, open_d),
c=diff(b),
d=round(c),
e=freqTable(d))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 1,
"cumPct": 0.00019409937888198756,
"value": -57
},
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 2,
"cumPct": 0.00038819875776397513,
"value": -51
},
{
"pct": 0.00019409937888198756,
"count": 1,
"cumFreq": 3,
"cumPct": 0.0005822981366459627,
"value": -49
},
...
]}}With Zeppelin-Solr the frequency table can be first visualized as a table:
The frequency table can then be plotted by switching to a scatter chart and selecting the value column for the x-axis and the count column for the y-axis
Notice that the visualization nicely displays the frequency of daily change in stock prices rounded to integers. The most frequently occurring value is 0 with 1494 occurrences followed by -1 and 1 with around 700 occurrences.
Percentiles
The percentile function returns the estimated value for a specific percentile in a sample set.
The example below returns a random sample containing the response_d field from the logs collection.
The response_d field is vectorized and the 20th percentile is calculated for the vector:
let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
b=col(a, response_d),
c=percentile(b, 20))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"c": 818.073554
},
{
"EOF": true,
"RESPONSE_TIME": 286
}
]
}
}The percentile function can also compute an array of percentile values.
The example below is computing the 20th, 40th, 60th and 80th percentiles for a random sample of the response_d field:
let(a=random(logs, q="*:*", rows="15000", fl="response_d"),
b=col(a, response_d),
c=percentile(b, array(20,40,60,80)))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"c": [
818.0835543394625,
843.5590348165282,
866.1789509894824,
892.5033386599067
]
},
{
"EOF": true,
"RESPONSE_TIME": 291
}
]
}
}Quantile Plots
Quantile plots or QQ Plots are powerful tools for visually comparing two or more distributions.
A quantile plot, plots the percentiles from two or more distributions in the same visualization. This allows for visual comparison of the distributions at each percentile. A simple example will help illustrate the power of quantile plots.
In this example the distribution of daily stock price changes for two stock tickers, goog and amzn, are visualized with a quantile plot.
The example first creates an array of values representing the percentiles that will be calculated and sets this array to variable p.
Then random samples of the change_d field are drawn for the tickers amzn and goog.
The change_d field represents the change in stock price for one day.
Then the change_d field is vectorized for both samples and placed in the variables amzn and goog.
The percentile function is then used to calculate the percentiles for both vectors.
Notice that the variable p is used to specify the list of percentiles that are calculated.
Finally zplot is used to plot the percentiles sequence on the x-axis and the calculated percentile values for both distributions on the y-axis.
And a line plot is used to visualize the QQ plot.
This quantile plot provides a clear picture of the distributions of daily price changes for amzn and googl. In the plot the x-axis is the percentiles and the y-axis is the percentile value calculated.
Notice that the goog percentile value starts lower and ends higher than the amzn plot and that there is a steeper slope. This shows the greater variability in the goog price change distribution. The plot gives a clear picture of the difference in the distributions across the full range of percentiles.
Correlation and Covariance
Correlation and Covariance measure how random variables fluctuate together.
Correlation and Correlation Matrices
Correlation is a measure of the linear correlation between two vectors. Correlation is scaled between -1 and 1.
Three correlation types are supported:
-
pearsons (default)
-
kendalls
-
spearmans
The type of correlation is specified by adding the type named parameter in the function call.
In the example below a random sample containing two fields, filesize_d and response_d, is drawn from the logs collection using the random function.
The fields are vectorized into the variables x and y and then Spearman’s correlation for the two vectors is calculated using the corr function.
Correlation Matrices
Correlation matrices are powerful tools for visualizing the correlation between two or more vectors.
The corr function builds a correlation matrix if a matrix is passed as the parameter.
The correlation matrix is computed by correlating the columns of the matrix.
The example below demonstrates the power of correlation matrices combined with 2 dimensional faceting.
In this example the facet2D function is used to generate a two dimensional facet aggregation over the fields complaint_type_s and zip_s from the nyc311 complaints database.
The top 20 complaint types and the top 25 zip codes for each complaint type are aggregated.
The result is a stream of tuples each containing the fields complaint_type_s, zip_s and the count for the pair.
The pivot function is then used to pivot the fields into a matrix with the zip_s field as the rows and the complaint_type_s field as the columns.
The count(*) field populates the values in the cells of the matrix.
The corr function is then used correlate the columns of the matrix.
This produces a correlation matrix that shows how complaint types are correlated based on the zip codes they appear in.
Another way to look at this is it shows how the different complaint types tend to co-occur across zip codes.
Finally the zplot function is used to plot the correlation matrix as a heat map.
Notice in the example the correlation matrix is square with complaint types shown on both the x and y-axises. The color of the cells in the heat map shows the intensity of the correlation between the complaint types.
The heat map is interactive, so mousing over one of the cells pops up the values for the cell.
Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearesttenth).
Covariance and Covariance Matrices
Covariance is an unscaled measure of correlation.
The cov function calculates the covariance of two vectors of data.
In the example below a random sample containing two fields, filesize_d and response_d, is drawn from the logs collection using the random function.
The fields are vectorized into the variables x and y and then the covariance for the two vectors is calculated using the cov function.
If a matrix is passed to the cov function it will automatically compute a covariance matrix for the columns of the matrix.
Notice in the example below that the x and y vectors are added to a matrix. The matrix is then transposed to turn the rows into columns, and the covariance matrix is computed for the columns of the matrix.
let(a=random(logs, q="*:*", fl="filesize_d, response_d", rows=50000),
x=col(a, filesize_d),
y=col(a, response_d),
m=transpose(matrix(x, y)),
covariance=cov(m))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"covariance": [
[
4018404.072532102,
80243.3948172242
],
[
80243.3948172242,
1948.3216661122592
]
]
},
{
"EOF": true,
"RESPONSE_TIME": 534
}
]
}
}The covariance matrix contains both the variance for the two vectors and the covariance between the vectors in the following format:
x y
x [4018404.072532102, 80243.3948172242],
y [80243.3948172242, 1948.3216661122592]The covariance matrix is always square. So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.
Statistical Inference Tests
Statistical inference tests test a hypothesis on random samples and return p-values which can be used to infer the reliability of the test for the entire population.
The following statistical inference tests are available:
-
anova: One-Way-Anova tests if there is a statistically significant difference in the means of two or more random samples. -
ttest: The T-test tests if there is a statistically significant difference in the means of two random samples. -
pairedTtest: The paired t-test tests if there is a statistically significant difference in the means of two random samples with paired data. -
gTestDataSet: The G-test tests if two samples of binned discrete data were drawn from the same population. -
chiSquareDataset: The Chi-Squared test tests if two samples of binned discrete data were drawn from the same population. -
mannWhitney: The Mann-Whitney test is a non-parametric test that tests if two samples of continuous data were pulled from the same population. The Mann-Whitney test is often used instead of the T-test when the underlying assumptions of the T-test are not met. -
ks: The Kolmogorov-Smirnov test tests if two samples of continuous data were drawn from the same distribution.
Below is a simple example of a T-test performed on two random samples. The returned p-value of .93 means we can accept the null hypothesis that the two samples do not have statistically significantly differences in the means.
let(a=random(collection1, q="*:*", rows="1500", fl="price_f"),
b=random(collection1, q="*:*", rows="1500", fl="price_f"),
c=col(a, price_f),
d=col(b, price_f),
e=ttest(c, d))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"e": {
"p-value": 0.9350135639249795,
"t-statistic": 0.081545541074817
}
},
{
"EOF": true,
"RESPONSE_TIME": 48
}
]
}
}Transformations
In statistical analysis its often useful to transform data sets before performing statistical calculations. The statistical function library includes the following commonly used transformations:
-
rank: Returns a numeric array with the rank-transformed value of each element of the original array. -
log: Returns a numeric array with the natural log of each element of the original array. -
log10: Returns a numeric array with the base 10 log of each element of the original array. -
sqrt: Returns a numeric array with the square root of each element of the original array. -
cbrt: Returns a numeric array with the cube root of each element of the original array. -
recip: Returns a numeric array with the reciprocal of each element of the original array.
Below is an example of a ttest performed on log transformed data sets:
let(a=random(collection1, q="*:*", rows="1500", fl="price_f"),
b=random(collection1, q="*:*", rows="1500", fl="price_f"),
c=log(col(a, price_f)),
d=log(col(b, price_f)),
e=ttest(c, d))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"e": {
"p-value": 0.9655110070265056,
"t-statistic": -0.04324265449471238
}
},
{
"EOF": true,
"RESPONSE_TIME": 58
}
]
}
}Back Transformations
Vectors that have been transformed with the log, log10, sqrt and cbrt functions can be back transformed using the pow function.
The example below shows how to back transform data that has been transformed by the sqrt function.
let(echo="b,c",
a=array(100, 200, 300),
b=sqrt(a),
c=pow(b, 2))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"b": [
10,
14.142135623730951,
17.320508075688775
],
"c": [
100,
200.00000000000003,
300.00000000000006
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}The example below shows how to back transform data that has been transformed by the log10 function.
let(echo="b,c",
a=array(100, 200, 300),
b=log10(a),
c=pow(10, b))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"b": [
2,
2.3010299956639813,
2.4771212547196626
],
"c": [
100,
200.00000000000003,
300.0000000000001
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}Vectors that have been transformed with the recip function can be back-transformed by taking the reciprocal of the reciprocal.
The example below shows an example of the back-transformation of the recip function.
let(echo="b,c",
a=array(100, 200, 300),
b=recip(a),
c=recip(b))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"b": [
0.01,
0.005,
0.0033333333333333335
],
"c": [
100,
200,
300
]
},
{
"EOF": true,
"RESPONSE_TIME": 0
}
]
}
}Z-scores
The zscores function converts a numeric array to an array of z-scores.
The z-score is the number of standard deviations a number is from the mean.
The example below computes the z-scores for the values in an array.
let(a=array(1,2,3),
b=zscores(a))When this expression is sent to the /stream handler it responds with:
{
"result-set": {
"docs": [
{
"b": [
-1,
0,
1
]
},
{
"EOF": true,
"RESPONSE_TIME": 27
}
]
}
}