org.apache.flink.examples.scala.relational

Class WebLogAnalysis

java.lang.Object

org.apache.flink.examples.scala.relational.WebLogAnalysis

public class WebLogAnalysis
extends Object

This program processes web logs and relational data. It implements the following relational query:

 SELECT
       r.pageURL,
       r.pageRank,
       r.avgDuration
 FROM documents d JOIN rankings r
                  ON d.url = r.url
 WHERE CONTAINS(d.text, [keywords])
       AND r.rank > [rank]
       AND NOT EXISTS
           (
              SELECT * FROM Visits v
              WHERE v.destUrl = d.url
                    AND v.visitDate < [date]
           );
 

Input files are plain text CSV files using the pipe character ('|') as field separator. The tables referenced in the query can be generated using the [org.apache.flink.examples.java.relational.util.WebLogDataGenerator} and have the following schemas

 CREATE TABLE Documents (
                url VARCHAR(100) PRIMARY KEY,
                contents TEXT );

 CREATE TABLE Rankings (
                pageRank INT,
                pageURL VARCHAR(100) PRIMARY KEY,
                avgDuration INT );

 CREATE TABLE Visits (
                sourceIP VARCHAR(16),
                destURL VARCHAR(100),
                visitDate DATE,
                adRevenue FLOAT,
                userAgent VARCHAR(64),
                countryCode VARCHAR(3),
                languageCode VARCHAR(6),
                searchWord VARCHAR(32),
                duration INT );
 

Usage

   WebLogAnalysis --documents <path> --ranks <path> --visits <path> --output <path>
 

If no parameters are provided, the program is run with default data from WebLogData.

This example shows how to use:

- tuple data types - projection and join projection - the CoGroup transformation for an anti-join

Constructor Summary

Constructors

Constructor and Description
WebLogAnalysis()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WebLogAnalysis

public WebLogAnalysis()

Method Detail

main

public static void main(String[] args)