Complex Event Processing in Esper - UiO

applications. © Jarle Søberg, 2010 .... These graphs are manually created by the application programmer ... optimization techniques hidden to the application.
667KB Sizes 26 Downloads 193 Views
Data Stream Management and Complex Event Processing in Esper INF5100, Autumn 2010 Jarle Søberg

Outline • Overview of Esper • DSMS and CEP concepts in Esper – Examples taken from the documentation • A lot of possibilities • We focus at the ones that extend traditional SQL

• Esper’s query processing model • Code example taken from the documentation – Neither the concept descriptions nor the code example are complete – Consult with the documentation for more details • © Jarle Søberg, 2010

Overview of Esper • A Java/.NET library for complex event processing – Stream based – Our examples are from the Java version

• Processes complex queries written in a language called EPL (event processing language) – Uses DBMS, DSMS and CEP concepts

• The code is available at • Tutorial(s), code examples

• Open source • Can be used in several data stream and CEP applications © Jarle Søberg, 2010

Architecture Overview • The main elements in the Esper architecture resemble any DSMS/CEP system – Input data is processed by one or more queries Esper engine

EPL queries

Listeners Subscribers © Jarle Søberg, 2010

JDBC adapter

Event objects



Data Model (2) • A data stream of event objects

The parenthesis denote the section/chapter in the documentation where this is described.

– Attributes that are defined with types and values • String, boolean, integer, long, float, byte, ... • One needs to write a method that converts to event objects from the data stream or file

– Represented as Java objects

• Stream types – Input stream: istream • All the new events arriving, and entering a data window or aggregation • The default stream type

– Remove stream: rstream • All the old events leaving a data window or aggregation

– Both: irstream

© Jarle Søberg, 2010

Event Declaration (2) •

The event objects can be declared by –

Schema •


create schema schema_name [as] (property_name property_type [,property_name property_type [,...]) [inherits inherited_event_type[, inherited_event_type] [,...]] Events can be represented as org.w3c.dom.Node instances

Code package org.myapp.event; public class OrderEvent { private String itemName; private double price; public OrderEvent(String itemName, double price) { this.itemName = itemName; this.price = price; } public String getItemName() { return itemName; } public double getPrice() { return price; } }

Many other types •

E.g. java.util.Map

© Jarle Søberg, 2010

<xs:schema xmlns:xs=" 2001/XMLSchema"> <xs:element name="Sensor"> <xs:complexType> <xs:sequence> <xs:element name="ID" type="xs:string"/> <xs:element ref="Observation" /> ...

Creating Events • Create a new event object and send the object to the event processor runtime: OrderEvent event = new OrderEvent("shirt", 74.50); epService.getEPRuntime(). sendEvent(event);

© Jarle Søberg, 2010

Windows (9) • •

In Esper a window is called a view Provides many types of windows/views – some examples:




Length window


Sliding window by number of elements

Length batch window


Tumbling window up to size events

Time window

win:time(time period)

Sliding time window

Externally-timed window

win:ext_timed(timestamp expression, time period)

Sliding time window, based on the millisecond tim