Microsoft Research reveals details of new streaming analytics engine

Sean Dudley
Sean Dudley
By Sean Dudley on 28 January 2015
Microsoft Research reveals details of new streaming analytics engine

Details of a new high-performance streaming analytics engine developed by Microsoft researchers have been revealed.

Named Trill, the engine can process data at two to four orders of magnitude faster than modern streaming engines.

One surprising fact about Trill – which stemmed from a research project based on the idea of ‘a trillion events per day’ – is that it is ‘just a .NET library’ according to a post on the Inside Microsoft Research page by George Thomas Jr.

‘As a single-node engine library, any .NET application, service, or platform can easily include it and start processing queries’ the post reads.

Trill also features a temporal query language which allows users to ‘express complex queries over real-time and/or offline data sets’ and get results significantly faster than previously possible thanks to the analytics engine’s ‘high performance across intended usage scenarios’.

Badrish Chandramouli, one of the Microsoft researchers who developed Trill, said: “Prior systems have only achieved subsets of these benefits, but Trill provides all of these advantages in one package, so to speak.”

Trill uses new techniques and algorithms that process events in batches. The data within these batches are then organised in a way that enables queries to execute more efficiently.

The engine has already helps Bing Ads customers, who, thanks to the Trill paradigm shift, are seeing results in less than an hour of launching Bing ad campaigns.

“While it can be integrated into today’s distribution fabrics such as SCOPE (in Bing ads) and Orleans (in Halo) to achieve scale-out, we are currently looking at developing new techniques to achieve even better performance in distributed computing and Internet-of-Things scenarios,” Chandramouli said.

The project began back in 2012, and Trill’s roots can be traced to earlier research in Complex Event Detection and Response algebra (CEDR), which was carried out in 2007.

Though not available to the public, Trill is currently being used as a query processor within the Azure Stream Analytics service.

“From CEDR to Trill to multiple Microsoft products: This body of work is a great example of how within Microsoft Research we evolve from science to technology to business impact,” said Jeannette Wing, corporate vice president of Microsoft Research. “It also shows the nature and value of long-term research, where patience and persistence really pay off.”

More information on Trill can be found in the whitepaper Trill: A High-Performance Incremental Query Processor for Diverse Analytics

Number of views (3700)/Comments (-)

Comments are only visible to subscribers.

Theme picker