Kylin is an open source Distributed Analytics Engine from eBay Inc offering a different kind of OLAP, and specifically OLAP that works on very large data sets (10+ billion rows of data). It seems the product overcomes many of the problems associated with traditional OLAP models.
MOLAP (Multi-dimensional OLAP) is to pre-compute data along different dimensions of interest and store resultant values in the cube. MOLAP is much faster but is inflexible. ROLAP (Relational-OLAP) is to use star or snow-flake schema to do runtime aggregation. ROLAP is flexible but much slower. Kylin builds data cube (MOLAP) from hive table (ROLAP) according to the metadata definition. If the query can be fulfilled by data cube, Kylin will route the query to data cube that is MOLAP. If the query can’t be fulfilled by data cube, Kylin will route the query to hive table that is ROLAP. Basically, you can think Kylin as HOLAP on top of MOLAP and ROLAP.
It offers ANSI-SQL on Hadoop and supports most ANSI-SQL query functions, and currently offers integration capability with BI Tools like Tableau. Integration with Microstrategy and Excel is coming soon.
Kylin depends on HDFS, MapReduce, Hive and HBase. Hive and MapReduce is used for cube building. Hive is used for pre-join and MapReduce is used for pre-aggregation. HDFS is used to store intermediated files during cube building. HBase is used to store data cube and answer the query. HBase coprocessor is also used for query processing. Fundamental framework of Kylin OLAP Engine comprises of Metadata Engine, Query Engine, Job Engine and Storage Engine to run the entire stack. It also includes a REST Server to service client requests. Some features of the product include:Job Management and Monitoring
- Compression and Encoding Support
- Incremental Refresh of Cubes
- Leverage HBase Coprocessor for query latency
- Approximate Query Capability for distinct Count (HyperLogLog)
- Easy Web interface to manage, build, monitor and query cubes
- Security capability to set ACL at Cube/Project Level
- Support LDAP Integration