RISE SICS Open House 2017 - Theofilos Kakantousis



Photo of Theofilos Kakantousis                      Theofilos Kakantousis, Researcher,  RISE SICS

Hops FS is the first production-grade distributed hierarchical filesystem to store metadata normalized in an in-memory, shared nothing database. In this talk, we present how HopsFS reached a performance milestone by scaling the Hadoop Filesystem (HDFS) by 16X, from 70K ops/s to 1.2 million ops/s on Spotiy's industrial Hadoop workload. We discuss the challenges in building secure multi-tenant streaming applications on YARN that are metered and easy-to-debug. We also show how users in Hopsworks use the ELK stack for logging their running streaming applications as well as how they use Grafana and Graphite for monitoring.

Multi-tenant Streaming-as-a-Service with Hops

Presentation slides