ProjectREST: Real-time, scalable, self-managing storage for Telecoms

REST: Real-time, scalable, self-managing storage for Telecoms

Subscriber databases are among the most high performance, highly available computer systems in the world, managing tens of millions of users and handling thousands of transactions per second, whilst being almost always available. High performance, reliable, real-time subscriber databases have long been a key building block for Telecom services, while in recent years, they have been adopted by Internet companies, often building in-house using open-source software, to deliver widely used services, including Amazon, Facebook, Wikipedia, Youtube and Twitter. The goal of REST project is to design and build a new architecture for a subscriber database based on a structured overlay network (SON) that will run over multiple data centres on common off-the-shelf (COTS) hardware. This research plan involves developing:

  • smart data partitioning algorithms for the SON to enable increased storage size and system throughput,
  • data-aware and network-aware consensus algorithms for transactions to improve availability in the face of network partitions,
  • a message passing component model to improve scalability for multi-core hardware,
  • self-managing features to reduce system administration costs.

In REST project, we develop a new SON architecture for subscriber systems that will support a higher number of transactions/second and store a larger amount of data storage per user than existing systems, while maintaining real-time guarantees on data read and write latencies.  REST is required to deliver a system that can adapt not just to node failures but also network partitions. REST will "intelligently" adapt its replication algorithms and their consistency models to network partitions. Similarly, intelligence will be required to build replication algorithms that learn about the frequency with which subscriber attributes are updated. REST will be "faster" than existing systems by virtue of the SON handling a higher number of subscriber operations in parallel, as well our multicore support, which will mean fewer machines are required to manage the same number of subscribers. Fewer machines will result in power savings that will increase proportionally to the number of cores in a system.