ProjectScalable Transactional Storage for Web 2.0 Services

Scalable Transactional Storage for Web 2.0 Services

Scalaris is a self-managing scalable transactional database that is suitable for building global services. This and similar systems will be core components in future Cloud Computing environments and Web 2.0 services. Scalaris is designed and implemented by the Zuse Institute Berlin, KTH, and SICS as part of the European SELFMAN project.

Scalaris provides a distributed key/value store on the top of a replicated storage layer and an enhanced structured P2P overlay network. The service has high availability and supports strong data consistency in the face of concurrent data operations, computer failures, and network problems. Its implementation uses a fast consensus protocol with low communication overhead that has been optimally embedded into the overlay network.

We demonstrated the capabilities of Scalaris by re-implementing the core of Wikipedia. This implementation won first prize in the IEEE International Scalable Computing Challenge 2008. The implementation is both fast and scalable: using eight servers it executes 2,500 transactions per second. All operations are performed within transactions to guarantee data consistency and replication synchronization. Adding more computers improves the performance almost linearly. The public Wikipedia, in contrast, employs ten servers to execute the 2,000 requests that hit the backend of its large master/slave MySQL database (Wikipedia handles around 50,000 requests per second, with 48,000 handled by proxies).

For many Web 2.0 services, the total cost-of-ownership is dominated by the costs needed for personnel to maintain and optimize the service. Scalaris greatly reduces these costs with its built-in self management properties. It is self healing: when it detects a computing node crash or network problem, it immediately repairs its P2P overlay network and the database. Management tasks such as adding or removing nodes require no or very little human intervention. It is self tuning: it autonomously moves items to distribute the load evenly over the system to improve the response time. When Scalaris is deployed over multiple data centres, these algorithms are used to place frequently accessed items near the users.

External homepage: