constructing a fresh execution platform for Redis users.

Redis has developed into a well-liked and extensively used in-memory cache during the past ten years. It features a compelling collection of integrated data structures, unlike the majority of distributed caches, that expand its functionality beyond just storing serialised items.

 

Redis, a single-server store that was developed in 2009 to aid in the development of a web server, provides applications with a wide range of practical options for classifying and retrieving stored data, including sets, lists, hashes, and sorted sets. It also added support for clusters to extend throughput and enable data replication for high availability a few years later. System administrators must comprehend and take care of unique concepts created by Redis’ cluster implementation, like hashslots and master/replica shards. These ideas have produced complexity that makes cluster administration more difficult while lowering flexibility and reliability. Redis uses eventual rather than full consistency.

 

On the other hand, ScaleOut StateServer®, a distributed cache for serialised objects with an enterprise focus that was first released in 2005, was created from the ground up to run on a server cluster with the specific design goals of high performance and reliable data storage while maximising ease of use. It automates load balancing, data replication, and recovery while storing data across copies with complete consistency (sequential consistency). For optimal throughput, it also puts all available CPU cores to use when processing client requests. Particularly for enterprise users, these features significantly reduce TCO, increase flexibility, and simplify cluster management. Contrary to Redis, ScaleOut server clusters, for instance, can easily expand from a single server to many servers, and system administrators are not required to maintain hashslots or primary/replica shards.

 

Integrating Redis Command Processing into a New Clustering Architecture Is the Challenge

Redis commands running on a ScaleOut StateServer cluster would provide Redis users with the best of both worlds: well-known and rich data structures paired with noticeably easier cluster management and complete data consistency. This was realised by ScaleOut Software. To ensure that client commands behave exactly like those used by open-source Redis clusters, the ideal implementation would need to run Redis commands using actual open-source code. The next problem was to integrate Redis code into ScaleOut StateServer’s execution platform and utilise its highly automated clustering features while getting rid of Redis’s event-loop architecture’s single-threaded limitations.

 

The company recently completed this integration, and in ScaleOut StateServer version 5.11, they made a community preview available. This preview adds support for the publish/subscribe commands, transactions, and various utility commands (like FLUSHDB and EXPIRE) on both Windows and Linux, as well as the most popular Redis data structures (strings, sets, lists, hashes, and sorted sets). The servers handle Redis commands using open-source Redis version 6.2.5 code.

 

Scalable throughput, automated cluster operations, and complete data consistency

 

Through this integration, Redis clients can make connections via the Redis RESP protocol to a ScaleOut StateServer cluster of one or more servers. Redis access requests are then routed to the proper ScaleOut server via internal calls to the CLUSTER SLOTS or CLUSTER NODES commands in Redis client libraries, which subsequently get the mapping of hashslots to servers. Redis clients automatically use the RESP protocol to route commands to the appropriate servers when the cluster membership changes.

 

Each ScaleOut server handles incoming Redis commands over multiple threads using all available CPU cores to maximise throughput. To scale performance or enable data replication, only one process is required to be deployed on each server; additional shards are not required.

 

As the demand for more throughput arises, system administrators can simply join new servers to the cluster. As servers are added or removed from the cluster, ScaleOut StateServer automatically rebalances the hashslots across the cluster. Redis commands are additionally delayed during load balancing (and recovery) in order to preserve a consistent view of hashslot placement and prevent client exceptions. Client requests return a -MOVED indicator so that they can be forwarded to the new server after a hashslot has fully migrated to a remote server.

 

Additionally, ScaleOut StateServer generates copies for each and every hashslot. During membership changes and recovery, primary and replica shards don’t need to be manually created or moved from one server to another. For the best load-balancing, ScaleOut StateServer automatically migrates replica hashslots after membership changes and places them on other servers from their associated primary hashslots. ScaleOut StateServer “self-heals” by elevating replicas to primaries and generating more replicas as needed in the event of a server failure or network outage.

 

For updating replicas, Redis employs an eventual consistency mechanism. This strategy is quick, but if a server goes down or there is a network outage, it could supply clients with outdated information. ScaleOut StateServer uses a proprietary quorum approach to implement fully consistent changes to stored objects in order to get around this issue. Prior to returning to the client after receiving a Redis command, the server finishes the command’s execution on a quorum made up of the primary hashslot and its replicas. The same method is used to process all transactions.

 

Conclusion: New Features for Redis Users

Redis customers will benefit from significant new capabilities thanks to ScaleOut Software’s creative integration of Redis open-source code into its cluster architecture, which also guarantees native Redis behaviour for client applications. With automatic cluster administration, built-in throughput scaling, and fully consistent changes to stored data executed by Redis commands, enterprise customers now have a simpler method to manage Redis clusters. Additionally, they can obtain access to ScaleOut StateServer’s native APIs, which include sophisticated functionality not found in open-source Redis clusters such as coherent, wide-area data replication, streaming analytics, and data-parallel computation.