ScyllaDB — Perfect choice for your enterprise

If you were to ask me the single most important architectural decision for the success of any application, I would say it is the choice of the database. It defines your ability to accumulate, process and analyze data. Any compromise or limitations here enforces compromises and limitations in what you can do with your data — which in turn compromises and limits the possibilities, performance as well as profitability of your application.

Hence, it is very important to spend that additional time evaluating various possibilities available to us. We must understand them in detail, and then identify the best possible solution for you.

Introduction to ScyllaDB

If you are a techie exposed to social media, I am sure you already know that ScyllaDB is a modern, NoSQL Database. It is designed for extreme scalability and performance. It is based on some strong design paradigms that make it a strong contender for the primary database in your system.

ScyllaDB started as a reimplementation of Apache Cassandra — the columnar database. If you think Cassandra is old, we have another Avatar of the ScyllaDB that is compatible with the DynamoDB.

I can hear an obvious question in your mind — why do we need ScyllaDB when we already have the Apache Cassandra? What is the big deal here? That is an important point. ScyllaDB is not just a copy of the Cassandra. It is an independent implementation that provides a syntactical identity. Internally, its architecture is way better and stronger. It is far more resilient and scalable. At the same time, you do not have to rewrite the code that accessed Cassandra.

Now that is something interesting. But the next question is, what makes it better than the predecessors, or any other database in this range? When and why should I use ScyllaDB instead of the popular ones?

Let’s understand how.

Design principles behind ScyllaDB

You cannot get scalable and high-performance applications like ScyllaDB without strong design principles. The architects of ScyllaDB made sure they have rock solid principles that go into every aspect of this database. These principles are the key to its strength. Let us understand them one by one.

Lessons from Cassandra

Before we dive into the design principles, we should understand what was missing in Cassandra. Apache Cassandra was a great product no doubt. That is exactly what ScyllaDB. However, there were some serious drawbacks that forced ScyllaDB to rewrite the application, just retaining the API compatibility.

Positive aspects

It is not correct to criticize it without first highlighting the positive aspects. So let us do that first. These points made Cassandra a robust monster that we all wanted:

1. Masterless Replication: Cassandra does not have any master or slave node. All nodes are symmetric and capable of read/write. Hence, there is no single point of failure.
2. Global Distribution: There is no restriction on the proximity of the cluster nodes. They can be spread across datacenters across the globe.
3. Linear Scale: This is very important for scaling the database. The increase is linear, not exponential as in most cases.
4. Tunable Consistency: Consistency and Performance tradeoff is a part of the domain and design. Cassandra allows us to tune it as per our requirements, instead of imposing it on us.
5. Simple Data Model: Cassandra provides a simple data model that is easy to work with — thus reducing the complexity of the application.

Improvement Areas

Despite these wonderful aspects, Cassandra did not pick up as much as it could have. Below are the limitations that held it back.

1. Operating Complexity: Managing Cassandra was not as easy. It required a dedicated team of experts, just to keep it running.
2. JVM Challenges: Memory management of JVM was good for petty applications. However, it was miserable when working with applications of such a huge scale.
3. Inefficient CPU Utilization: Because of the limitations of JVM, Cassandra could not make the best of the high-end multicore CPUs.
4. Manual Tuning: The clusters are not smart enough to tune themselves. They require explicit, continuous, and elaborate tuning. Moreover, this requires detailed knowledge of database internals. It is not a black box tuning.

As a result of these complexities, many architects were forced to forego the features of Cassandra and choose other options.

ScyllaDB was built to address these issues ground up. It is based on design decisions that make sure we do not have these problems.

Decision 1: C++ instead of Java

Java in enterprise scale applications is the greatest blunder one can make. (In my opinion, Java itself was a blunder).

Any sensible architect would not need platform independence or all those features they boast about. Automated memory management of Java is a myth, useful only in petty desktop applications. Java is a severe cost to performance. We have all seen those ruthless and frustrating “stop-the-world” GC pauses in Java applications. Furthermore, platform independence means that the programmer loses low level control. We cannot exploit the advantages of multi core architectures and that means miserable performance despite costly hardware. Why %#@?

ScyllaDB architects made the sensible decision of switching to C++. That means better control, performance and hence scalability.

Decision 2: Cassandra Compatibility

Despite some of the problems in Cassandra, it was quite popular in its time. A lot of open-source effort was invested in applications around it. There was no point wasting this effort. Hence, the architects ensured perfect compatibility with Cassandra in every sense.

Similarly, we now have another version of ScyllaDB that is compatible with DynamoDB. That is a big relief to folks who are stuck with AWS because of DynamoDB.

We can just plug and play the same code with ScyllaDB instead of Cassandra or DynamoDB.

Decision 3: Everything Async

Any amount of investment in high end CPUs cannot solve the problem of the active thread waiting for an I/O operation. If we want to scale linearly, we must make sure that we do not have any such delays in our architecture. The only way to achieve this is to make it asynchronous.

Architects of ScyllaDB understood this well, and ensured everything is asynchronous. This makes sure we never have a scenario of one resource waiting for another. To enable this, ScyllaDB relies on a dedicated asynchronous engine called Siestar.

Decision 4: Shard per Core

Sharing implies waiting. When two threads share a resource, we will have situations where one thread waits for another. And of course, it adds to other complexities like lock and deadlock prevention. To simplify and improve performance, ScyllaDB makes sure we have just one shard dedicated to each core. That ensures we have nothing waiting for something else. That is the way we go with Shared Nothing Architecture.

Decision 5: Unified Cache

The idea behind caching is great and seems absolutely straightforward. However, anyone with reasonable experience with life will agree that it is perhaps the most complicated part of the story. It is the single component of a database architecture that can enhance or ruin its performance.

Here is an important point that we should understand. Linux is a general-purpose operating system. Its architecture is optimized for optimal performance across a wide range of applications. It is not optimized for databases. The file system and caching are part of that. Linux treats files as chunks of 4KB. And caching is also optimized around this 4KB. This is great for large chunks of data.

However, most DB reads result in much smaller responses. This results in significant wastage of disk IO and cache memory. It causes a lot of read amplification and cache misses — results in blocking the thread on I/O, and we all know the rest of the story.

To overcome this problem, ScyllaDB implements its own row-level cache. This bypasses the Linux cache, hence does not suffer from the above problems. Another feature of this customization is that a cache miss does not result in thread blocking. Instead, it triggers another thread for the processes the IO asynchronously.

Seastar makes sure there is no blocking, heavy weight context switching or any kind of wastage of the precious CPU cycles.

Decision 6: Scheduler

Data is committed to the disk is volatile. Hence, we all want to do it ASAP. However, that means costly disk writes, which result in delays. It is important to identify an optimal strategy for scheduling such disk writes, so that the system is stable as well as fast. This is further complicated by the fact that the optimal point varies based on the configuration of the node hosting the database. We cannot leave it to humans (as Cassandra did), to identify and configure the optimal settings for each node in all the clusters across the globe.

ScyllaDB uses an optimal strategy for scheduling such disk writes in batches — at the same time, auto detecting a perfect balance based on the configuration of the current node. As soon as we spin a new node, it runs a utility called scylla_io_setup, that runs several diagnostic tests. In doing so, it identifies the ideal disk concurrency and other parameters that help it configure the write scheduler.

With this, ScyllaDB can maximize performance by giving high priority to user requests and schedules disk writes when it is able to. This gives us best of both the worlds — low latency as well as stability.

To manage the data on disk, ScyllaDB uses an algorithm called LSM trees (Log structure Managed trees). In simple words, the data is written to immutable files, appending writes to leaves of the tree, instead of updating the data continuously. This ensures that the writes are superfast. Naturally, it has a price on reads. However, that is simplified by another batch process that continuously optimizes the tree.

The core idea is that an element at the end of the tree is obviously available in the cache, hence has a low probability of generating a disk read.

Decision 7: Autonomous Capabilities

Perhaps the strongest selling point of ScyllaDB is its ability to manage itself. Even in the age of cloud, SAAS, and DAAS, we have scenarios where enterprises cannot afford the risk of having a database away from their campus. However, an enterprise scale database on campus means a significant cost on administrative tasks.

ScyllaDB, or any meaningful database, has a dozen of components, each coming with a dozen parameters for tuning the performance. There is no default, textbook answer to identifying the optimal values for these parameters. They vary with the conditions of the node configuration, disk size and latency, and the network conditions. To complicate things, these factors can vary with time and depend on the system load. It takes serious expertise to manage them efficiently. Humans are prone to mistakes and when we talk about enterprises catering to global customers, any such mistake can be very costly.

That is where we need autonomous tuning. When we work with ScyllaDB, the complicated tuning parameters are identified and configured by the database node itself — and not left at the mercy of an administrator.

Based on these rigorous design principles, ScyllaDB is battle hardened in production. It is coming up as one of the wonderful databases available for your next application. Of course, none of these decisions was possible without the first decision of ditching the JVM!

Core Concepts

Let us now look at some of the core concepts required for using ScyllaDB in your application.

Nodes & Clusters

The building blocks of a cluster is the node, which is one instance of the database running on an independent server, or in a logical unit like a docker container. A node consists of several shards. As we saw above, the number of shards depends on the number of cores available in the hardware.

Unlike the traditional databases, ScyllaDB does not provide master and slave nodes. They are all equal. That means no node is sacrosanct. Any node can be discarded without affecting the cluster.

ScyllaDB Cluster has a ring architecture. The individual nodes are connected in a ring, so that we do not have chaos with all nodes talking to each other. This simplifies delegation. Such clusters result in a distributed database with high availability, high performance, low maintenance, and high scalability.

Obviously, we cannot have the entire data replicated on each Node. That would be highly wasteful. Nor can we afford to keep just one copy of any given data. We must choose an appropriate alternative amongst the provided replication strategies. Beyond that, ScyllaDB takes care of distributing and replicating the data on the cluster according to the chosen strategy. For example, we can choose the replication factor that defines the number of copies of any given row or partition.

ScyllaDB is not limited to one single data center. It can be spread across multiple data centers across the globe.

Consistency Level

When we work with distributed databases, we have to remember the CAP theorem. We have to live with this tradeoff between the Consistency, Availability and Performance. But the advantage of ScyllaDB is that it allows us to choose the balance between them. Let us check how we can choose the consistency:

We can choose the default Consistency Level (CL), and also for each individual transaction. This determines how many replicas in a cluster must acknowledge a read or write operation before it is considered successful.

Based on the need, we can choose one of the below values for CL:

• ANY: A read/write must get a response from at least one node in the cluster. Naturally, this provides the highest availability at the risk of low consistency.
• ONE: If one replica responds, the request is honored. Note that this is different from ANY — where the request goes to multiple nodes in the first place.
• QUORUM: When the majority of the nodes respond, the request is honored. If we have set RF = 5, then the request waits for 3 of them to respond positively.
• LOCAL_ONE: This is similar to ONE, except that the request is restricted to the nodes in the local data center.
• LOCAL_QUORUM: This is similar to QUORUM, except that the request is restricted to nodes in the local data center.
• EACH_QUORUM: This is a step beyond QUORUM. The write request is honored only if we can achieve a local quorum in each of the data centers. This makes more sense for Write operations and is not supported for reads.
• ALL: Finally, we can ask for absolute consistency, where all the request is honored if and only if all the concerned nodes in the cluster respond correctly. This is not just a textbook scenario as it may seem to be. It is necessary for certain mission-critical applications.

Note that the consistency level in write operations is a measure of how many nodes should respond. However, the write request is always sent to all the concerned nodes.

Playing with the DB

All this blabber is meaningless until we get our hands on the code. Enough of theory. Now, let us try to build something using the ScyllaDB. Of course, we can install the database on a server; or else, pull it from the docker repository.

ScyllaDB in Cloud

However, all that is not necessary. In the age of cloud based SAAS, we can simply provision what we need in the cloud. Visit this link and create the account. Once you provide the information required to sign-up, you will land on this screen.

Select Free Trial for now, and then provide the required details.

After this, we are asked to choose the compatibility of the ScyllaDB instance. We can choose between Cassandra and DynamoDB. Cassandra is chosen by default. For this exercise, let us change it to DynamoDB

Change this to DynamoDB.

Next, it asks us for more details about the isolation policy. For now, just leave it at its default value

Since we chose AWS as the preferred cloud, we go ahead to specify the required EC2 instance type on AWS.

Again, leave it the default value. For learning, we should be good with the smallest instance.

It is a good practice to protect the database with restrictions on the source IP address. This will ensure better security.

Provide all the required information and then submit to create the DB cluster.

This will take some time and finally, we will have the ScyllaDB cluster ready to use.

Connecting to the DB

Once the cluster is ready, we will get node end points that look like this:

"node-0.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud",
"node-1.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud", 
"node-2.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud"

Running DB Commands

Since we created a DynamoDB compatible instance we can run the below command to create a new table in this ScyllaDB cluster

aws --region=us-east-1 \
    --endpoint-url 'http://node-0.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud:8000' \
    dynamodb create-table \
    --table-name MusicCollection \
    --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S \
    --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE \
    --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Python Code

Let us now write some Python code to work with this database. Again, since we are working with a DynamoDB compatible instance, we can use the AWS Python SDK (boto3) to connect with it. So let us first install the boto3 (assuming you already have Python installed)

sudo pip install --upgrade boto3

With the boto3 in place, we can run the same create table command using the below script.

import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://node-0.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud:8000',
                region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

dynamodb.create_table(
    AttributeDefinitions=[
    {
        'AttributeName': 'key',
        'AttributeType': 'S'
    },
],
BillingMode='PAY_PER_REQUEST',
TableName='usertable',
KeySchema=[
    {
        'AttributeName': 'key',
        'KeyType': 'HASH'
    },
])

Similarly, we can use the DynamoDB BatchWrite and Put commands to add data to this table

import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://node-0.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud:8000',
                    region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

    dynamodb.batch_write_item(RequestItems={
        'usertable': [
            {
                'PutRequest': {
                    'Item': {
                        'key': 'test', 'x' : {'hello': 'world'}
                    }
                },
        }
    ]
})

We can verify that the write was successful by reading back the same record.

import boto3
dynamodb = boto3.resource('dynamodb',endpoint_url='http://node-0.aws-us-east-1.1b7b127ddb9c42403839.clusters.scylla.cloud:8000',
                    region_name='None', aws_access_key_id='None', aws_secret_access_key='None')

    print(dynamodb.batch_get_item(RequestItems={
    'usertable' : { 'Keys': [{ 'key': 'test' }] }
}))

ScyllaDB on Premise

Now let us check out how we can run the ScyllaDB on-premise. This time, let us look at the CQL compatible version

Before starting the cluster, make sure the aio-max-nr value is high enough (1048576 or more). This parameter determines the maximum number of allowable Asynchronous non-blocking I/O (AIO) concurrent requests by the Linux Kernel, and it helps ScyllaDB perform in a heavy I/O workload environment. Check the value as below:

cat /proc/sys/fs/aio-max-nr

If this is low, and it needs to be changed, we can run the below commands:

echo "fs.aio-max-nr = 1048576" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf

Now, let us start with the ScyllaDB cluster. First, we’ll start a single instance and call it ScyllaDBU:

docker run --name scyllaU -d scylladb/scylla:5.2.0 --overprovisioned 1 --smp 1

Again, I assume that you already have the docker CLI in place.

After waiting for a few seconds, we’ll verify that the cluster is up and running with the Nodetool Status command:

docker exec -it scyllaU nodetool status

The node scyllaU has a UN status. “U” means up, and N means normal. Read more about Nodetool Status in the documentation.

Finally, we use the CQL Shell to interact with ScyllaDB:

docker exec -it scyllaU cqlsh

Now that we have a single ScyllaDB instance running, let us try to run some basic CQL commands. We will work with our newly created ScyllaDB instance to create a table, insert data into it, and read the data.

• Create a Keyspace

CREATE KEYSPACE mykeyspace WITH REPLICATION = { 'class' : 'NetworkTopologyStrategy', 'replication_factor' : 1};

• Next, create a table with three columns: user id, first name, and last name, and insert some data:

use mykeyspace; 
CREATE TABLE users ( user_id int, fname text, lname text, PRIMARY KEY((user_id)));

• Insert into the newly created table two rows:

insert into users(user_id, fname, lname) values (1, 'rick', 'sanchez'); 
insert into users(user_id, fname, lname) values (4, 'rust', 'cohle');

• Read the table contents:

select * from users;

Useful Links

This was a simple introduction. If you like to learn more, you can always go to their site for a detailed understanding of the architecture. They also have a tutorial site.