Cloud Corner: Distributed Systems / CAP / Kafka

Welcome to Cloud Corner, where we explore all things Cloud Computing, rain or shine! My name is Emma Genesen, and today, we’ll be establishing the foundations for future conversations. First, we’ll talk about distributed systems - what they are, and how they form the underpinning of modern cloud computing solutions. Then we’ll define the CAP theorem, and finally we’ll look at a popular technology in that space, Apache Kafka. 

Let’s begin with distributed systems. At its core, a distributed system is a collection of independent computers that work together as a single system. They can be located in the same physical location or spread out across the world. The key characteristic of a distributed system is that it allows for the sharing of resources and information between different computers. This is opposed to a network, where computers can send and receive messages, but do not coordinate and share resources. 

One of the main advantages of distributed systems is that they are highly scalable. This means that they can easily handle large amounts of data and a large number of users. They are also highly fault-tolerant, which means that if one component of the system fails, the rest of the system can continue to operate.

Some of the most popular distributed system technologies include Hadoop, an open-source software framework for storing and processing big data. Another popular distributed system is Apache Kafka, which is a distributed streaming platform that is used for handling real-time data feeds. But more on Kafka later.

Another important aspect of distributed systems is the concept of consistency and availability. Consistency referring to all nodes in the system having the same data at the same time, and availability referring to the ability of the system to respond to requests. But they also come with some challenges, like network latency. Because all these computers and processes are communicating via networks. And requests can take time. So what are some important considerations to keep in mind when designing a distributed system?

Scalability: Distributed systems must be able to handle large amounts of data and a large number of users. This requires careful planning and design to ensure that the system can scale horizontally as needed.

Fault tolerance: Distributed systems must be able to continue operating even if one or more components fail. This requires implementing mechanisms such as redundancy and replication to ensure the system can continue to function.

Network latency: Distributed systems rely on communication between different components, and network latency can greatly affect the performance of the system. It is important to design the system to minimize the impact of network latency and to handle it correctly.

Security: Distributed systems have a larger attack surface and more complex communication patterns, making them more vulnerable to security breaches. It is important to implement robust security measures to protect the system and the data it stores.

Monitoring and debugging: Distributed systems are complex and can be difficult to understand and debug. It is important to implement monitoring and logging mechanisms to gather data on the system's performance and behavior, and to make it easier to identify and fix problems.

Testability: Distributed systems can be difficult to test, especially when it comes to testing for failure scenarios. It is important to design the system in a way that makes it easy to test different parts of the system in isolation and to test for failure scenarios.

It's a lot to keep in mind! But this does bring us to the trade-offs between consistency, availability and partition tolerance. This brings us to the CAP theorem. We’ve already discussed consistency and availability, but what is Partition tolerance? It's the system continuing to function even when network partitions occur, meaning that communication between different parts of the system may be lost temporarily.

Thus, the CAP theorem states that it is impossible for a distributed system to provide all three guarantees simultaneously. Instead, a distributed system must make a trade-off between consistency, availability, and partition tolerance.

One example of a system that prioritizes consistency is a relational database, where writes may be delayed to ensure that all nodes have the same data at the same time. A system that prioritizes availability might be a NoSQL database like MongoDB, which allows for writes to continue during network partitions and prioritizes responding to requests over maintaining consistency.

A system that prioritizes partition tolerance is a distributed system that is able to continue operating even if communication between different parts of the system is lost temporarily. This is the case in some distributed systems that use consensus algorithms like Paxos and Raft. These algorithms enable the system to continue operating even if network partitions occur, but at the cost of availability and consistency.

So it's best to think long and hard about how you're going to implement you system and what parts of CAP are most important to your system.

For our last segment, we’ll be taking a look at Kafka, the distributed streaming platform for handling real-time data feeds. It is designed to handle large volumes of data and to handle data streams in real-time. It is often used for building real-time data pipelines and streaming applications.

The main goal in creating Kafka was to build a distributed and fault-tolerant system that could handle the high volume and real-time nature of the data streams generated by LinkedIn's website. The founders also wanted a platform that could handle both batch and real-time data, and that could be integrated with other big data systems such as the aforementioned Hadoop.

They also wanted something simple to operate, with minimal maintenance and administration. It was developed at LinkedIn in 2011 and has become widely adopted in the industry for a variety of use cases such as log aggregation, real-time data processing and stream analytics, event-driven architectures, and more.

Kafka is based on a publish-subscribe model, where producers write data to topics and consumers read from those topics. Each topic is a partitioned log of records, where each partition is an ordered, immutable sequence of records. This allows for high-throughput and low-latency processing of data streams.

One of the key features of Kafka is its ability to handle a large number of concurrent users and high throughput of data. It can handle millions of events per second and can handle petabytes of data.

Kafka has built-in fault tolerance, meaning that it can continue to operate even if one or more nodes fail. It achieves this by replicating data across multiple nodes, so that if one node fails, the data can still be accessed from another node.

Kafka also has strong durability guarantees, meaning that once data is written to a topic, it will be available for consumption even if the producing client or broker goes offline. When a message is written to a topic, it is written to the leader replica of the partition. The leader replica then replicates the message to all the follower replicas. In this way, data is replicated across multiple nodes, providing redundancy in case of a node failure.

In the event of a node failure, the leader replica will automatically detect the failure and choose a new replica to take over as leader. This process is called "leader election" and it happens automatically, with no interruption to the service.

Additionally, Kafka uses a technique called "quorum replication" to ensure that the data is safe and consistent even in the presence of network partitions. This means that a write to a topic is considered successful only if it is written to a certain number of replicas, called the "quorum," to ensure data consistency.

Kafka is powerful, but it also has some limitations and drawbacks.

Complexity: Kafka can be complex to set up and operate, especially for organizations that are new to distributed systems. It requires a good understanding of distributed systems and data streaming concepts to properly configure and maintain a Kafka cluster.

Scalability limitations: While Kafka is designed to handle large volumes of data, it can have scalability limitations, especially when it comes to handling a large number of small messages. This can lead to increased resource usage and higher costs.

Limited querying capabilities: Kafka is not a database, and it does not have built-in querying capabilities. This means that if you need to query or analyze the data stored in a Kafka cluster, you will need to use additional tools or technologies.

Limited data retention: By default, Kafka only retains data for a certain period of time, after which it is automatically deleted. This means that if you need to retain data for a longer period of time, you will need to set up additional storage and archiving solutions.

Limited security features: Kafka has some basic security features such as authentication and encryption, but it does not have advanced security features such as role-based access control. This means that organizations with strict security requirements may need to implement additional security measures.

Limited support for non-Java clients: Kafka was originally developed in Java, and while it does have support for other languages, the support for non-Java clients can be limited.

Limited support for transactions: While Kafka does support transactions to some extent, it is not designed to handle the same level of transactional semantics as a traditional database. This means that it may not be the best choice for applications that require strong transactional guarantees.

If you're looking for something a little more language agnostic, with even better performance characteristics, Apache Pulsar is an open-source pub-sub messaging and streaming platform that can more than compete with Kafka.

That’s it for today’s Cloud Corner. I hope you learned something, and I’ll see you next time!