How to scale down a Pinot cluster without downtime on Kubernetes?

-

 



I have been using Apache pinot lately for an analytics project at my company for a bit now. We deployed it on Kubernetes and on the test environment, the infra that was setup was too large for the traffic we had. So, I had to learn how to downsize a pinot cluster that has data on it, without affecting the application or losing the data. So, the way we do that is by rebalancing the servers on which the tables are hosted.

Below are the steps I did manually in order to scale down the k8s cluster. 

Some context about my setup

- Kubernetes cluster with 5 pinot servers. The pods were named as pinot-server-0 till pinot-server-4.
- 1 table which had 10 segments distributed across the 5 servers.

My target

- Downsize the number of pinot-servers from 5 to 4.

Steps:

1. First step is to make sure this setting is `pinot.set.instance.id.to.hostname=true` enabled on the cluster.

2. Since we are running on K8s, the cluster names are starting from pinot-server-0 to pinot-server-4. When we downscale the service on k8s, only pinot-server-0 to pinot-server-3 will remain. So, When deciding to downscale, we must choose to detach the server-4 first. So start from n, then n-1, n-2 and so on when deciding to downsize the k8s cluster.

3. Go to pinot server ui and select the pinot-server-4 and click "Edit Tags". Remove all the tags.

4. Now, since I have only 1 table, I go to the table config UI and click "Rebalance Servers" which then shows the status of "IN_PROGRESS" . You will also notice that it now excludes pinot-server-4 from the segment assignment. Now just wait for this to complete and monitor the controller logs if need be. There are a few configs you should input as per your need - they are mentioned here in a table - https://docs.pinot.apache.org/operators/operating-pinot/rebalance/rebalance-servers#running-a-rebalance . For my case, since the table had only replication=1, I had to set the value in rebalance for minAvailableReplicas as 0 for rebalancing to proceed.
5. Wait for sometime since it can take a while for rebalance to complete depending on the size of the table. If rebalance is complete, on the table's config ui page, you should not see pinot-server-4 listed under the instances.

6. In your case if there are more tables, all of them need to be rebalanced so that all the segments on this server are moved to other servers. So make sure to do that, other wise you will not be able to drop the server.

7. Next is to kill the pod in k8s for which you should scale down the pinot-server service. For instance, this is the command I used.
kubectl get statefulsets
kubectl scale statefulsets pinot-server --replicas=4

8. Once the pod pinot-server-4 is killed, go back to pinot server UI page and then disable and drop the server. It should already be in dead state since the pod was killed.

Those were the steps for scaling down the pinot-servers without any loss of data. Let me know if you have further questions.





Comments

Post a Comment

Popular posts from this blog

Exploring the Trie Data Structure: Applications and Efficiency

-
Image
 Introduction The Trie data structure, also known as a prefix tree, is a versatile and efficient tree-based data structure used primarily for storing and searching strings. In this blog post, we will delve into the Trie data structure, its applications in various domains, and its efficiency compared to other data structures. What is a Trie? A Trie is a tree-based data structure that allows for efficient insertion, deletion, and retrieval of strings. It is particularly suited for applications involving large sets of strings, such as word dictionaries, autocomplete systems, and spell checkers. Unlike other tree-based structures, Tries provide fast access to strings with a common prefix. Structure and Operations of a Trie - Trie Node: Each node in a Trie represents a character and contains references to its child nodes, forming a hierarchical structure. - Insertion: When inserting a string into a Trie, each character is traversed, and new nodes are created if necessary. - Search: To s...
Read more

Demystifying Class Loading and Dynamic Linking in Java

-
 Introduction As a Java app developer, understanding the concepts of class loading and dynamic linking is essential for mastering the intricacies of the Java programming language. In this blog post, we will delve into the details of class loading and dynamic linking, exploring how these mechanisms enable the execution and integration of Java code. By the end of this article, you will have a clear understanding of these concepts and their significance in Java development. Class Loading Class loading is the process by which the Java Virtual Machine (JVM) loads classes into memory at runtime. Let's explore the key aspects of class loading: - Class Loading Phases : Class loading consists of three primary phases: loading, linking, and initialization.   - Loading : The class loader locates the bytecode for a class and reads it into memory. It searches for the class file in the classpath, which includes directories, JAR files, and other locations specified during runtime.   - Li...
Read more

Inter-Thread Communication Part 2: Communication Patterns

-
 Introduction In concurrent programming, effective inter-thread communication is crucial for coordinating the execution and data exchange between multiple threads. Java provides various communication patterns and mechanisms that enable threads to synchronize their actions, share data, and ensure proper coordination. In this second part of our series on inter-thread communication, we will explore different communication patterns in Java. By understanding these patterns, you will be able to design robust and collaborative multithreaded applications. Let's dive into the world of communication patterns for inter-thread coordination. Wait and Notify The wait() and notify() methods, available in the Object class, provide a classic inter-thread communication pattern in Java. Threads use these methods to wait for specific conditions to be met before proceeding with their execution or notifying other threads about a change in state. Example class SharedResource {     private boole...
Read more