How to scale down a Pinot cluster without downtime on Kubernetes?
I have been using Apache pinot lately for an analytics project at my company for a bit now. We deployed it on Kubernetes and on the test environment, the infra that was setup was too large for the traffic we had. So, I had to learn how to downsize a pinot cluster that has data on it, without affecting the application or losing the data. So, the way we do that is by rebalancing the servers on which the tables are hosted. Below are the steps I did manually in order to scale down the k8s cluster. Some context about my setup - Kubernetes cluster with 5 pinot servers. The pods were named as pinot-server-0 till pinot-server-4. - 1 table which had 10 segments distributed across the 5 servers. My target - Downsize the number of pinot-servers from 5 to 4. Steps: 1. First step is to make sure this setting is `pinot.set.instance.id.to.hostname=true` enabled on the cluster. 2. Since we are running on K8s, the cluster names are starting from pinot-server-0 to pinot-server-4. When we dow...