Kafka, Work Distribution, and Work Stealing
Hello, We use Kafka to distribute batches work across several boxes. These batches may take anywhere from 1 hour to 24 hours to complete. Currently, we have N partitions, each allocated to one of N...
View ArticleKafka, Work Distribution, and Work Stealing
David, It looks like the consumer throughput suffers because of imbalance of data across partitions. When you say the batch nears completion, it seems like the number of partitions that have new data...
View Article
More Pages to Explore .....
