Stateful Application as a Service

As an application developer, I want Confluent to provide me with a fully managed Kafka environment (KaaS) so I can point my current Kafka producer and consumer clients at KaaS and forget about the complexities of operating a complex stateful system.

It is safe to assume that the only tunable that KaaS customers have to convey to the KaaS provider (i.e., you) are data throughput, reliability, plus data retention and the rest will be taken care of by the system you are designing.

Please document your design of the KaaS system using the following format:

• Summarize the high-level goals of your design
• Describe the components of this design
• What role does each component play in the overall design?
• How do the components depend on each other?
• How does each component scale with an increase/decrease in produce/consume traffic?
• Describe the life of a Kafka producer and consumer request in this design
• Describe how to enforce security in this design
• Describe your data model needed to keep track of the multiple tenants in this KaaS product
• Describe the concerns/risks of operating this design and how you would mitigate them as much as possible in the production

The goal is to design a multi-tenanted Kafka setup. Given the multi-tenanted setup, (throughput, reliability, security, cost)