Open Source Summit Europe 2024

ASML is building a self-service, multi-tenant data platform that enables teams to work independently on data products and applications. In a secure multi-tenant Kubernetes setup, tenants get an isolated view of the cluster, allowing them to deploy workloads independently of each other. Data sharing among tenants is governed by a global policy-based access control layer. Our challenge was to introduce a shared Kafka cluster into this setup, with proper tenant isolation and seamless integration with the global access control layer. We used Istio to enable zero-configuration authentication for Kafka clients operating inside the Kubernetes cluster. This allows Kafka authentication to be done with the Kubernetes-native tenant/namespace/serviceaccount idiom. A custom operator reconciles the tenant's topic definitions and topic access requests with the platform administrator's resource allocations and the global data sharing policy, and dynamically configures the right Kafka ACLs and resource quota.