Hands On: Create Kafka Topics with TopicBuilder
Viktor Gamov
Developer Advocate (Presenter)
Hands On: Create Kafka Topics with TopicBuilder
Note: This exercise is part of a larger course. You are expected to have completed the previous exercises.
This exercise shows you how to programmatically create topics on Confluent Cloud. You can see the code for modules 1–10 in a combined GitHub repo and you can also refer there for a list of imports as well as a sample build.gradle file.
Add a Topic using the Confluent Cloud Console
-
To manually create a topic on Confluent Cloud, go to Topics on the left-hand menu, then Add Topic. You can use the default settings, or you can customize your settings, specifying the number of partitions, the cleanup policy, the retention time, the maximum message size, etc.
Create a Topic Programmatically
-
In order to tell Spring that this particular topic should be created, you need to create a bean with the type of NewTopic and with the number of partitions and replicas specified in your application class; then use TopicBuilder to establish the topic:
@Bean NewTopic hobbit2() { return TopicBuilder.name("hobbit2").partitions(12).replicas(3).build(); }Spring uses the topic configuration and automatically instantiates an AdminClient to execute the required operations.
-
On Confluent Cloud, you should see your new hobbit2 topic.
-
Now change the number of partitions in the topic from 12 to 15:
@Bean NewTopic hobbit2() { return TopicBuilder.name("hobbit2").partitions(15).replicas(3).build(); }You should see the change in Confluent Cloud. Note that because the topic already existed, the partitions were simply updated (had it not existed, a new topic would have been created).
Use the promo code SPRING101 to get $25 of free Confluent Cloud usage
Be the first to get updates and new content
We will only share developer content and updates, including notifications when new content is added. We will never send you sales emails. 🙂 By subscribing, you understand we will process your personal information in accordance with our Privacy Statement.
Hands On: Create Kafka Topics with TopicBuilder
Hi, this is Victor Gamov with Confluent and in this lesson, I will show you how we can use TopicBuilder to programmatically create topics. Let's get to it. So when I go here in my Confluent Cloud cluster, I can click Add topic and after that I can create the topic with some default configuration, topic1 or I can customize those settings. I can go change the number of partitions, I can change a cleanup policy, I can set a retention, by default as you can see, it use infinite retention so you can store your data inside this Confluent Cloud as long as you want. Also, you can specify the maximum size of the message, so those things that you can change in your UI. So, let's take a look how Spring Boot and the Spring Kafka can help us to create these topics programmatically. So in order to do so, we need to do couple of things here. In order to tell the Spring that this particular topic or this something needs to be created, we need to create a bean with a specific type and in this case, Spring will understand this, it will use this topic configuration and automatically instantiate AdminClient and execute required operation. So let's get to it. Let's create a new bean with type of NewTopic and we will use a TopicBuilder class in order to instantiate this topic. NewTopic hobbit2, we'll make it as a Bean. We just do return something, TopicBuilder with name hobbit2, we can specify a number of partitions, in this case partition count will be, let's do 12. Next thing, we'll also specify number of replicas. It's 3, because we're writing this in a cloud, this is the minimum required. And after that when I click build, I will be able to get a new topic that I will be able to return. And here I can also inline this, so it will not be used. Now, so what we will expecting to see here when I run this application, I will expecting this topic will be available in my Confluent Cloud UI. Go into Topics, and now I have a topic hobbit2 with configuration of 12 partition, this is what just we did. Let's try to change configuration. So if this topic was created before we'll be able to change our configuration, say we want to change number of partitions from 12 to 15. As you can see my configuration just changed. And because the topic was already existed in my cluster, Admin API will try to alter config because the previous version of this topic included 12 partitions, and this topic includes now 15 partitions. That's what how does it work. Once again, you need to use this Bean, and in this case, Spring Boot will be able to identify, will be able to identify that this topic needs to be created. It will initiate AdminClient and this AdminClient will execute methods create or create or replace for new topics. And now you know how we can programmize a good create topic or change the config of existing topic using Spring Boot and Spring Kafka.