How to use Shed Lock with a Kafka - based system?

Oct 29, 2025

Leave a message

Amanda Li
Amanda Li
As a sustainability analyst, I focus on implementing eco-friendly practices in our production processes. At Lianhu, we’re committed to minimizing our environmental footprint.

Hey there! I'm a supplier of Shed Locks, and today I want to chat about how you can use Shed Locks in a Kafka - based system. It might sound a bit technical, but don't worry, I'll break it down for you in a simple way.

First off, let's quickly go over what Kafka and Shed Locks are. Kafka is a distributed streaming platform that's super popular for handling real - time data feeds. It allows you to publish, subscribe, store, and process streams of records. On the other hand, Shed Lock is a great tool when it comes to ensuring that only one instance of a task runs at a time in a distributed system.

So, why would you want to use a Shed Lock in a Kafka - based system? Well, in a Kafka setup, you often have multiple consumers processing messages from topics. Sometimes, you might have tasks that shouldn't run concurrently. For example, you could have a task that updates a shared resource based on the Kafka messages. If multiple instances of this task run at the same time, it could lead to data inconsistencies or other issues. That's where Shed Lock comes in handy.

Integrating Shed Lock with Kafka

The first step in using Shed Lock with a Kafka - based system is to add the necessary dependencies. If you're using a Java project, you can add the Shed Lock dependency to your pom.xml if you're using Maven.

<dependency>
    <groupId>net.javacrumbs.shedlock</groupId>
    <artifactId>shedlock-spring</artifactId>
    <version>4.44.0</version>
</dependency>

Once you've added the dependency, you need to configure the lock provider. There are different types of lock providers available, like database - based or Redis - based. For simplicity, let's assume we're using a database - based lock provider. You'll need to set up a table in your database to store the lock information.

CREATE TABLE shedlock(
    name VARCHAR(64),
    lock_until TIMESTAMP(3) NULL,
    locked_at TIMESTAMP(3) NULL,
    locked_by VARCHAR(255),
    PRIMARY KEY (name)
);

In your Spring Boot application, you can configure the lock provider like this:

import net.javacrumbs.shedlock.core.LockProvider;
import net.javacrumbs.shedlock.provider.jdbctemplate.JdbcTemplateLockProvider;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.jdbc.core.JdbcTemplate;

import javax.sql.DataSource;

@Configuration
public class ShedLockConfig {

    @Bean
    public LockProvider lockProvider(DataSource dataSource) {
        return new JdbcTemplateLockProvider(
                JdbcTemplateLockProvider.Configuration.builder()
                        .withJdbcTemplate(new JdbcTemplate(dataSource))
                        .usingDbTime() 
                        .build()
        );
    }
}

Using Shed Lock in Kafka Consumers

Now that we have the Shed Lock set up, let's see how we can use it in our Kafka consumers. Suppose you have a Kafka consumer that processes messages from a topic and updates a shared resource. You can use the @SchedulerLock annotation provided by Shed Lock to ensure that only one instance of the task runs at a time.

import net.javacrumbs.shedlock.spring.annotation.SchedulerLock;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Service;

@Service
public class KafkaConsumerService {

    @KafkaListener(topics = "your - topic", groupId = "your - group - id")
    @SchedulerLock(name = "kafkaConsumerTask", lockAtMostFor = "PT10M", lockAtLeastFor = "PT1M")
    public void consumeMessage(String message) {
        // Process the message and update the shared resource
        System.out.println("Received message: " + message);
    }
}

In the above code, the @SchedulerLock annotation ensures that the consumeMessage method is locked for at least 1 minute and at most 10 minutes. The name attribute is used to identify the lock.

Considerations and Best Practices

When using Shed Lock with a Kafka - based system, there are a few things to keep in mind.

21-62-2

  • Lock Duration: You need to carefully choose the lockAtMostFor and lockAtLeastFor values. If the lockAtMostFor value is too short, the lock might be released before the task is completed, leading to multiple instances running at the same time. If it's too long, it could cause unnecessary delays.
  • Error Handling: You should have proper error handling in place. If an error occurs during the execution of the locked task, the lock should be released gracefully. Otherwise, it could lead to a deadlock situation.
  • Scalability: As your Kafka system scales, make sure that the lock provider can handle the increased load. For example, if you're using a database - based lock provider, ensure that your database can handle the concurrent lock requests.

Other Types of Locks for Different Use Cases

While Shed Lock is great for ensuring single - instance task execution, there are other types of locks that might be useful in a Kafka - based system. For example, if you're dealing with physical locks on doors or enclosures in a data center where your Kafka infrastructure is located, you might consider Quarter Turn Lock or Round Door Lock. These locks can provide an extra layer of security for your equipment.

Conclusion

Using Shed Lock with a Kafka - based system can help you avoid data inconsistencies and other issues that can arise from concurrent task execution. By following the steps outlined above, you can easily integrate Shed Lock into your Kafka application. If you're interested in purchasing our high - quality Shed Locks or have any questions about how they can fit into your Kafka system, feel free to reach out for a procurement discussion. We're here to help you find the best solution for your needs.

References

  • Spring Boot Documentation
  • Kafka Documentation
  • Shed Lock GitHub Repository
Send Inquiry