Introduction to Apache Pulsar and Plugin Development

Apache Pulsar is a highly scalable, distributed pub-sub messaging system that has gained significant traction in recent years due to its performance, reliability, and flexibility. One of the key features of Pulsar is its extensibility through plugins, which allows developers to customize and enhance its functionality. In this article, we will delve into the world of developing Apache Pulsar plugins using Java, providing you with a comprehensive guide, complete with code examples and diagrams.

Setting Up Your Environment

Before diving into plugin development, you need to set up your environment with the necessary tools and libraries.

Installing Java Client Library

To work with Pulsar, you need to include the Pulsar Java client library in your project. Here’s how you can do it using Maven or Gradle:

Maven

Add the following dependencies to your pom.xml file:

<properties>
    <pulsar.version>3.3.1</pulsar.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.pulsar</groupId>
        <artifactId>pulsar-client</artifactId>
        <version>${pulsar.version}</version>
    </dependency>
    <dependency>
        <groupId>org.apache.pulsar</groupId>
        <artifactId>pulsar-client-admin</artifactId>
        <version>${pulsar.version}</version>
    </dependency>
</dependencies>

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.apache.pulsar</groupId>
            <artifactId>pulsar-bom</artifactId>
            <version>${pulsar.version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Gradle

Add the following to your build.gradle file:

def pulsarVersion = '3.3.1'

dependencies {
    implementation enforcedPlatform("org.apache.pulsar:pulsar-bom:${pulsarVersion}")
    implementation 'org.apache.pulsar:pulsar-client'
    implementation 'org.apache.pulsar:pulsar-client-admin'
}

Connecting to Pulsar Cluster

To connect to a Pulsar cluster, you need to specify the Pulsar protocol URL. Here is an example of how to connect to a local Pulsar cluster:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

Developing Authentication and Authorization Plugins

One of the most critical aspects of any messaging system is security. Let’s explore how to develop authentication and authorization plugins for Apache Pulsar using the Biscuit framework.

Biscuit Authentication and Authorization

Biscuit is a token-based authentication system that provides fine-grained access control. Here’s how you can integrate Biscuit with Pulsar:

Dependencies

You need to include the following dependencies in your Pulsar setup:

wget -P "pulsar/lib" "https://repo1.maven.org/maven2/io/vavr/vavr/0.10.3/vavr-0.10.3.jar"
wget -P "pulsar/lib" "https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/3.25.0/protobuf-java-3.25.0.jar"
wget -P "pulsar/lib" "https://repo1.maven.org/maven2/com/clever-cloud/biscuit-java/<VERSION>/biscuit-java-<VERSION>.jar"
wget -P "pulsar/lib" "https://repo1.maven.org/maven2/com/clever-cloud/biscuit-pulsar/<VERSION>/biscuit-pulsar-<VERSION>.jar"

Configuration

To enable Biscuit authentication and authorization, you need to configure your broker.conf, proxy.conf, or standalone.conf files:

# Enable Biscuit authentication
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.BiscuitAuthenticationProvider

# Enable Biscuit authorization
authorizationEnabled=true
authorizationProvider=org.apache.pulsar.broker.authorization.BiscuitAuthorizationProvider

Example Workflow

Here is a high-level workflow diagram showing how Biscuit authentication and authorization work with Pulsar:

sequenceDiagram participant Client as Pulsar Client participant Broker as Pulsar Broker participant Biscuit as Biscuit Server Note over Client,Broker: Client requests connection to Pulsar Client->>Broker: Connect request with Biscuit token Broker->>Biscuit: Validate Biscuit token Biscuit->>Broker: Token validation response Broker->>Client: Connection response (success or failure) Note over Client,Broker: Client requests access to topic Client->>Broker: Access request to topic Broker->>Biscuit: Check permissions for topic access Biscuit->>Broker: Permission response Broker->>Client: Access response (success or failure)

Creating Producers, Consumers, and Readers

Now that we have our environment set up and security plugins in place, let’s dive into creating the core components of a Pulsar application: producers, consumers, and readers.

Creating a Producer

A producer is responsible for sending messages to a Pulsar topic. Here’s an example of how to create a producer:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

Producer<byte[]> producer = client.newProducer()
        .topic("my-topic")
        .create();

producer.send("My message".getBytes());
producer.close();

Creating a Consumer

A consumer subscribes to a topic and handles messages published by producers. Here’s how you can create a consumer:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

Consumer consumer = client.newConsumer()
        .topic("my-topic")
        .subscriptionName("my-subscription")
        .subscribe();

while (true) {
    Message message = consumer.receive();
    System.out.println("Message received: " + new String(message.getData()));
    consumer.acknowledge(message);
}
consumer.close();

Creating a Reader

A reader allows you to manually position yourself within a topic and read messages from a specified point. Here’s an example:

PulsarClient client = PulsarClient.builder()
        .serviceUrl("pulsar://localhost:6650")
        .build();

byte[] msgIdBytes = // Some message ID byte array
MessageId id = MessageId.fromByteArray(msgIdBytes);

Reader reader = client.newReader()
        .topic("my-topic")
        .startMessageId(id)
        .create();

while (true) {
    Message message = reader.readNext();
    // Process message
}
reader.close();

Packaging Java Functions

In addition to producers, consumers, and readers, Pulsar also supports Java functions that can be packaged and deployed. Here’s how you can package a Java function as a JAR:

Maven Project Setup

Create a new Maven project with the following pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>java-function</groupId>
    <artifactId>java-function</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.pulsar</groupId>
            <artifactId>pulsar-functions-api</artifactId>
            <version>3.3.1</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
                <configuration>
                    <release>17</release>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Packaging and Running the Function

Package the function using Maven:

mvn package

Run the function using the pulsar-admin command:

./bin/pulsar-admin functions localrun \
  --classname org.example.test.ExclamationFunction \
  --jar $PWD/target/java-function-1.0-SNAPSHOT.jar \
  --inputs persistent://public/default/my-topic-1 \
  --output persistent://public/default/test-1 \
  --tenant public \
  --namespace default \
  --name JavaFunction

Conclusion

Developing plugins for Apache Pulsar using Java is a powerful way to extend and customize the functionality of this robust messaging system. From setting up your environment and connecting to the Pulsar cluster, to creating producers, consumers, readers, and even packaging Java functions, this guide has walked you through the key steps involved in plugin development.

Remember, the world of messaging systems is vast and complex, but with the right tools and a bit of creativity, you can build highly scalable and secure applications. So, go ahead, dive into the world of Pulsar, and see what wonders you can create!

flowchart LR A[Set_Up_Environment] --> B[Connect to Pulsar Cluster] B --> C[Develop Authentication & Authorization Plugins] C --> D[Create Producers, Consumers, Readers] D --> E[Package Java Functions] E --> F[Deploy and Run] F --> G[Monitor and Optimize] G --> B[Enjoy_Your_Scalable_Messaging_System]