What’s Kafka?

Apache Kafka is a distributed event store and stream-processing platform. It lets you store streams of records in a fault-tolerant way and process streams of records as they occur. Kafka is commonly used for building real-time streaming data pipelines and applications.

Confluent is a complete event streaming platform built on top of Kafka. It offers a hosted Kafka service, along with additional tools and features to make working with Kafka easier and more secure.

Sequin makes it easy to stream data from APIs like Salesforce, Jira, and Stripe into Kafka topics. This enables a variety of real-time use cases, such as:

  • Triggering notifications when a new high-priority Jira ticket is created
  • Updating an internal database whenever a Salesforce contact is updated
  • Calculating real-time metrics on Stripe payments

To stream data to Confluent Kafka via Sequin, you will:

  • Create a Confluent Kafka cluster
  • Create a Kafka topic to receive data
  • Create an API key for Sequin
  • Connect Sequin to an API source
  • Create a Sequin Kafka consumer that streams to the Kafka topic

Sequin will first send all historical data from the API to the Kafka topic, then will keep the topic up-to-date by sending new records and updates as they occur. This means your Kafka topics will always have a complete copy of your API data.

Create a Confluent Kafka cluster

Sign up for a Confluent account at https://www.confluent.io/get-started/.

Once logged in, navigate to the “Environments” page and click “Create cluster”. Give your cluster a name, select a cloud provider and region, and choose the “Basic” cluster type.

After your cluster is provisioned, select it from the “Environments” page. This will take you to the cluster overview.

Create a Kafka topic to receive data

From the cluster overview page, navigate to the “Topics” tab and click “Create topic”. Give the topic a name (e.g. sequin-records). The default number of partitions (6) is fine.

We recommend toggling on “Enable infinite retention for this topic.” Click “Show advanced settings.” Set the “Cleanup policy” to “Compact.” This will ensure that Kafka retains the latest version of each record indefinitely.

Click “Save & create.”

Create an API key for Sequin

Next, create an API key that Sequin will use to authenticate with your Kafka cluster.

In your cluster, navigate to the “API keys” tab and click “Add key”. Give your key a description (e.g. sequin-producer), select the “Global access” permission, and click “Save”.

On the next screen, make sure to save the API key and secret. You’ll need these in a later step and won’t be able to access them again.

Connect Sequin to an API source

You can connect Sequin to an upstream API using either the Sequin console or API.

Here’s a curl request that connects Sequin to Stripe:

curl --request POST \
  --url https://api.sequin.io/v1/syncs \  
  --header 'Authorization: Bearer {your-api-token}' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "stripe",
    "collection_ids": ["stripe:customer", "stripe:charge", "stripe:payout"],
    "credential": {      
      "kind": "secret_key",
      "secret_key": "sk_live_51I..."
    }
  }'

This request instructs Sequin to sync Stripe customers, charges, and payouts using the provided API key.

Create a Sequin Kafka consumer

Next, you’ll create a Kafka consumer to stream data from Stripe to your Confluent Kafka topic. Consumers are how you stream data from Sequin’s syncs to destinations.

You can create consumers in Sequin’s console or API. Here’s an example curl request to create a Kafka consumer:

curl --request POST \
  --url https://api.sequin.io/v1/kafka_consumers \
  --header 'Authorization: Bearer {your-api-token}' \  
  --header 'Content-Type: application/json' \
  --data '{
    "name": "Confluent - Stripe",
    "kafka": {
      "brokers": ["pkc-abcde.us-west-2.aws.confluent.cloud:9092"],
      "topic": "sequin-records",
      "key_prefix": "sequin",
      "credential": {
        "kind": "SASL_SSL",
        "mechanism": "PLAIN",
        "username": "PwKEZlr8tU0EhEsjtQ",
        "password": "G09bw7Umv5g7mLdzvotTJJP1BltV7nWiBmBY1stQ9zwWN"
      }
    },
    "sync_id": "8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8"
  }'

For brokers, use the bootstrap server URL from your Confluent cluster’s settings page. This typically looks like pkc-n98pk.us-west-2.aws.confluent.cloud:9092.

For topic, use the name of the Kafka topic you created earlier.

For credential.username and credential.password, use the API key and secret you created for Sequin.

key_prefix is optional and can be used to prefix the Kafka record key (defaults to "sequin").

The sync_id field specifies which Sequin sync to consume data from. Omit this field to consume data from all syncs.

By default, the consumer will start sending records from the beginning of time, giving you the complete data set in Kafka. If you’d like, you can instead have Sequin send records from only this point forward.

At this point, your Stripe data should be flowing into Kafka! Let’s verify by consuming from the topic.

Consume from the Kafka topic

There are many ways to consume from a Kafka topic. For this example, we’ll use the Confluent CLI.

First, install the Confluent CLI by following these instructions.

Then, from your terminal, log in to your Confluent account:

confluent login

You’ll be prompted to enter your Confluent credentials.

Next, list your Kafka clusters:

confluent kafka cluster list

You should see your cluster in the list. Make note of its ID. Now, you can set that cluster to the active one:

confluent kafka cluster use [cluster-id]

Next, you’ll need to import your API key into the confluent CLI. Run this command:

confluent api-key store --resource [cluster-id]

The CLI will prompt you for the key and secret, which you downloaded when creating the cluster. After importing the credentials, you can tell the CLI to use them:

confluent api-key use [key]

Finally, consume from your topic:

confluent kafka topic consume sequin-records

You should see Stripe records flowing in! They’ll look something like this:

{
  "key": "sequin.default.8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8.stripe.stripe:charge.ch_1N5w6wI5GiWQwPxYVZQ5DvIR",
  "value": {
    "collection_id": "stripe:charge",
    "sync_id": "63b8f7d2-c71a-4ad4-8e2e-e20877d812ac",
    "data": {
      "id": "ch_1N5w6wI5GiWQwPxYVZQ5DvIR",
      "object": "charge",
      "amount": 2000,
      "amount_captured": 2000,
      "amount_refunded": 0,
      // ...
    },
    // ...
  }
}

Now, any time a Stripe customer, charge, or payout is created or updated, Sequin will send the record to your Kafka topic in real-time!

Where to next?

By streaming your API data to Kafka, you’ve opened up a world of possibilities for real-time applications. Some ideas:

  • Use Kafka Connect to sync data from Kafka to other systems like databases, search indexes, or data warehouses
  • Build real-time dashboards by processing the Kafka topic with a stream processing framework like ksqlDB or Kafka Streams
  • Trigger real-time actions by consuming the Kafka topic in your application code

Check out the Confluent documentation to learn more about their platform, and the Sequin docs for more on streaming API data.

Happy streaming!