Streaming to WarpStream

​

What’s WarpStream?

WarpStream is a fully-managed messaging system. It’s Kafka-compatible, meaning it will work with any Kafka client. WarpStream enhances the functionality of Apache Kafka by integrating seamlessly with object storage. This design reduces storage costs, making WarpStream an affordable alternative to Kafka.

Sequin makes it easy to stream data from platforms like Salesforce, Jira, and Stripe into WarpStream. This enables a variety of real-time use cases, such as:

• Triggering notifications when a new high-priority Jira ticket is created
• Updating an internal database record whenever a Salesforce contact is updated
• Calculating real-time metrics on Stripe payments

To stream data to WarpStream via Sequin, you will:

• Create a WarpStream Kafka cluster and topic
• Connect Sequin to an API source
• Create a Sequin Kafka consumer that streams to the WarpStream topic

Sequin will first load all historic data into the Kafka topic and then stream new records and updates as they occur. This means your Kafka topic will always have a complete copy of your API data.

​

Create a WarpStream cluster

Sign up for a WarpStream account at https://console.warpstream.com/signup.

Once logged in, create a new cluster to receive data from Sequin. Give the cluster a name (e.g. stripe), select your cluster type (e.g. serverless), and choose a region.

​

Create a Kafka topic

Next, create a Kafka topic for your Sequin data. You can do this using the WarpStream CLI or in the WarpStream console. For simplicity, we’ll walk you through the WarpStream console:

1. From your cluster overview page, click the “Topics” button on the cluster you just created.
2. Click ”+ Create Topic”. Give the topic a name (e.g. sequin-records), set the number of partitions (1 is fine to start), and set the retention (e.g. 168 hours / 7 days).
3. Click ”+ Create Topic”.

​

Create credentials for Sequin

Now that your cluster is provisioned and you have a Kafka topic, you need to create a set of credentials for Sequin to use when sending data to WarpStream.

1. Click the “Credentials” tab.
2. Click ”+ Create Credentials.”
3. Give your user a name (i.e. sequin).
4. Sequin does not need Superuser permissions, so you can leave that unchecked.
5. Click create and note your credentials - you’ll need these later.

​

Connect Sequin to an API source

You can connect Sequin to an upstream data source using either the Sequin console or API.

Here’s a curl request that connects Sequin to Stripe:

curl --request POST \
  --url https://api.sequin.io/v1/syncs \
  --header 'Authorization: Bearer {your-api-token}' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "stripe",
    "collection_ids": ["stripe:customer", "stripe:charge", "stripe:payout"],
    "credential": {
      "kind": "secret_key",
      "secret_key": "sk_live_51I..."
    }
  }'

This request instructs Sequin to sync Stripe customers, charges, and payouts using the provided API key.

​

Create a Sequin Kafka consumer

Next, you’ll create a Kafka consumer to stream data from Stripe to your WarpStream Kafka topic. Consumers are how you stream data from Sequin’s syncs to destinations.

You can create consumers in Sequin’s console or API. Here’s an example curl request to create a Kafka consumer:

curl --request POST \
  --url https://api.sequin.io/v1/kafka_consumers \
  --header 'Authorization: Bearer {your-api-token}' \
  --header 'Content-Type: application/json' \
  --data '{
    "name": "WarpStream - Stripe",
    "kafka": {
      "brokers": ["serverless.warpstream.com:9092"],
      "topic": "sequin-records",
      "key_prefix": "sequin",
      "credential": {
        "kind": "SASL_SSL",
        "mechanism": "SCRAM-SHA-256",
        "username": "ccun_1fadd837bde...",
        "password": "ccp_cb90a906c3c50..."
      }
    },
    "sync_id": "8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8"
  }'

You’ll find the brokers for your WarpStream cluster in the WarpStream console on the “Connect” tab.

Use the topic and credential you provisioned for Sequin earlier. key_prefix is optional and can be used to prefix the Kafka record key (defaults to "sequin").

This configuration will send each Stripe record to the Kafka topic sequin-records.

At this point, your Stripe data should be flowing into WarpStream! Let’s verify by consuming from the topic.

​

Consume from the WarpStream Kafka topic

There are many ways to consume data from a WarpStream topic.

For this example, we’ll use the WarpStream CLI to consume the latest record from your topic:

warpstream kcmd --type fetch --topic sequin-records --offset 0

You should see a response like this:

result: partition:0 offset:0 key:"sequin" value:"{\"id\":\"8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8\",\"data\": {\"collection_id\":\"stripe:charge\",\"sync_id\":\"63b8f7d2-c71a-4ad4-8e2e-e20877d812ac\",\"data\": {\"id\":\"ch_1N5w6wI5GiWQwPxYVZQ5DvIR\",\"object\":\"charge\",\"amount\": 2000,\"amount_captured\": 2000,\"amount_refunded\": 0,...}},\"timestamp\":1683749961535}"

The value field contains the stringified JSON record from Sequin.

Now, any time a Stripe customer, charge, or payout is created, updated, or deleted, Sequin will send the record to your WarpStream Kafka topic in real-time!

​

Where to next?

By streaming your API data to WarpStream, you’ve opened up a world of possibilities for real-time applications.

If you weren’t following along, head over to WarpStream to sign up for an account and start streaming.

Check out the WarpStream documentation to learn more about their platform, and the Sequin docs for more on streaming API data.