​

What’s Redpanda?

Redpanda is a Kafka API-compatible streaming data platform that simplifies real-time data processing. Redpanda Serverless is a fully managed, hosted solution for Redpanda clusters in the cloud, making it easy to get started with streaming without managing infrastructure.

With Sequin, you can easily stream data from APIs like Salesforce, Jira, and Stripe into Redpanda topics. This enables a variety of real-time use cases, such as:

• Triggering notifications when a new high-priority Jira ticket is created
• Updating an internal database whenever a Salesforce contact is updated
• Calculating real-time metrics on Stripe payments

To stream data to Redpanda Serverless via Sequin, you will:

• Create a Redpanda Serverless account and cluster
• Create a Redpanda topic to receive data
• Create an API key for Sequin
• Connect Sequin to an API source
• Create a Sequin Redpanda consumer that streams to the topic

Sequin will first send all historical data from the API to the Redpanda topic, then will keep the topic up-to-date by sending new records and updates as they occur. This means your Redpanda topics will always have a complete copy of your API data.

​

Create a Redpanda Serverless account and cluster

Sign up for a Redpanda Serverless account at https://console.redpanda.com/signup.

Once logged in, click “Create Cluster”. Give your cluster a name, select a cloud provider and region, and click “Create”.

After your cluster is provisioned, click on it to view the cluster details.

​

Create a Redpanda topic to receive data

From the cluster details page, navigate to the “Topics” tab and click “Create Topic”. Give the topic a name (e.g. sequin-events). If you’re not sure about the right values for “Partitions” and “Retention time”, specify 6 for “Partitions” and 7 days for “Retention time”.

Click “Create Topic”.

​

Create a user for Sequin

Next, create a Kafka user that Sequin will use to authenticate with your Redpanda Serverless cluster.

Navigate to the “Security” tab and click “Create user”. Give your user a name (e.g. sequin). You can let Redpanda generate a password for you. Choose the SASL mechanism SCRAM-SHA-512 and click “Create”.

Be sure to store the credentials for the user somewhere secure. You’ll need them in a bit when you’re setting up Sequin.

​

(Optional) Create a Kafka ACL for Sequin

Access Control Lists (ACLs) in Kafka help you manage access to topics within your Redpanda cluster. We recommend you restrict Sequin’s user to just the access it needs:

1. Go to the “Security” tab of your cluster’s details page.
2. Click on “ACLs” and then “Create ACL”.
3. Make sure “User” is selected in the dropdown, then type in the username you created earlier (e.g., sequin).
4. Under “Topics”, specify the topic you created earlier in the input field (e.g., sequin-events).
5. Make sure “Write”, “Describe”, and “Describe Configs” are set to “Allow”. The rest on that topic can be set to “Deny”.
6. Click “OK”.

​

Connect Sequin to an API source

You can connect Sequin to an upstream API using either the Sequin console or API.

Here’s a curl request that connects Sequin to Stripe:

curl --request POST \
  --url https://api.sequin.io/v1/syncs \  
  --header 'Authorization: Bearer {your-api-token}' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "stripe",
    "collection_ids": ["stripe:customer", "stripe:charge", "stripe:payout"],
    "credential": {      
      "kind": "secret_key",
      "secret_key": "sk_live_51I..."
    }
  }'

This request instructs Sequin to sync Stripe customers, charges, and payouts using the provided API key.

​

Create a Sequin Kafka consumer

Next, you’ll create a Kafka consumer to stream data from Stripe to your Redpanda topic. Consumers are how you stream data from Sequin’s syncs to destinations.

You can create consumers in Sequin’s console or API. Here’s an example curl request to create a Kafka consumer:

curl --request POST \
  --url https://api.sequin.io/v1/kafka_consumers \
  --header 'Authorization: Bearer {your-api-token}' \  
  --header 'Content-Type: application/json' \
  --data '{
    "name": "Redpanda - Stripe",
    "redpanda": {
      "brokers": ["abc123.us-west-2.aws.redpanda.com:9092"],
      "topic": "sequin-records",
      "key_prefix": "sequin",
      "sasl": {
        "mechanism": "SCRAM-SHA-512",
        "username": "{redpanda-sequin-user}",  
        "password": "{redpanda-sequin-password}"
      }
    },
    "sync_id": "8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8" 
  }'

For brokers, use the bootstrap server URL from your Redpanda cluster details page. This typically looks like abc123.us-west-2.aws.redpanda.com:9092.

For topic, use the name of the Redpanda topic you created earlier.

For sasl.username and sasl.password, use the Kafka user you created for Sequin.

key_prefix is optional and can be used to prefix the Redpanda record key (defaults to "sequin").

The sync_id field specifies which Sequin sync to consume data from. Omit this field to consume data from all syncs.

At this point, your Stripe data should be flowing into Redpanda! Let’s verify by consuming from the topic.

​

Consume from the Redpanda topic

To consume from your Redpanda topic, navigate to the “Topics” tab in your Redpanda cluster, click on your topic, and then click “Messages”.

You should see Stripe records flowing in! They’ll look something like this:

{
  "key": "sequin.default.8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8.stripe.stripe:charge.ch_1N5w6wI5GiWQwPxYVZQ5DvIR",
  "value": {
    "collection_id": "stripe:charge",
    "sync_id": "63b8f7d2-c71a-4ad4-8e2e-e20877d812ac",
    "data": {
      "id": "ch_1N5w6wI5GiWQwPxYVZQ5DvIR",
      "object": "charge",
      "amount": 2000,
      "amount_captured": 2000,
      "amount_refunded": 0,
      // ...
    },
    // ...
  }
}

Now, any time a Stripe customer, charge, or payout is created or updated, Sequin will send the record to your Redpanda topic in real-time!

​

Where to next?

By streaming your API data to Redpanda, you’ve opened up a world of possibilities for real-time applications. Some ideas:

• Use Redpanda’s Kafka Connect integration to sync data from Redpanda to other systems like databases, search indexes, or data warehouses
• Build real-time dashboards by processing the Redpanda topic with a stream processing framework like ksqlDB or Kafka Streams
• Trigger real-time actions by consuming the Redpanda topic in your application code

Check out the Redpanda documentation to learn more about the platform, and the Sequin docs for more on streaming API data.

Happy streaming!