Streaming to NATS (Synadia)

​

What’s NATS?

NATS is a high-performance, open-source messaging system backed by the CNCF. While NATS is often compared to tools like Apache Kafka, RabbitMQ, and AWS Kinesis, it differentiates with a minimal protocol and ease of operation. Common NATS use cases include event streaming and microservice communication.

Synadia is a hosted and managed NATS service from the creators of NATS. Their service runs in a variety of data centers, cloud providers, and regions, enabling low latency messaging worldwide.

NATS is a phenomenal tool for working with API data. With Sequin, you can stream API data from services like Salesforce, Jira, and Zendesk to NATS in real-time. This allows you to build reactive, event-driven applications that respond to changes in your data. You can trigger workflows when a Stripe subscription status changes, ingest GitHub data into your database, or push data to an internal API when a Zendesk ticket is created.

To stream data to NATS via Sequin, you will:

• Create a Synadia account and NATS stream
• Connect Sequin to an API source
• Create a Sequin NATS consumer that streams to the NATS subject
• Create a consumer in NATS to receive the data

Sequin will first publish all historical data from the API to the NATS stream, then will publish changes as they appear in the API. This means NATS consumers will receive the complete data set plus incremental updates.

​

Create a Synadia account and NATS stream

​

Preparation

​

1. Install the CLI

First, install the nats CLI. You’ll need this to provision streams and consumers:

brew tap nats-io/nats-tools
brew install nats-io/nats-tools/nats

​

2. Create your Synadia account

Next, sign up for a Synadia account at https://synadia.com/.

​

3. Create your CLI user

Create a user for your account so you can interface with your NATS cluster via the CLI:

1. Click Default to open the default account.
2. Click the Users tab.
3. Click “Create user.” You can call this user whatever you’d like, like “CLI.”
4. For now, unrestricted access for this user is fine. Click “Save.”
5. In the “Users” table, click on the user you just created.
6. Click Download > Download Credentials to download the user’s credentials.

Now you can interface with your NATS cluster using the CLI by referring nats to the location of this credential file.

​

4. Create a user for Sequin

While you’re here, create a user for Sequin to use to publish to your NATS stream.

Restrict this user so that it can only publish and subscribe to a given subject prefix. We recommend using the subject prefix sequin-records, so restrict this user to publish and subscribe to sequin-records.> (meaning all messages that start with that prefix).

​

Provision a NATS stream

Sequin will publish records to your NATS cluster using the following subject pattern:

{subject_prefix}.default.{sync_id}.hubspot.{collection_id}.{record_upstream_id}

Where:

• subject_prefix - A prefix for all Sequin messages, set by you.
• sync_id - The ID of the sync that the record came from.
• collection_id - The ID of the collection that the record belongs to, like stripe:subscription or salesforce:contact.
• record_upstream_id - The ID of the record in the upstream API.

You’ll create a new NATS stream to store the messages Sequin publishes. NATS streams are a durable log of messages.

We recommend setting up your NATS stream such that:

• All Sequin messages go to one stream
• “Max messages per subject” is set to 1 and “discard” is set to old so that only the latest version of each record is kept
• There is no maximum age for messages, so that the stream contains every record from the upstream API

Here’s how you use the nats CLI to create a stream with those properties:

nats stream add SEQUIN-RECORDS --subjects "sequin-records.>" --max-msgs-per-subject=1 --discard=old

In this example, sequin-records is the subject_prefix for Sequin messages (the default). Importantly, the > character is a wildcard that matches any subject that starts with sequin-records.. This means that the stream will receive messages published to any subject that starts with sequin-records..

We recommend the stream name SEQUIN-RECORDS and the subject prefix sequin-records for Sequin messages, but you can use any names you’d like.

​

Connect Sequin to an API source

You can connect Sequin to an upstream API using either the Sequin console or API.

Here’s a curl request that connects Sequin to Salesforce:

curl --request POST \
  --url https://api.sequin.io/v1/syncs \  
  --header 'Authorization: Bearer {your-api-token}' \
  --header 'Content-Type: application/json' \
  --data '{
    "provider": "salesforce",
    "name": "Choam Corp - Salesforce Production",    
    "collection_ids": ["salesforce:opportunity", "salesforce:contact"],
    "credential": {
      "properties": {
        "kind": "salesforce_oauth2",
        "secret": "dBeAXpLjPaYKN67uJLsh8ZEKZYQz1O52bQ",  
        "name": "Salesforce prod"
      },
      "oauth_app_id": "8ff58ef0-d376-4ae8-b2e2-9f0206aa65b8",      
    }
  }'

This request provides Sequin with an OAuth2 refresh token it will use to make requests to Salesforce. The collection_ids field specifies that Sequin should sync Salesforce Opportunity and Contact records.

​

Create a Sequin → NATS consumer

Next, create a Sequin → NATS consumer to publish data from the Salesforce account to the NATS stream. Sequin consumers route data from Sequin’s API syncs to downstream destinations.

You’ll need the JWT and nkey seed for the user you created for Sequin. If you didn’t already create this user and download its credentials, refer to the preparation section above.

You can create consumers in Sequin’s console or API. Here’s an example curl request:

curl --request POST \
  --url https://api.sequin.io/v1/nats_consumers \
  --header 'Authorization: Bearer {your-api-token}' \  
  --header 'Content-Type: application/json' \
  --data '{
    "sync_id": "e39875af-cd83-4406-8d46-3b30e0b5b624",
    "name": "Synadia - Salesforce", 
    "subject_prefix": "sequin-records",
    "nats": {
      "host": "connect.ngs.global",
      "port": 4222,
      "tls": true,
      "jwt": "eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiIsImtpZCI6IjEifQ.",
      "nkey_seed": "SUACj3ZI512ZjN1xE7iTYQNRcW20ySpZWhQ"
    }
  }'

You can specify a sync_id and Sequin will only publish data from that sync. Omit it and Sequin will publish data from all your API syncs.

At this point, your Salesforce Opportunities and Contacts should be streaming into NATS in real-time! Let’s verify by creating a consumer and fetching a message for it.

​

Create a consumer in NATS to receive the data

Now that Sequin is publishing data to your NATS stream, you can create a consumer and receive messages via the CLI to see everything working end-to-end.

To create a consumer, you can use these settings:

nats consumer add SEQUIN-RECORDS --pull --deliver=all --ack=explicit --defaults my_consumer

To verify that the consumer is receiving messages, you can use the nats consumer next command to retrieve the next message:

nats consumer next SEQUIN-RECORDS my_consumer

This will print out the first message in the stream. You should see a Salesforce Opportunity or Contact record in JSON format, like this:

{
  "collection_id":"salesforce:opportunity",
  "sync_id": "e39875af-cd83-4406-8d46-3b30e0b5b624", 
  "data": {
    "AccountId": "0013t00002YourAccountId",
    "Amount": 5000,
    "CloseDate": "2023-06-01",
    "Description": "Big new deal", 
    "Name": "Cyberdyne Systems POC",
    "StageName": "Prospecting", 
    "upstream_id": "0063t000YourOpportunityId"  
  },
  "upstream_created_at": "2023-04-01T12:00:00Z",
  "upstream_updated_at": "2023-04-01T12:00:00Z"
}

nats consumer next SEQUIN-RECORDS my_consumer | tail -n 1 | jq

Now, any time a Salesforce Opportunity or Contact is created, updated, or deleted, Sequin will publish the change to NATS and your consumer will receive it!

​

Where to next?

By publishing your API data to NATS, you’ve made it available for any service to consume and process in real-time. You can now easily build reactive, event-driven applications off your API data.

If you weren’t following along, head over to Synadia to sign up for an account and start streaming.

Check out the NATS documentation to learn more about building applications with NATS, and the Sequin docs for more on syncing API data.