NATS
Sequin comes with managed consumers you can connect to your streams that will forward data to popular targets. One of those targets is a NATS instance or cluster.
NATS is a simple yet high-performance open-source messaging system. It’s comparable to systems like Apache Kafka and AWS’ SQS, but has a few additional features that make it a great fit for processing API data at scale. NATS is part of the Cloud Native Computing Foundation (CNCF).
NATS is a great option for streaming API data to your application or database.
Overview
You can use Sequin’s NATS consumer to replicate data from Sequin’s records stream or events stream to your own NATS instances.
A single NATS consumer can consume records or events from multiple syncs and collections.
You can choose to initialize your NATS consumer at the beginning or at the end of the stream.
Sequin’s NATS consumer is located in AWS’ us-west-2.
Subject patterns
By default, Sequin will publish records to your NATS cluster using the subject pattern:
sequin-records.default.{sync_id}.{provider}.{collection_uuid}.{record_upstream_id}
For example:
sequin-records.default.902261b0-6b22-48a0-8d6b-2fdb16cf8402.servicenow.servicenow:incident.inc_2891
Events will be published using the subject pattern:
sequin-events.default.{sync_id}.{provider}.{collection_uuid}.{record_upstream_id}.{event_type}
Where event_type is one of inserted, updated, or deleted.
For example:
sequin-events.default.902261b0-6b22-48a0-8d6b-2fdb16cf8402.servicenow.servicenow:incident.inc_2891.inserted
Setup
Create your NATS streams
We recommend creating two streams, one for records and one for events. Your records stream will have the most recent version of every API record. We recommend initializing it like this:
nats stream add SEQUIN-RECORDS --subjects 'sequin-records.>' --retention limits --storage file--max-msgs-per-subject 1 --allow-purge
The key parameter is max-msgs-per-subject 1. This will ensure that your stream only holds the most recent version of each record.
For the events stream, you can initialize it like this:
nats stream add SEQUIN-EVENTS --subjects 'sequin-events.>' --retention limits --storage file --max-msgs-per-subject 100 --discard old --max-age 7d --allow-purge
You’ll want to choose some reasonable parameter for max-msgs-per-subject and max-age based on your use case.
Create a NATS user for Sequin
You’ll create a NATS user for Sequin and then supply Sequin with the credentials. Sequin will use this NATS user to connect to your cluster and publish messages to your streams.
Add the NATS cluster to Sequin (console)
After creating a NATS user for Sequin, you’ll connect Sequin to your NATS cluster.
Click on “Targets” in the Sequin console and then click on “Add new → NATS.” That will bring up Sequin’s NATS connection flow.
You’ll be prompted for details like your cluster’s hostname, port, and the credentials for the NATS user you created for Sequin. You can have Sequin connect to your cluster via a bastion host if you wish.
Add the NATS cluster to Sequin (API)
Alternatively, you can add clusters to Sequin via our Management API.
Setup the consumer
The consumer is the worker cluster that will pull objects off your Sequin streams and push them to your NATS cluster. You can have one consumer pull data from multiple collections across multiple syncs writing to one NATS cluster.
In the Sequin console, click on “Consumers” and then click on “Add new → NATS.” That will bring up Sequin’s NATS consumer setup flow:
1. Select target
Select your NATS cluster as the target.
2. Select stream
Select which stream you want to consume from, the record or event stream.
The record stream is great if downstream consumers only need to know the current state of an API object. This is desireable for most workflows. Otherwise, the event stream will give you each change event for all API objects.
3. Select syncs
Select which syncs you want this consumer to pull data from. You can change this list of syncs at any time in the future.
You can have one NATS consumer pull data from one sync or thousands of syncs.
4. Select collections
Select which collections you want this consumer to consume, for example Salesforce contacts or Stripe subscriptions.
5. Select start position
You can have your NATS consumer start consuming from the beginning of the stream or from the end.
6. Save
Save your consumer. Sequin will provision a consumer cluster and begin populating your NATS topic right away.
Don’t have NATS in production?
One of the best parts about NATS is that it’s simple to setup and run (it’s a single Go binary). Alternatively, the creators of NATS offer a serverless managed version, Synadia.