Core concepts
A rundown of Sequin from top to bottom:
- Streams contain a queue of records that are emitted by syncs and consumed by consumers.
- Syncs extract records from APIs and push them into a stream.
- Credentials are the API keys, OAuth tokens, or other credentials that Sequin uses to access the API.
- Collections are classes of records in the API. For example, Stripe has a
stripe:subscription
collection and Salesforce has asalesforce:contact
collection. - Frequency is the rate at which Sequin detects changes in the API and push them to your stream.
- Backfills are the process of extracting historical data from the API.
- Consumers are how you pull data from your stream using Sequin’s Serverless Message Bus.
- Mutations are how you write changes back to the API.
All this and more can be managed in the console or via the Management API.
Streams
A stream contains a queue of all the API records extracted by the syncs within the stream. A record contains the current state of an object in the API.
The record stream contains each record only once:
- When Sequin detects a record insert in the API, it appends the record to the stream.
- When Sequin detects a record update in the API, it pops the old record and appends the updated record to the stream.
- When Sequin detects a record delete in the API, it pops the old record and appends the updated record to the stream with the
deleted
property set totrue
.
Records
A Record
is a JSON object that contains the following properties:
The ID of the collection that generated this event (e.g. stripe:subscription
or salesforce:contact
).
All the fields of the API object. This is the raw response from the API.
Whether the record has been deleted in the API.
Sequin’s ID of the record in the stream.
The time the record was inserted into the stream, in ISO 8601.
The API provider of the record (e.g. stripe
or salesforce
).
The ID of the stream that generated this record message.
The sync that generated this event.
A integer representing the version of this record. updated_idx
increases every time a record is created, updated, or deleted.
The time the record was created in the API, in ISO 8601.
The ID of the record in the API.
The time the record was last updated in the API, in ISO 8601.
Here’s an example of a Record
:
{
"collection_id": "salesforce:lead",
"data": {
"FirstName": "Annadiane",
"Company": "Rhynyx",
...
},
"deleted": false,
"id": "15060066-6265-462b-9032-10c8ac3ec846",
"inserted_at": "2024-05-15T18:41:07.574770Z",
"provider": "salesforce",
"stream_id": "a213ee65-20b7-46d5-add6-54c027d0eecc",
"sync_external_id": null,
"sync": {
"external_id": "user_123",
"id": "10ab504c-cb92-4662-8116-8bc1aee0f609",
"metadata": {},
},
"updated_idx": 2630091204,
"upstream_created_at": "2023-04-20T16:20:40.000000Z",
"upstream_id": "00Q8V00001RCmLwUAL",
"upstream_updated_at": "2023-04-20T16:39:41.000000Z"
}
Sequin may add fields to records at any time. If you’re using a JSON validator to parse records, you should allow for unknown fields.
Syncs
To extract records from an API and push them into a stream, you create a sync. When you create a sync, Sequin first extracts historical data from the API via a backfill. Then, Sequin monitors the API for changes and emits records when changes occur.
Credentials
To access an API, Sequin needs credentials. These are the API keys, OAuth tokens, or other credentials that Sequin uses to access the API. You can create credentials in three ways:
- Console: You can create a credential when creating a sync in the Sequin console.
- Import: You can import a credential using the Management API. This is the case if you own the auth flow.
- Sequin OAuth: You can implement Sequin’s OAuth2 flow to allow users to connect their own accounts.
Collections
A collection is a class of records in the API. In RESTful APIs, a collection usually corresponds to one API endpoint. For example, Stripe has a stripe:subscription
collection located at /v1/subscriptions
and Salesforce has a salesforce:contact
collection that you can access at /services/data/v51.0/sobjects/Contact
.
Sequin uses unique IDs to refer to each collection. The ID is a combination of the API provider’s name and the collection name. Some examples:
- Stripe subscriptions ->
stripe:subscription
- Salesforce contacts ->
salesforce:contact
- HubSpot pipelines ->
hubspot:pipeline
- Shopify products ->
shopify:product
- GitHub repositories ->
github:repository
For API providers that have custom collections, like Salesforce and HubSpot, Sequin assigns the collection a unique ID. For example:
- Salesforce Custom Object
MyObject__c
->salesforce:szMO52XHsHps1DZRrmdvKbch
- HubSpot Custom Object
MyObject
->hubspot:DR8V5vaukyB2uJZW085AjILH
Backfills
When you first setup your sync, Sequin will backfill historical data from the API. You can configure how far back in time Sequin will backfill. Sequin will also kickoff a backfill process whenever you add a new collection to your sync.
The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you’ve allocated for Sequin.
Backfills involving millions of records can take several hours. You can view the status of your backfill at any time in the Sequin console or via the Management API.
Frequency
Sequin syncs data from APIs at a frequency that you specify. You can sync data in real-time (e.g. continuous
syncs) or on a schedule (e.g. 4h
syncs).
For continuous syncs, the latency between the API change and the record appearing in your stream is usually a couple seconds - but this depends on the API and the rate limit you’ve set for Sequin.
Rate limits
Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of an API’s total quota. The amount of your quota we use is configurable in the Sequin console or via the Management API.
The time it takes Sequin to detect changes in the API and push to your streams is called the “max lag time” of any given change. Max lag time is subject to a few factors:
- the API you’re syncing from
- the collections you’re syncing from that API
- the rate limit you set Sequin to consume
- whether the collection supports webhooks or other fast means of detecting changes
Consumers
You can use Sequin’s serverless message bus to consume from your record stream. The Consume APIs are a set of HTTP interfaces to your streams. You can paginate through all the records in the stream, ack
and nack
records, and replay records.
Learn more about the Consume API.
Mutations
Sequin makes it easy to write changes back to upstream APIs. Most collections that Sequin syncs are mutable. To create a record in that collection or mutate an existing one, you can use Sequin’s Mutation API.
Management API
You can use Sequin’s Management API to develop applications that create and manage Sequin syncs. You can automate everything that you can do in the Sequin console. This includes creating, updating, and deleting syncs on behalf of your users using their provided credentials, as well as turning on and off consumers.
Was this page helpful?