Core concepts

A rundown of Sequin from top to bottom:

• Streams contain a queue of records that are emitted by syncs and consumed by consumers.
• Syncs extract records from APIs and push them into a stream.
  • Credentials are the API keys, OAuth tokens, or other credentials that Sequin uses to access the API.
  • Collections are classes of records in the API. For example, Stripe has a stripe:subscription collection and Salesforce has a salesforce:contact collection.
  • Frequency is the rate at which Sequin detects changes in the API and push them to your stream.
  • Backfills are the process of extracting historical data from the API.
• Consumers are how you pull data from your stream using Sequin’s Serverless Message Bus.
• Mutations are how you write changes back to the API.

All this and more can be managed in the console or via the Management API.

​

Streams

A stream contains a queue of all the API records extracted by the syncs within the stream. A record contains the current state of an object in the API.

The record stream contains each record only once:

• When Sequin detects a record insert in the API, it appends the record to the stream.
• When Sequin detects a record update in the API, it pops the old record and appends the updated record to the stream.
• When Sequin detects a record delete in the API, it pops the old record and appends the updated record to the stream with the deleted property set to true.

​

Records

A Record is a JSON object that contains the following properties:

Here’s an example of a Record:

{
  "collection_id": "salesforce:lead",
  "data": {
    "FirstName": "Annadiane",
    "Company": "Rhynyx",
    ...
  },
  "deleted": false,
  "id": "15060066-6265-462b-9032-10c8ac3ec846",
  "inserted_at": "2024-05-15T18:41:07.574770Z",
  "provider": "salesforce",
  "stream_id": "a213ee65-20b7-46d5-add6-54c027d0eecc",
  "sync_external_id": null,
  "sync": {
    "external_id": "user_123",
    "id": "10ab504c-cb92-4662-8116-8bc1aee0f609",
    "metadata": {},
  },
  "updated_idx": 2630091204,
  "upstream_created_at": "2023-04-20T16:20:40.000000Z",
  "upstream_id": "00Q8V00001RCmLwUAL",
  "upstream_updated_at": "2023-04-20T16:39:41.000000Z"
}

​

Syncs

To extract records from an API and push them into a stream, you create a sync. When you create a sync, Sequin first extracts historical data from the API via a backfill. Then, Sequin monitors the API for changes and emits records when changes occur.

​

Credentials

To access an API, Sequin needs credentials. These are the API keys, OAuth tokens, or other credentials that Sequin uses to access the API. You can create credentials in three ways:

1. Console: You can create a credential when creating a sync in the Sequin console.
2. Import: You can import a credential using the Management API. This is the case if you own the auth flow.
3. Sequin OAuth: You can implement Sequin’s OAuth2 flow to allow users to connect their own accounts.

​

Collections

A collection is a class of records in the API. In RESTful APIs, a collection usually corresponds to one API endpoint. For example, Stripe has a stripe:subscription collection located at /v1/subscriptions and Salesforce has a salesforce:contact collection that you can access at /services/data/v51.0/sobjects/Contact.

Sequin uses unique IDs to refer to each collection. The ID is a combination of the API provider’s name and the collection name. Some examples:

• Stripe subscriptions -> stripe:subscription
• Salesforce contacts -> salesforce:contact
• HubSpot pipelines -> hubspot:pipeline
• Shopify products -> shopify:product
• GitHub repositories -> github:repository

For API providers that have custom collections, like Salesforce and HubSpot, Sequin assigns the collection a unique ID. For example:

• Salesforce Custom Object MyObject__c -> salesforce:szMO52XHsHps1DZRrmdvKbch

• HubSpot Custom Object MyObject -> hubspot:DR8V5vaukyB2uJZW085AjILH

​

Backfills

When you first setup your sync, Sequin will backfill historical data from the API. You can configure how far back in time Sequin will backfill. Sequin will also kickoff a backfill process whenever you add a new collection to your sync.

The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you’ve allocated for Sequin.

Backfills involving millions of records can take several hours. You can view the status of your backfill at any time in the Sequin console or via the Management API.

​

Frequency

Sequin syncs data from APIs at a frequency that you specify. You can sync data in real-time (e.g. continuous syncs) or on a schedule (e.g. 4h syncs).

For continuous syncs, the latency between the API change and the record appearing in your stream is usually a couple seconds - but this depends on the API and the rate limit you’ve set for Sequin.

​

Rate limits

Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of an API’s total quota. The amount of your quota we use is configurable in the Sequin console or via the Management API.

The time it takes Sequin to detect changes in the API and push to your streams is called the “max lag time” of any given change. Max lag time is subject to a few factors:

• the API you’re syncing from
• the collections you’re syncing from that API
• the rate limit you set Sequin to consume
• whether the collection supports webhooks or other fast means of detecting changes

​

Consumers

You can use Sequin’s serverless message bus to consume from your record stream. The Consume APIs are a set of HTTP interfaces to your streams. You can paginate through all the records in the stream, ack and nack records, and replay records.

Learn more about the Consume API.

​

Mutations

Sequin makes it easy to write changes back to upstream APIs. Most collections that Sequin syncs are mutable. To create a record in that collection or mutate an existing one, you can use Sequin’s Mutation API.

Learn more about mutations.

​

Management API

You can use Sequin’s Management API to develop applications that create and manage Sequin syncs. You can automate everything that you can do in the Sequin console. This includes creating, updating, and deleting syncs on behalf of your users using their provided credentials, as well as turning on and off consumers.

Learn more about the Management API.