A rundown of Sequin from top to bottom:

  • Every API produces one or more collections that Sequin can consume.
  • To add an API’s data to Sequin, you select which collections you want to sync.
  • Sequin provisions a sync cluster which first extracts all data from the API, then monitors it for changes.
  • As changes occur, Sequin propagates those changes to your account’s record stream and event stream.
  • You can consume these streams directly via Sequin’s Stream APIs (HTTP).
  • Or you can use one of Sequin’s consumers to forward the data to a destination like Postgres or Kafka.
  • And to write back to the API, you can use the Collection Mutation API.

All this and more can be managed in the console or via the Management API.

Collections

A collection is a class of records in the API. In RESTful APIs, a collection usually corresponds to one API endpoint. For example, Stripe has a subscription collection located at /v1/subscriptions and Salesforce has a contact collection that you can access at /services/data/v51.0/sobjects/Contact.

Sequin uses unique IDs to refer to each collection. The ID is a combination of the API provider’s name and the collection name. Some examples:

  • Stripe subscriptions -> stripe:subscription
  • Salesforce contacts -> salesforce:contact
  • HubSpot pipelines -> hubspot:pipeline
  • Shopify products -> shopify:product
  • GitHub repositories -> github:repository

Syncs

To generate a stream of API data, you create a sync. When you create a sync, Sequin first extracts all data from the API. Then, Sequin monitors the API for changes and emits events when changes occur.

Data consistency is a huge priority for Sequin, so the backbone of most syncs is polling. Sequin augments polling with other strategies (like listening to API webhooks) where applicable to offer the fastest possible sync.

Backfills

When you first setup your sync, Sequin will backfill your event stream with all historical data from the API. Sequin will also kickoff a backfill process whenever you add a new collection to your sync.

The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you’ve allocated for Sequin.

Backfills involving millions of records can take several hours. You can view the status of your backfill at any time on the Sequin console:

Speed

Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of an API’s total quota. The amount of your quota we use is configurable in the Sequin console.

The time it takes Sequin to detect changes in the API and push to your streams is called the “max lag time” of any given change. Max lag time is subject to a few factors:

  • the API you’re syncing from
  • the collections you’re syncing from that API
  • the rate limit you set Sequin to consume
  • whether the collection supports webhooks or other fast means of detecting changes

For most syncs and most collections, the max lag time is rarely above one minute.

Streams

Sequin provisions two streams for your account. All data from your syncs is pushed to these two streams.

The record stream contains a stream of all records across syncs, while the event stream contains all the recent create, update, and delete events across syncs.

Record stream

The record stream is a stream of all API records across all APIs you are syncing. A record contains the current state of an object in the API.

The record stream contains each record only once:

  • When Sequin detects a record insert in the API, it appends the record to the stream.
  • When Sequin detects a record update in the API, it pops the old record and appends the updated record to the stream.
  • When Sequin detects a record delete in the API, it pops the old record and appends the updated record to the stream with the deleted property set to true.

You’ll want to use records whenever a consumer just needs the most recent state of records. For example, a sync to a data store like Redis or Postgres.

Records

Records have the following shape:

Record{
  sync_id: string;
  collection_id: string;
  upstream_id: string;
  upstream_updated_at: string;
  data: object;
  deleted: boolean;
}

Here’s an example of a Record:

{
    "sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
    "collection_id" : "salesforce:task",
    "upstream_id": "0018b000021BldZAAS",
    "upstream_updated_at": "2024-11-10T18:39:00.070453Z",
    "data": {
      "activity_date": "2023-09-12",
      "description" : "task description [ … ] ",
      // …
    },
    "deleted": false
 }

A Record is a JSON object that contains the following properties:

  • sync_id: The ID of the sync that generated this event.
  • collection_id: The ID of the collection that generated this event (e.g. stripe:subscription or salesforce:contact).
  • upstream_id: The ID of the record in the API.
  • upstream_updated_at: The time the record was last updated in the API, in ISO 8601.
  • data: All the fields of the API object.
  • deleted: Whether the record has been deleted in the API.

Sequin may add fields to records at any time. If you’re using a JSON validator to parse records, you should allow for unknown fields.

Event stream

An event represents a single change to a record. Every time Sequin detects that a record was changed in the API, it creates an event and appends it to the event stream. The event stream is a log of all changes to all records across all syncs that happened in the last 7 days.

You’ll want to use the event stream when you need to know how a record has changed over time. For example, you might want to trigger a side effect when a specific field has changed or kick off a workflow when a record is created.

Both events and records share the same data property, which contains the fields of the API object. Here’s an example of an inserted event:

{
    "sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
    "collection_id" : "salesforce:task",
    "upstream_id": "0018b000021BldZAAS",
    "upstream_updated_at": "2024-11-10T18:39:00.070453Z",
    "data": {
      "activity_date": "2023-09-12",
      "description" : "task description [ … ] ",
      // …
    },
    "event": "inserted",
    "event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
    "event_inserted_at": "2024-11-10T18:39:00.070453Z"
 }

For any given record, Sequin’s event stream is strictly ordered. So, if a record was updated twice in the API, the event stream will contain two updated events in the order they occurred.

Note that while Sequin will always emit an event for the final state of a record after an update, Sequin can “miss” ephemeral state changes. For example, if a Salesforce opportunity switches from Prospecting to Closed Won but then quickly back to Prospecting before Sequin picks it up, Sequin may not emit an event for the Closed Won state.

Sequin’s ability to detect ephemeral state changes depends on the API. For example, it’s easier to pick up ephemeral state changes in APIs that support webhooks.

We think this is rarely a problem, but if you have a use case where this is an issue, please let us know.

Events

Event{
  sync_id: string;
  collection_id: string;
  upstream_id: string;
  upstream_updated_at: string;
  data: object;
  event: "inserted" | "updated" | "deleted";
  event_id: string;
  event_inserted_at: string;
  // Only present on "updated" events
  previous_data?: object;
}

The event is a JSON object that contains the following properties:

  • sync_id: The ID of the sync that generated this event.
  • collection_id: The ID of the collection that generated this event (e.g. stripe:subscription or salesforce:contact).
  • upstream_id: The ID of the record in the API.
  • upstream_updated_at: The time the record was last updated in the API, in ISO 8601.
  • data: All the fields of the API object. For deleted events, this is the entire record before it was deleted. This matches the data attribute found on records.
  • event: Either created, updated, or deleted.
  • event_id: A Sequin-generated ID for this event. This ID is universally unique.
  • event_inserted_at: The time the event was created (and inserted into the event stream), in ISO 8601.
  • previous_data (updated events only): contains the prior values of all columns that were changed. (See below.)

Here’s an example of an created event:

{
    "sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
    "collection_id" : "salesforce:task",
    "upstream_id": "0018b000021BldZAAS",
    "upstream_updated_at": "2024-11-10T18:39:00.070453Z",
    "data": {
      "activity_date": "2023-09-12",
      "description" : "task description [ … ] ",
      // …
    },
    "event": "created",
    "event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
    "event_inserted_at": "2024-11-10T18:39:00.070453Z"
 }

And an example of a deleted event:

{
    "sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
    "collection_id" : "salesforce:task",
    "upstream_id": "0018b000021BldZAAS",
    "upstream_updated_at": "2024-11-10T18:39:00.070453Z",
    "data": {
      "activity_date": "2023-09-12",
      "description" : "task description [ … ] ",
      // …
    },
    "event": "deleted",
    "event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
    "event_inserted_at": "2024-11-10T18:39:00.070453Z"
 }

And finally an updated event:

{
    "sync_id": "70a313f8-7c94-43d4-8d0c-5659a52de5c4",
    "collection_id" : "salesforce:task",
    "upstream_id": "0018b000021BldZAAS",
    "upstream_updated_at": "2024-11-10T18:39:00.070453Z",
    "data": {
      "activity_date": "2023-09-12",
      "description" : "Some description of the task [...]",
      "completed_at": "2023-09-12T18:39:00.070453Z",
      // …
    },
    "previous_data": {
      "activity_date": "2023-08-30",
      "description": "Some description of the task [...]",
      "completed_at": null
    },
    "event": "updated",
    "event_id":"079013db-8b17-44cd-8528-f5e68fc61333",
    "event_inserted_at": "2024-11-10T18:39:00.070453Z"
 }

Unlike created and deleted events, updated events have a previous_data property. previous_data contains the prior values of all columns that were changed. In the example above, activity_date was changed from "2023-08-30" to "2023-09-12". Likewise, completed_at was previously null but now has a value. Because description was not changed, it does not appear in the previous_data object.

Stream APIs

You can use Sequin’s Stream APIs to consume from your record or event streams. The Stream APIs are a set of HTTP interfaces to your streams. You can paginate through all records across all your syncs, or you can filter to just the collections or events you need.

Learn more about the Stream APIs.

Consumers

Sequin offers a number of consumers that you can use to forward your data to a target. For example, you can use the Postgres consumer to forward your data to a Postgres database or the Kafka consumer to forward your data to Kafka.

Consumers make it easy to build on APIs using the data platforms you already know and use.

Learn more about consumers.

Targets

A target is infrastructure in your application environment, such as a Postgres database or a Kafka cluster, that you register with Sequin.

Sequin’s consumers forward data from either the event stream or the record stream to a target.

Mutations

Sequin makes it easy to write changes back to upstream APIs. Most collections that Sequin syncs are mutable. To create a record in that collection or mutate an existing one, you can use Sequin’s Collection Mutation API.

Learn more about mutations.

Management API

You can use Sequin’s Management API to develop applications that create and manage Sequin syncs. You can automate everything that you can do in the Sequin console. This includes creating, updating, and deleting syncs on behalf of your users using their provided credentials, as well as turning on and off consumers.

Learn more about the Management API.