When you connect a source like Salesforce or HubSpot to your database, Sequin first backfills all API data to your database. Then, Sequin provisions a continuous sync.

Sequin’s sync is always running and checking the source API for changes.

​
Backfills

When you first setup your sync, Sequin will backfill your tables with all historical data from the API. Sequin will also kickoff a backfill process whenever you add a new table or column or change a column type (see configuring your schema).

The time it takes Sequin to run a backfill depends on (1) how much historical data you have in the API and (2) what rate limit you’ve allocated for Sequin.

Backfills involving millions of records can take several hours. You can view the status of your backfill at any time on the Sequin console:

Additionally, when your initial backfill completes, Sequin will email you letting you know that your database is ready to go.

​
Speed

Every API has a rate limit or quota that specifies how many requests you can make to the API in a given interval. Sequin retrieves data from APIs at a frequency that respects this rate limit. By default, we use a fraction of your total quota. The amount of your quota we use is configurable in the Sequin console.

The speed of your sync from the API to your database is the “max lag time” of any given change. That is: when your data changes in the API, what is the maximum amount of time you’d expect to wait before seeing that change reflected in your database?

Max lag time is subject to a few factors:

  • the API you’re syncing with
  • the collections you’re syncing from that API
  • the rate limit you set Sequin to consume

For most syncs and most collections, the max lag time is rarely above one minute. For example, by default we make one request every three seconds for Salesforce syncs. Assume you kept this default and you’re syncing five Salesforce collections. It will take Sequin about 15 seconds to check each collection endpoint for changes. So, the max lag time for this hypothetical sync is 15 seconds.

There are two notable exceptions.

The first is Airtable. For Airtable, the lag time of the sync depends on (1) how big your base is and (2) how much of your base’s API quota we’re using. The Sequin console shows the lag time of your Airtable database. By default, we use 3 out of 5 of your allotted requests per second, but you can configure this in your console. (More about how our sync works for Airtable on our blog.)

Similarly, for HubSpot, all objects and custom objects sync near-immediately. The exception is HubSpot associations. The max lag time for associations is roughly dependent on the number of associations you’re syncing.

Learn more about rate limits.

​
Consistency

Sequin’s sync relies primarily on polling the API. We use this strategy because it gives us the strongest consistency guarantee.

We’re rolling out support for webhooks for the fastest propagation where supported. But all syncs are built on polling to ensure we never miss a create, update, or delete.

Writing dataRate limits