GitHub
Create an event stream from GitHub’s API to rapidly integrate with GitHub using your database, Kafka stream, and more.
GitHub is where developers collaborate to ship code. GitHub’s API allows you to customize and automate your development workflows.
Sequin syncs GitHub’s API to an ordered event stream you can use to easily integrate with any GitHub account.
You can build directly on the Sequin stream or forward data to your database, Kafka or any other system you use to store and process data. We’ll handle the authentication, rate limit, and consistency for you.
You’ll create, update, and delete GitHub objects (e.g. repositories, pull requests, and issues) using Sequin’s Record Mutation API. Sequin simultaneously applies changes to GitHub and the event stream.
All this means an easy-to-use, unified interface to GitHub.
Here is how our GitHub sync works:
Sync process
Authentication
To connect Sequin to your GitHub data, you’ll install Sequin’s GitHub app. In the Sequin Console, you’ll click Install the Sequin GitHub app. You’ll then login to GitHub and follow the prompts to install the app to either a GitHub user (i.e. your user) or to a GitHub organization:
During installation, GitHub will prompt you which repositories to share with Sequin. Sequin will sync whichever repositories you choose as well as the related objects (e.g. Pull Requests and Commits) for those objects.
You can also authenticate with GitHub by importing a credential via the Management API. This means users can go through your OAuth flow with your GitHub app. After your user has authorized your GitHub app, you can import their GitHub account into Sequin:
curl --request POST \
--url https://api.sequin.io/v1/credentials \
--header 'Content-Type: application/json' \
--data '{
"properties": {
"account_login": "user_name",
"account_type": "user",
"installation_id": 46000086,
"kind": "github",
"name": "Choam Spice"
},
"metadata": { "custom_property": 42 }
}'
Backfill
When you first connect to GitHub, Sequin will extract all the data for the collections you want to work with. This is called backfilling.
To backfill GitHub, Sequin will paginate through each collection starting with the oldest records and working forwards in time. This ensures that you get all the data from GitHub into your stream.
For larger GitHub accounts, this process can take a few hours.
Real-time
After the initial backfill, we’ll rely on GitHub’s webhooks to monitor changes. We backstop the webhooks with an occasional polling process to ensure we don’t miss any events. This means changes on GitHub should propagate to your event stream in just a few seconds.
Pacing
If you’re using Sequin’s OAuth app, Sequin’s GitHub application is given a dedicated API rate limit quota. This means that Sequin’s requests will not affect any other apps or services you have connected to your account.
If you’ve imported your OAuth app into Sequin, you can set rate limit parameters on a per-sync or global basis.
Schema
Collections
At the moment, we support the following GitHub objects:
- Repositories
- Pull Requests
- Commits
- Issues
- Releases
If there is an object you need that isn’t listed, just send us a message and we’ll add it for you in a day or two.
Fields
Sequin syncs all fields from each collection. The collections are structured as per the GitHub API.
Streams
Your GitHub sync will populate data into the record and event streams. Here’s how Sequin structures GitHub objects in the stream:
Record stream
The record stream contains the current state of an object in the GitHub API. For instance, you’ll see the current state of a pull_request as it exists in GitHub. Below is the shape of a record in the GitHub pull_request collection:
{
"collection_id": "github:pull_request",
"deleted": false,
"id": "52077657-9a6e-4aca-ab12-7f6ff9917fcd",
"inserted_at": "2024-02-06T22:27:18.399173Z",
"payload": {
"activeLockReason": null,
"additions": 213,
"authorAssociation": "OWNER",
"baseRefName": "main",
"baseRefOid": "15611438682f9b0a6eae8b009b26eabd58e7c2c7",
"body": "",
"changedFiles": 4,
"closedAt": "2023-12-12T22:20:25Z",
"createdAt": "2023-12-09T19:08:00Z",
"databaseId": 1637354212,
"deletions": 57,
"headRefName": "Phase-2",
"headRefOid": "8090d0c0ec3a0c5b659bcdf113e4f9c353cf28da",
"id": "PR_kwDOK3v93s5hmArk",
"isDraft": false,
"locked": false,
"maintainerCanModify": false,
"mergeable": "UNKNOWN",
"merged": true,
"mergedAt": "2023-12-12T22:20:25Z",
"number": 1,
"repositoryId": 729546206,
"state": "MERGED",
"title": "Phase 2",
"totalCommentsCount": 22,
"updatedAt": "2023-12-12T22:20:25Z",
"url": "https://github.com/thisisgoldman/syncer/pull/1"
},
"payload_hash": "IkKe08wUZOVYCJDJ4SgtMw==",
"resource_id": "cef62ddb-0f19-4e07-8a1f-f07376aabed1",
"updated_at": "2024-02-06T22:27:18.399173Z",
"upsert_opts": null,
"upstream_id": "1637354212",
"upstream_updated_at": "2023-12-12T22:20:25.000000Z"
}
Event stream
The event stream contains the changes to an object in the GitHub API. As an example, if the title field of a pull_request object is updated, the event stream will contain the following event:
{
"changes": {
"body": "",
"state": "MERGED",
"title": "Phase 2",
"totalCommentsCount": 22,
"updatedAt": "2023-12-12T22:20:25Z"
},
"collection_id": "github:pull_request",
"created_at": "2024-02-06T22:34:09.890577Z",
"data": {
"activeLockReason": null,
"additions": 213,
"authorAssociation": "OWNER",
"baseRefName": "main",
"baseRefOid": "15611438682f9b0a6eae8b009b26eabd58e7c2c7",
"body": null,
"changedFiles": 4,
"closedAt": "2023-12-12T22:20:25Z",
"createdAt": "2023-12-09T19:08:00Z",
"databaseId": 1637354212,
"deletions": 57,
"headRefName": "Phase-2",
"headRefOid": "8090d0c0ec3a0c5b659bcdf113e4f9c353cf28da",
"id": "PR_kwDOK3v93s5hmArk",
"isDraft": false,
"locked": false,
"maintainerCanModify": false,
"mergeable": "UNKNOWN",
"merged": true,
"mergedAt": "2023-12-12T22:20:25Z",
"number": 1,
"repositoryId": 729546206,
"state": "CLOSED",
"title": "Phase 2 - UPDATED NAME",
"totalCommentsCount": 1,
"updatedAt": "2024-02-06T22:34:05Z",
"url": "https://github.com/thisisgoldman/syncer/pull/1"
},
"event_type": "updated",
"id": "1612a932-5fe0-44ed-9dfc-dbb80505bd2d",
"resource_id": "cef62ddb-0f19-4e07-8a1f-f07376aabed1",
"upstream_id": "1637354212",
"upstream_updated_at": "2024-02-06T22:34:05.000000Z"
}
Consumers
You can build directly on Sequin’s stream using the stream APIs. Or forward data to your database, Kafka or any other system you use to store and process data.
GitHub syncs work with all the consumers Sequin supports.
Mutations
When you need to create, update, or delete a GitHub object, you can use Sequin’s Record Mutation API. For instance, if you want to create an issue in github, you’ll use the create mutation:
curl -X POST \
https://api.sequin.io/v1/mutations/run/create \
-H 'content-type: application/json' \
-d '{
"sync_id": "my-sync-id",
"collection_id": "github.issue",
"data": [
{
"body": "This is a new issue",
"repositoryId": 247145224,
"state": "OPEN",
"title": "[Challenge] Replace deep object accessors with getter functions",
}
]
}'
Sequin will handle the authentication, rate limit, and consistency for the request by simultaneously applying changes to GitHub and the event stream. As an added consistency guarantee, you can also specify which consumers you want to await before returning the mutation response.
For instance, you can ensure that a new issue has been added to your database before the the API returns a 200 OK.
Development
For development, you can setup a test repository in your personal GitHub account or within your GitHub organization. When you install the Sequin App, you can scope the app to include just the test repository.
Once you’ve tested your integration, you can use the Management API to easily replicate your sync to your production account.
Was this page helpful?