Github checks API vs check runs vs check suites - github-api

Question

I want to understand Github Checks API so that I can use them to retrieve data. On following Github documentation https://docs.github.com/en/rest/guides/getting-started-with-the-checks-api I am able to derive that check runs are associated with the sha of the change and at each commit on branch check suite is created. Checks API helps in getting all this information. But I want more clarity on three of them in terms of differences. Can anyone please explain these three terms using simple example and terms?

Answer 1

So, first of all, the GitHub Commit Statuses API is separate from the GitHub
Checks API (includes suites & runs), so let's look at them individually first then I'll explain the differences.
Before we get into it, I want to differentiate a PR Check from a Check Run, to avoid confusion. A PR Check describes the current state (trying not to say status here 😉) of a given job or task running in CI or elsewhere on a specific PR commit. These can be created via either a Commit Status or a Check Run. All the items in the pink box below are PR Checks, notice the Hide all checks button.
GitHub statuses - API
I see this as the simple, all-purpose API for reporting PR Checks for a given commit. It's easy and doesn't require jumping through hoops to just display a simple result of a PR Check for a given commit. This API also came before the check API so it's a little less powerful.
Pros
Simple and easy API
Simple relation with context as the identifier.
Can have a fully customizable text description on the PR Checks UI.
You can create statuses as a user with a Personal Access Token (aka PAT) without needing to create a GitHub App, though does work with GitHub Apps too!
Cons
Limited status options, only allow error, failure, pending, success, no conclusion subset option to define completed jobs like Check Runs.
No concept of job timing or duration.
No grouping of statuses, like Check Runs are grouped into Check Suites by their GitHub App
No annotations
No detailed output logs. This is not too important as you could just link to the URL where the actual PR Check was run such as in CircleCI, Jenkins, etc. But the user is not always authorized to view these runs so the output could be helpful for open source repos that have non-public CI.
GitHub Checks - API
The Checks API is the latest and greatest tool for displaying task results on commits, which can essentially do everything the Commit Statuses API can do and more. Check Runs belong to one Check Suite, one Check Suite can have many Check Runs. You can only have one Check suite per commit (i.e. head_sha) per GitHub App, attempting to create another for a given App will just return the previously created Check Suite. Thus, a new Check Run is automatically assigned to the Check Suite based on the authenticated GitHub App, you cannot manually assign Run to Suites.
Contrary to statuses, Check runs are identified by an auto-generated check_run_id and not a context string.
I haven't touched too much on the Check Suites API because they are really just a grouping of Check Runs which is pretty self-explanatory and they don't affect any of the PR Checks UI, only the grouping of Check Runs in the checks tab. One thing to note is that by default you can create a Check Run without having to first create a Check suite and GitHub will just create a new Check Suite for you.
Pros
Greater granularity of status/conclusion for a run.
A lot of power to display the result of a PR Check.
Can provide run context via output summary in Markdown.
Can create annotations for specific lines of code to add information about the analysis performed, like linting error for a given line of code. These will show up in the PR files tab UI similar to PR code comments as well as in the PR Checks tab with any other Check Run output.
Has time awareness to report on durations automatically with little effort.
Cons
A little more complicated API and relationships to manage.
Part of the description in the PR UI is auto-generated based on the status conclusion and duration of the check run task.
Cannot be created via a user PAT, must be created from an authenticated GitHub App. Read access to this API does not require authenticating as a GitHub App.
One edge case you likely won't come across but is good to know: If you are creating a new Check Run with the exact same name of an existing check run under the same authenticated app. The resulting behavior is a little strange but doing this will create the new Check Run under the same name, but will not delete the existing Check Run. However, the Check Suite will not see or link to the existing Check Run, even in the PR UI. BUT!! If you change the name of either such that the runs now have unique names, it will be linked up again. It seems GitHub just does a sort by date and then filters by unique names when looking up Check Runs in a Check Suite. This does not apply with identical names from different authenticated apps.
Comparisons
Below is a mapping of sorts to compare similar options between the Commit Statuses API and the Check Runs API. They are not exactly 1:1 but similar.
Commit Status
Check Run
Option Desciption
sha
head_sha
These are equivalent with the minor exception that the Commit Status is linked to the sha directly, whereas the head_sha is linked to the Check Suite that the Check Run belongs to.
context
name
The context is used as an identifier but both define the title of the PR Check in the PR UI. Because Check Runs are tracked by the check_run_id the name option may change without creating a new Check Run. This is not the case for context, changing the context will always create a new Commit Status. You may not have duplicate names for Check Runs created with the same GitHub App, see note above.
context*
external_id
The external_id is meant for keeping track of Check Runs without having to store ids or always keep the name constant. This is only somewhat similar to the context option but only for the purpose of identifying the Check Run.
description
output.title
The main difference here is that description gives you the full space to work with, where output.title will be displayed after the auto-generated status string.
target_url
details_url
These are somewhat equivalent, the first difference is that target_url will not show unless defined whereas details_url will default to the check run URL in the PR UI. The other difference is the Check Runs Details button on the PR Checks UI will always link the the Checks page which will present a link to the details_url if defined.
state
status
These are very similar but have slightly different allowed values but effectively appear the same in the PR UI.
N/A
conclusion
This just shows the increased power of the PR Checks API where you have more granular control over the PR Check status, though not a big difference in the UI, see example variations below.
N/A
started_at
No comparable option for Commit Statuses.
N/A
completed_at
No comparable option for Commit Statuses.
N/A
actions
No comparable option for Commit Statuses.
N/A
output
No comparable option for Commit Statuses.
N/A
output.annotations
No comparable option for Commit Statuses.
* Only somewhat similar, not a direct equivalent.
PR Checks UI Component mappings
I've taken a simple PR Check and highlighted the differences between the elements of the UI between the Commit Status API and the Check Runs API. Very similar but slight differences.
Below are example variations to relate the options to their impact on the
PR Checks UI.
Commit status variations
Check Run variations
If a check run is in an incomplete state for more than 14 days, then the Check Run's conclusion becomes stale. Only GitHub can mark Check Runs as stale.
Checks Tab in PR UI
The Checks Tab in the PR UI will display all created Check Suites and their child Check Runs and allows you to set the output option to control the content of this page. This page would also show images and annotations if you defined those too.
One final note: You cannot DELETE any Commit Status, Check Run or Check Suite once created, only updating is allowed. With these statuses being so ephemeral this is not that big of a deal but in case you were wondering.
Update (6/21/2022)
I found two more quirks to explain. First, the details_url for Check Runs does not set the value of the Details button, instead the Details button actually always redirects to the Checks page which has a link to the details_url if defined.
Second, once a Check Run status is set to 'completed', there is not way to un-complete the Check run. But you can get around this by creating a new Check Run with the same name and set that status to something other than 'completed'. See this edge case with duplicate-named Check Runs explained above in detail.
Update (6/29/2022)
As mentioned above, the Check Runs API keeps track of the duration of the run. It is important to note that the started_at time is set only once whenever a check run is created, regardless of the defined status. For example, if I trigger a CI build and set all jobs status to queued, the jobs run duration will be inaccurate. A good way to fix this is to always set the started_at time whenever you set the status to in_progress. Something as simple as const started_at = new Date().toISOString() (javascript) will do the trick.