Fetch commit of last successful WebKit build (default: GTK-Linux-64-bit-Release-Ubuntu-LTS-Build)

last-successful-build-to-git-commit.sh

#!/usr/bin/env bash

# set -x

# Return commit of last successful WebKit build (default: GTK-Linux-64-bit-Release-Ubuntu-LTS-Build).

# Changes:
#   2022-11-16: Reworked script using changes suggested by asutherland to fetch commit number of last successful build directly from build.webkit.org.

BUILDER_NAME=${1:-GTK-Linux-64-bit-Release-Ubuntu-LTS-Build}
if [[ -z $WEBKIT_CHECKOUT_DIR ]]; then
    WEBKIT_CHECKOUT_DIR=${2:-$(realpath $(dirname "$0"))}
fi

usage() {
    local exit_code="$1"
    local program_name=$(basename "$0")

    echo -e "Usage: $program_name BOT-NAME"
    echo -e "Returns commit of last successful build for BOT-NAME"

    exit $exit_code
}

fatal() {
    echo -n "Fatal: "
    echo $@
    exit 1
}

get_last_successful_build() {
    local builder_name="$1"

    curl "https://build.webkit.org/api/v2/builders/$builder_name/builds?order=-number&limit=1&complete=true&state_string=build%20successful" 2>/dev/null
}

query_build_number() {
    local build_info="$1"

    jq -Mr ".builds[0].number" <<< "$build_info"
}

query_builder_id() {
    local build_info="$1"

    jq -Mr ".builds[0].builderid" <<< "$build_info"
}

get_build_changes() {
    local builder_id="$1" build_number="$2"

    curl "https://build.webkit.org/api/v2/builders/$builder_id/builds/$build_number/changes" 2>/dev/null
}

query_revision() {
    local changes_info="$1"

    jq -Mr ".changes[0].revision" <<< "$changes_info"
}

# Main.

if [[ $# -gt 1 ]]; then
    usage 1
fi

if [[ "$1" == "-h" || "$1" == "--help" ]]; then
    usage 0
fi

build_info=$(get_last_successful_build "$BUILDER_NAME")
if [[ $? -ne 0 ]]; then
    fatal "could not fetch build from $BUILDER_NAME"
fi

build_number=$(query_build_number "$build_info")
builder_id=$(query_builder_id "$build_info")
changes_info=$(get_build_changes "$builder_id" "$build_number")
revision=$(query_revision "$changes_info")

echo "$revision"

This seems like a fantastic tool for searchfox to be able to use to improve the wubkat experience, thank you so much for creating and link this! I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1786930 on hooking this up but ran into an issue when experimenting manually.

It seems like there may be some kind of mirroring lag when using a remote of https://github.com/WebKit/WebKit.git. Right now the script is looking for 253733@main but the main branch as fetched from the above is only up to Canonical link: https://commits.webkit.org/253719@main. Is there a better git mirror to use? Should searchfox try and pull directly from subversion using a helper?

I think the reason why the canonical id could be found in the local git repository was that likely the local repository was not in the main branch or was not up to date (or both). I've updated the script to fetch and move to the main branch before searching for the canonical id.

Since two months ago or so, the Github hosted WebKit repository is now the official WebKit repository. The former WebKit hosted Git repository (https://git.webkit.org/git/WebKit.git) as well as the until now official SVN repository are not getting new commits anymore. They're both stuck on the last commit before making the switch (r295779). There was an email in webkit-dev mailing list announcing this change.

I finally put this script into place yesterday after a build failure, but I think there may be a bug (or I'm using it wrong), as today's run chose d257ea20c5996ada3c009a6a2f4a639f92c0e2ca AKA 254947@main which corresponded to the first failed build at https://build.webkit.org/#/builders/68 which had a status of "failed compiled (failure)". Note that timing-wise this check ran at Wed Sep 28 09:17:18 UTC 2022 so that build was also the most recent build, so I don't think we should necessarily assume an off-by-one error.

I'm going to eat some food and then look at the script since it's of course easier to iterate on a script like this in a situation like this where the data is perfect to test with! :)

So I was both using it wrong and the initial get_last_successful_build_number query of https://build.webkit.org/api/v2/builders/GTK-Linux-64-bit-Release-Ubuntu-LTS-Build/builds?order=-number&limit=1&complete=true seems to be not expressing the constraint against failure, as the current payload I get is:

{
  "builds": [
    {
      "builderid": 68,
      "buildid": 1898239,
      "buildrequestid": 1358658,
      "complete": true,
      "complete_at": 1664383280,
      "masterid": 34,
      "number": 17394,
      "properties": {},
      "results": 2,
      "started_at": 1664383105,
      "state_string": "failed compiled (failure)",
      "workerid": 397
    }
  ],
  "meta": {
    "total": 1
  }
}

So adding &state_string=build%20successful to the query to get https://build.webkit.org/api/v2/builders/GTK-Linux-64-bit-Release-Ubuntu-LTS-Build/builds?order=-number&limit=1&complete=true&state_string=build%20successful does get us what we want:

{
  "builds": [
    {
      "builderid": 68,
      "buildid": 1897560,
      "buildrequestid": 1357425,
      "complete": true,
      "complete_at": 1664355452,
      "masterid": 34,
      "number": 17378,
      "properties": {},
      "results": 0,
      "started_at": 1664352694,
      "state_string": "build successful",
      "workerid": 397
    }
  ],
  "meta": {
    "total": 1
  }
}

and from that:

{
  "meta": {},
  "steps": [
    {
      "buildid": 1897560,
      "complete": true,
      "complete_at": 1664352701,
      "hidden": false,
      "name": "clean-and-update-working-directory",
      "number": 3,
      "results": 0,
      "started_at": 1664352696,
      "state_string": "Cleaned and updated working directory",
      "stepid": 17970248,
      "urls": [
        {
          "name": "Updated to 254946@main",
          "url": "https://commits.webkit.org/254946@main"
        }
      ]
    }
  ]
}

I found out about the extra arg's identifier via the REST API dynamic docs thing at https://build.webkit.org/#/about where I also found there's a way to get the "changes" based on the builderid and number, and that gets us: https://build.webkit.org/api/v2/builders/68/builds/17378/changes

{
  "changes": [
    {
      "author": "Lauro Moura <lmoura@igalia.com>",
      "branch": "main",
      "category": null,
      "changeid": 28430,
      "codebase": "",
      "comments": "REGRESSION(254269@main) [GTK] Fix path to InspectorTestServer in API test\nhttps://bugs.webkit.org/show_bug.cgi?id=245763\n\nReviewed by Adrian Perez de Castro.\n\nThe test failed to spawn the InspectorTestServer, ending up raising\nSIGTERM and exiting the run-gtk-tests script.\n\n* Tools/TestWebKitAPI/Tests/WebKitGtk/TestInspectorServer.cpp:\n\nCanonical link: https://commits.webkit.org/254946@main",
      "committer": "Adrian Perez de Castro <aperez@igalia.com>",
      "files": [
        "Tools/TestWebKitAPI/Tests/WebKitGtk/TestInspectorServer.cpp"
      ],
      "parent_changeids": [
        28429
      ],
      "project": "WebKit/WebKit",
      "properties": {
        "event": [
          "push",
          "Change"
        ],
        "github_distinct": [
          true,
          "Change"
        ]
      },
      "repository": "https://github.com/WebKit/WebKit",
      "revision": "df3d7bf46a5192407a8a7d69d9de9db3918f15d6",
      "revlink": "https://github.com/WebKit/WebKit/commit/df3d7bf46a5192407a8a7d69d9de9db3918f15d6",
      "sourcestamp": {
        "branch": "main",
        "codebase": "",
        "created_at": 1664351560,
        "patch": null,
        "project": "WebKit/WebKit",
        "repository": "https://github.com/WebKit/WebKit",
        "revision": "df3d7bf46a5192407a8a7d69d9de9db3918f15d6",
        "ssid": 28497
      },
      "when_timestamp": 1664351477
    },
    {
      "author": "Kimmo Kinnunen <kkinnunen@apple.com>",
      "branch": "main",
      "category": null,
      "changeid": 28429,
      "codebase": "",
      "comments": "RunLoop lacks a create operation\nhttps://bugs.webkit.org/show_bug.cgi?id=245673\nrdar://problem/100411103\n\nReviewed by Antti Koivisto and \u017dan Dober\u0161ek.\n\nMove existing logic to RunLoop::create(). This makes future new invocations\nless error prone.\n\n* Source/WTF/wtf/RunLoop.cpp:\n(WTF::RunLoop::create):\n* Source/WTF/wtf/RunLoop.h:\n* Source/WTF/wtf/generic/WorkQueueGeneric.cpp:\n(WTF::WorkQueueBase::platformInitialize):\n* Source/WebCore/page/scrolling/ScrollingThread.cpp:\n(WebCore::ScrollingThread::ScrollingThread):\n(WebCore::ScrollingThread::isCurrentThread):\n* Source/WebCore/page/scrolling/ScrollingThread.h:\n(WebCore::ScrollingThread::runLoop):\n(): Deleted.\n* Source/WebKit/Shared/CoordinatedGraphics/threadedcompositor/CompositingRunLoop.cpp:\n(WebKit::CompositingRunLoop::CompositingRunLoop):\n(WebKit::createRunLoop): Deleted.\n* Tools/TestWebKitAPI/Tests/WTF/RunLoop.cpp:\n(TestWebKitAPI::TEST):\n\nCanonical link: https://commits.webkit.org/254945@main",
      "committer": "Kimmo Kinnunen <kkinnunen@apple.com>",
      "files": [
        "Tools/TestWebKitAPI/Tests/WTF/RunLoop.cpp",
        "Source/WebKit/Shared/CoordinatedGraphics/threadedcompositor/CompositingRunLoop.h",
        "Source/WebKit/Shared/CoordinatedGraphics/threadedcompositor/CompositingRunLoop.cpp",
        "Source/WebCore/page/scrolling/ScrollingThread.h",
        "Source/WebCore/page/scrolling/ScrollingThread.cpp",
        "Source/WTF/wtf/generic/WorkQueueGeneric.cpp",
        "Source/WTF/wtf/RunLoop.h",
        "Source/WTF/wtf/RunLoop.cpp"
      ],
      "parent_changeids": [
        28428
      ],
      "project": "WebKit/WebKit",
      "properties": {
        "event": [
          "push",
          "Change"
        ],
        "github_distinct": [
          true,
          "Change"
        ]
      },
      "repository": "https://github.com/WebKit/WebKit",
      "revision": "2b6f2e9cb51ed9c4a01ce76efdab7e1dff3c637b",
      "revlink": "https://github.com/WebKit/WebKit/commit/2b6f2e9cb51ed9c4a01ce76efdab7e1dff3c637b",
      "sourcestamp": {
        "branch": "main",
        "codebase": "",
        "created_at": 1664350478,
        "patch": null,
        "project": "WebKit/WebKit",
        "repository": "https://github.com/WebKit/WebKit",
        "revision": "2b6f2e9cb51ed9c4a01ce76efdab7e1dff3c637b",
        "ssid": 28496
      },
      "when_timestamp": 1664350392
    }
  ],
  "meta": {
    "total": 2
  }
}

I think this means we can bypass having to introspect the local git state at all and so the following script works for me quite quickly and correctly:

#!/usr/bin/env bash

# Debugging this?  Uncomment the following to see the commands as they run!
#set -x

BUILDER_NAME=${1:-GTK-Linux-64-bit-Release-Ubuntu-LTS-Build}
BUILD_INFO=$(curl "https://build.webkit.org/api/v2/builders/$BUILDER_NAME/builds?order=-number&limit=1&complete=true&state_string=build%20successful" 2>/dev/null)
NUMBER=$(jq -Mr '.builds[0].number' <<< "$BUILD_INFO")
BUILDER_ID=$(jq -Mr '.builds[0].builderid' <<< "$BUILD_INFO")
CHANGES_INFO=$(curl "https://build.webkit.org/api/v2/builders/$BUILDER_ID/builds/$NUMBER/changes" 2>/dev/null)
BUILD_REV=$(jq -Mr '.changes[0].revision' <<< "$CHANGES_INFO")

echo "$BUILD_REV"

NB: I tried going pure API here because the git scraping seemed to pick an ancestor of the correct revision rather than exact revision the builder number corresponded to.

Thank you Andrew for looking into this. I'm sorry the script didn't work as expected.

I remember I looked back in the day how to fetch the Git commit directly from the build JSON data, instead of digging into the commit history, but I didn´t find a way to do it. Also querying the JSON document with jq instead of grepping is a much better approach.

I think I'm going to update the script with your changes to avoid confussing anyone looking for how to query data from Bugzilla :D

No worries, iteration is always the name of the game in automation like this! (And it's always so energizing to be able to collaborate with people on things, especially searchfox! :)

👍 to updating the script to avoid confusion.

Updated script with changes.