Help:Toolforge/Auto-update a tool from GitHub/GitLab

From Wikitech
Note: Use of this deployment method has risks. If your GitHub repository is compromised by a 3rd party, your live tool could be impacted. Please think carefully before proceeding.

Overview

You can use GitHub's webhooks feature to automatically deploy any changes made on GitHub to Toolforge. With this setup, whenever a push is made to the repository on GitHub, the changes will be fetched to the Toolforge clone of the repository, without having to manually log in to the tool's account and running git pull.

Using webhooks with Github

Introductory-level information about using webhooks with GitHub can be found here:

PHP tool hosted in $HOME/public_html

First, set up a webhook listener that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git pull`; } ?>" > public_html/git-pull.php
    • Hint You can add more commands after git pull, e.g. to run a build script.
    • Info You can find useful information regarding running shell commands from PHP at php.net and at stackoverflow.com.

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

PHP tool hosted in $HOME/www/static

First, set up a webhook listener (in the non-static $HOME/public_html directory) that runs "git pull" whenever it's accessed from GitHub:

  1. ssh USER@login.toolforge.org
  2. become MYTOOL
  3. mkdir ~/public_html
  4. echo "<?php if(isset($_SERVER['HTTP_X_GITHUB_EVENT'])) { `git -C ../www/static pull`; } ?>" > public_html/git-pull.php
  5. webservice start

Then, set up the webhook emitter on GitHub's side:

  1. Go to https://github.com/USER/REPO/settings/hooks and click the "Add webhook" button
  2. In the "Payload URL" field, type https://MYTOOL.toolforge.org/git-pull.php
  3. Save the hook (the remaining fields can be kept with the default values)

That's it! Now whenever you push commits to the repo on GitHub, the Toolforge clone will be automatically updated.

Python tool hosted in $HOME/www/python/src

Given that we have a repository that has the code under the $HOME/www/python/src dir, we setup an endpoint to listen on the url https://MYTOOL.toolforge.org/update-server

  • if we have a Flask app we can add the following endpoint to our main file
@app.route("/update-server", methods=["POST"])
def webhook():
    if request.method == "POST":
        subprocess.check_output(["git", "pull", "origin", "main"])
        return "Updated Toolforge project successfully", 200
    else:
        return "Wrong event type", 400
  • if we have a FastAPI app then we add the following:
@app.post("/update-server")
async def webhook():
    subprocess.check_output(["git", "pull", "origin", "main"])
    return "Updated Toolforge project successfully"

In order to create the webhook we follow the same procedure with the PHP app but we put https://MYTOOL.toolforge.org/update-server as the webhook url.

GitLab webhook

In order to set up the webhook on a GitLab repository you follow the same steps on the application side as for GitHub, but for the php app you will replace HTTP_X_GITHUB_EVENT with HTTP_X_GITLAB_EVENT .

Then navigate to your GitLab project and got to Settings -> Webhooks and put the url and tick the "Push events" trigger and type the name of your main branch under the "wilcard pattern" field e.g. main, master etc. and then click the "Add webhook" button.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Auto-update_a_tool_from_GitHub/GitLab&oldid=2130493"