Project: ToolTool

Contact:Rok Garbas, (backup Release Engineering)

Tasks in the RelEng infrastructure and make use of generic binary artifacts, which are stored in dedicated artifacts repositories (S3 buckets). ToolTool application provides an interface to those artifacts repositories.

Request authentication credentials for the client (tooltool.py)

Open a bug on bugzila and request new taskcluster client credentials that that you will then use in.

Use the following points to guide you opening the bug:

  1. Product field should be Release Engineering
  2. Component field should be Applications: ToolTool
  3. Summary field should be Requesting taskcluster client credentials to use with tooltool.py
  4. Description field should contain:
    • who is the responsible person and which is the responsible team
    • what is the purpose of usage
    • what should be the expiration date of the credentials (suggested is one year)
    • which level of access is required: - Download PUBLIC files from tooltool. - Download INTERNAL files from tooltool. - Upload PUBLIC files to tooltool. - Upload INTERNAL files to tooltool. - Manage tooltool files, including deleting and changing visibility levels.

You should receive an authentication file in return which you pass into tooltool.py script via --authentication-file option.

How to generate taskcluster client credentials

  1. Go to https://tools.taskcluster.net/auth/clients.

  2. Make sure you are logged into taskcluster.

  3. Fill the Create New Client form:

    ClientId:

    Make sure to include the Bug number by following the template

    project/releng/services/tooltool/bug<NUMBER>
    Description:

    Who is responsible and which team, also where is this token used.
    Expires:

    Requested expiration, by default set it to 1 year.
    Client Scopes:

    List of scopes requested based on the requested level of access:

    • Download PUBLIC files from tooltool

      project:releng:services/tooltool/api/download/public

    • Download INTERNAL files from tooltool

      project:releng:services/tooltool/api/download/internal

    • Upload PUBLIC files to tooltool

      project:releng:services/tooltool/api/upload/public

    • Upload INTERNAL files to tooltool

      project:releng:services/tooltool/api/upload/internal

    • Manage tooltool files, including deleting and changing visibility levels

      project:releng:services/tooltool/api/manage

  4. Send clientId and accessToken in a JSON authentication file via https://send.firefox.com. Format of authentication file should be:

    {
    "clientId": "project/releng/services/tooltool/bug<NUMBER>",
    "accessToken": "<TOKEN-WHICH-WAS-PROMPTED-IN-TASKCLUSTER-TOOLS>"
}

Troubleshooting deployment

In case of an incident this five steps that should help you narrow down the problem.

  1. Look at Heroku metrics to get birds view on the running application.
  2. There might be some problems with Heroku. Make sure to also check their status page
  3. Check if there is any unsual high count of errors collected in Sentry.
  4. To see more logs (from the past) look at Papertrail.
  5. Sometimes restarting an application might solve the issue (at least temporary). Once you restart the application also verify that it is working correctly (follow instructions below).

How to check if ToolTool is working correctly?

To test and verify that the JSON API is running correctly please follow the following steps:

  1. Select which environement (production or staging).

    For production:

    $ export URL=https://tooltool.mozilla-releng.net

    For staging:

    $ export URL=https://tooltool.staging.mozilla-releng.net

  2. Known public sha512 should redirect (return code: 302)

    $ curl $URL/sha512/f93a685c8a10abbd349cbef5306441ba235c4cbfba1cc000299e11b58f258e9953cbe23463515407925eeca94c3f5d8e5f637c95be387e620845efa43cdcb0c0
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="..."></a>.  If not click the link.%

  3. Known private sha512 should stay protected (return code: 403)

    $ curl $URL/sha512/06a1cf7b1918ffd94210e8089cf48985fbf9af95f15cd9dd5007df76b934c2b825147334ba176c3f19a9f7d86585c58e017bc23a606e8831872c8b40560be874
{
   "detail": "You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.",
   "instance": "about:blank",
   "status": 403,
   "title": "403 Forbidden: You don't have the permission to access the requested resource. It is either read-protected or not readable by the server.",
   "type": "about:blank"
 }

  4. Unknown sha512 should return invalid error (return code: 400)

    $ curl $URL/sha512/invalid
{
  "detail": "Invalid sha512 digest",
  "instance": "about:blank",
  "status": 400,
  "title": "400 Bad Request: Invalid sha512 digest",
  "type": "about:blank"
}

Develop

To start developing tooltool/api you would need to:

  1. Install all requirements and read through general guide how to contribute.

  2. Read through python projects guide, how python projects are structured and how to add/update dependencies to a project.

  3. And last you will have to read about conventions we use to write REST endpoints using Flask.

    It is important to know that tooltool/api uses the following Flask extensions: