Automated testing for font families hosted on Google Fonts

Mar 25, 2019

How Travis and three internal tools were used to automate testing font families.

After several months we finally have all the tools/libs in place to build an automated qa pipeline to help onboard new/updated families in google/fonts.

Our QA chain uses the following three tools:

  • FontBakery: Helps ensure our fonts comply with the OpenType specification.
  • Diffenator: Allows us to check for regressions against the same family which is currently hosted on fonts.google.com.
  • Diffbrowsers: Use Browserstack to generate before and after gifs to show rendering regressions.

Each tool is independent and has a range of options for extended testing. However, we have found specific configurations which work best when releasing new or updated families. These configurations have been included in a simple orchestration script.

The .travis.yml file inside google/fonts triggers the script to run every time a user submits a pull request. It will only run if the pr contains changes to font files.

Once the script has finished, the diff reports are included as a comment in the pull request. The user can now download the reports and ask for an external reviewer to approve the new or updated family.

alt text

For the past couple of years, font contributions have only been made by myself and Dave Crossland. In 2019 we have taken on another five contributors. This automated workflow has greatly sped up and simplified our testing process. We have already made more contributions in the last three months than in the past two years.