Easy Flutter Golden Tests with Tolerance

Simple way how to snapshot test your app

If you try to change a complex mobile app’s theme, it can quickly backfire. You can break some drawers, textfields, making the UI dull or unusable. This can be avoided by implementing UI tests, which are tedious to maintain at a large scale.

However, you can always use snapshot testing or golden testing for any kind of widget in your app. which can monitor your widgets for any changes, so you can modify your themes and common widgets with confidence you are not breaking anything.

Snapshot testing / Golden testing is technique, when you render current state of UI in form of image and compare it to master image, which was pregenerated and verified by the developer. If they do not match, it signals that something has changed. If we are ok with a change, we need to change master image too.

Moreover, I want to show you how to enable percentual tolerance for the golden tests as you probably do not want to fail every test, if there is one pixel off.

Basic Golden Testing

The most basic golden test goes like this:

void main() {
  // define widget
  testWidgets("Basic golden test", (tester) async {
    await tester.pumpWidget(MaterialApp(
        theme: ThemeData(
            colorScheme: ColorScheme.fromSeed(seedColor: Colors.blue)),
        home: const AppScaffold(child: Center(child: Text("text")))));

    // search for widget and compare it to image of it
    expect(find.byType(AppScaffold), matchesGoldenFile('golden_test.png'));
  });
}

The snippet will generate one PNG file with a white background and obscured text in the middle. Before running the test, you need to generate an actual golden image:

flutter test --update-goldens "add path, to specify specific dart file if you want - otherwise it runs all tests"

Afterwards, the png file will be created and can be run in test.

Here is the most tedious problem with golden tests: setting up all themes, routers, and screen sizes can take a lot of time, but there is a solution.

Complex Golden Testing

Your app will most probably contain some custom themes, localizations, routers and other dependencies.

Everyone will probably have a different setup, but the golden test arrangement can be customized to fit everyone’s needs.

I am going to put the whole code snippet with comments and a description below it.

// running two tests - one for bright and second one for dark theme
// themes are declared in project in independent file and you should 
// use your own or generate one at least
// passing name of the widget, actual widget to test 
// and GoldenDeviceParams (custom model, which I will show later on)
// @isTest labels the method as runable test 
@isTest // 1.
void testGolden<T extends Widget> ( // 2.
    String name, T widget, GoldenDeviceParams params) {
  // 3.
  testWidgets(
      "Golden test - light - $name",
      (WidgetTester tester) =>
          _testGolden<T>("${name}Light", widget, lightTheme, tester, params));

  testWidgets(
      "Golden test - dark - $name",
      (WidgetTester tester) =>
          _testGolden<T>("${name}Dark", widget, darkTheme, tester, params));
}

Future<void> _testGolden<T extends Widget>(
    String imageName,
    Widget widget,
    ThemeData themeData,
    WidgetTester tester,
    GoldenDeviceParams testParams) async {

  // set up of size constraints - 4.
  tester.view.devicePixelRatio = testParams.devicePixelDensity;
  tester.view.physicalSize = testParams.size;

  // 5.
  final Widget testWidget = MaterialApp(
      home: Theme(
          data: themeData, // custom theme 
          child: MediaQuery(
              data: MediaQueryData( // a lot of widgets use MediaQuery, so it is good to include that 
                  size: testParams.size,
                  devicePixelRatio: testParams.devicePixelDensity),
              child: Container( // background for our widget - 6.
                  color: themeData.brightness == Brightness.light
                      ? Colors.white
                      : Colors.black,
                  child: Center(child: widget)))));
  await tester.pumpWidget(testWidget);

  // 7.
  expect(find.byType(T), matchesGoldenFile('$imageName.png'));
}

The setup contains multiple parts:

1. creates runnable tests with @Test
2. <T extends Widget> to make sure, we are passing widget and to pass generic to find the correct type, which we want to test
3. the test puts together 2 WidgetTests running together to test bright/dark themes. Do not forget to generate your theme for the app for good reusability.
4. the actual tests set up the size of the screen
5. afterwards creates MaterialApp with proper Theme and MediaQuery with proper sizing
6. Container creates a background for your widget in accordance to your theme
7. your actual golden test statement

GoldenDeviceParams is a class declaring sizing for the image in united form, where you can add your preferences and factories.

class GoldenDeviceParams {
  final double devicePixelDensity;
  final Size size;

  const GoldenDeviceParams({required this.devicePixelDensity, required this.size});

  factory GoldenDeviceParams.iPhoneSe() =>
      GoldenDeviceParams(devicePixelDensity: 1, size: const Size(320, 568));

  factory GoldenDeviceParams.fromSize(double width, double height) =>
      GoldenDeviceParams(devicePixelDensity: 1, size: Size(width, height));
}

I use measures of iPhone SE to have look, how the app would react to such small display. It is good rule of thumb, if it fits on smaller displays.

With this setup after, you can create a golden test as easily as this:

void main() async {
  testGolden("TextWidget", const Text("Test"), GoldenDeviceParams.iPhoneSe());
}

After golden test generation 2 files should appear.

I assume you use some library for state handling, localizations, routing and others. These libraries can be added in some way to the testGolden itself via mocking and preserving the simplicity and intentionality of the golden test.

This is highly dependable how well your widgets are written, because well written UI should be dumb enough to take state and show it without many dependencies. State handling and dependencies should be well separated from actual UI to make such tests simple.

Afterwards, you can write tests like this:

void main() {

  testGolden("IntroWelcomeScreen", const IntroWelcome(),
      GoldenDeviceParams.iPhoneSe());

  testGolden("IntroLoginScreen", const IntroLoginScreen(LoginState.loggedOut()),
      GoldenDeviceParams.iPhoneSe());

  testGolden("IntroTermsOfUse", const IntroQueryScreen(),
      GoldenDeviceParams.iPhoneSe());
}

Adding Tolerance

The unfortunate part of golden test implementation in Flutter is that it requires you to match the master image perfectly to one pixel.

There is no space for error. It can sometimes happen that some generated UI/image is not equal every time, because of some shading.

Golden tests can fail when run on different operating systems, even if the images were generated on one OS, due to minor rendering differences in widgets that are imperceptible to the human eye but cause the test to fail.

There is a way to add tolerance to the golden test, but it is not that obvious or well-documented how to do it, so here is what you need to do.

Flutter runs under the hood its configuration for the golden tests and uses the default golden comparator. This comparator is looking for perfect matches.

We can create our own and override the default one.

The comparator looks like this:

class ToleranceGoldenComparator extends LocalFileComparator {

  final double diffTolerance;

  ToleranceGoldenComparator(
    super.testFile, {
    required this.diffTolerance,
  });


  @override
  Future<bool> compare(Uint8List imageBytes, Uri golden) async {
    // compares list of bytes
    final ComparisonResult result = await GoldenFileComparator.compareLists(
      imageBytes,
      await getGoldenBytes(golden),
    );

    // check the the result or take into consideration the difference
    final bool passed =
        result.passed || result.diffPercent <= diffTolerance;

    // if we are within difference, we still want to pass the test
    // if we are not, we are throwing error
    if (!passed) {
      final String error = await generateFailureOutput(result, golden, basedir);
      result.dispose();
      throw FlutterError(error);
    }

    result.dispose();
    return passed;
  }
}

The comparator is an individual class taking in the test file path and actual tolerance used during the comparison. The tolerance is between 0 and 1, which is the range of how much an image can differ in %. In other words, the image can differ from a master image by x %.

The Flutter comparator acts as a global variable, which we need to change for our version. For the sake of the independence of tests, we need to preserve this comparator and put it back as some other test cases can be using the original comparator.

Flutter does not provide a clear way how to access the name of the testing file, which is currently run, so we have to write it down in a variable called testFileName, but we can get a directory of the testing file by using the previous golden comparator.

The original comparator must be added back when the test is finished by using the addTearDown method.

The truth is, the golden comparators do not care about the name of the testing file. If you go into code, you will see that they take the path to file and remove the last part of the path to get folder. So, if you are tired of writing down test file names as variable, you can put there any string and it will still work, because it is removed. However, I would discourage you to do so as in further Flutter versions it can be changed.

Future<void> _testGolden<T extends Widget>(String imageName, Widget widget,
    ThemeData themeData, WidgetTester tester, GoldenDeviceParams testParams,
    {String? testFileName, double? diffTolerance}) async {
  // set up of size constraints
  tester.view.devicePixelRatio = testParams.devicePixelDensity;
  tester.view.physicalSize = testParams.size;

  // --- TOLERANCE PART ---
  if (testFileName != null && diffTolerance != null) {
    // preserving the old comparator
    final previousGoldenFileComparator = goldenFileComparator;
    // getting the directory of the current test file
    final Uri basedir = (goldenFileComparator as LocalFileComparator).basedir;
    // adding our own comparator
    goldenFileComparator = ToleranceGoldenComparator(
      Uri.parse(path.join(basedir.toString(), testFileName)),
      diffTolerance: diffTolerance,
    );
    // putting previous comparator back, when we are done
    addTearDown(() => goldenFileComparator = previousGoldenFileComparator);
  }

  final Widget testWidget = MaterialApp(
      home: Theme(
          data: themeData, // custom theme 
          child: MediaQuery(
              data: MediaQueryData( // a lot of widgets use MediaQuery, so it is good to include that 
                  size: testParams.size,
                  devicePixelRatio: testParams.devicePixelDensity),
              child: Container( // background for our widget
                  color: themeData.brightness == Brightness.light
                      ? Colors.white
                      : Colors.black,
                  child: Center(child: widget)))));
  await tester.pumpWidget(testWidget);

  expect(find.byType(T), matchesGoldenFile('$imageName.png'));
}

Add the tolerance and name of the file to the previous implementation of running testWidgets and you can run your golden tests with tolerance.

@isTest
void testGolden<T extends Widget>(
    String name, T widget, GoldenDeviceParams params,
    {String? testFileName, double? diffTolerance}) {
  testWidgets(
      "Golden test - light - $name",
      (WidgetTester tester) => _testGolden<T>(
          "${name}Light", widget, lightTheme, tester, params,
          testFileName: testFileName, diffTolerance: diffTolerance));

  testWidgets(
      "Golden test - dark - $name",
      (WidgetTester tester) => _testGolden<T>(
          "${name}Dark", widget, darkTheme, tester, params,
          testFileName: testFileName, diffTolerance: diffTolerance));
}

// Example:
void main() async {
  await testInit();

  // tolerance is 5%
  testGolden("IntroWelcomeScreen", const IntroWelcome(),
      GoldenDeviceParams.iPhoneSe(), testFileName: "intro.dart", diffTolerance: 0.05);
  }

Creation of Global Config

It is even possible to have a global config for all tests to use your comparator. It might be viable for you to set some low global tolerance if you have many small deltas across your images, but this should not be the case in a normal setting.

You can create at test directory flutter_test_config.dart file, which Flutter takes in automatically. Before running tests in your actual file, you have the opportunity to run configs, which you require. This applies to the golden test comparator too.

const globalTolerance = 0.005; // 0.5%

Future<void> testExecutable(FutureOr<void> Function() testMain) async {

  // replace comparator for every test, if tests are run locally
  if (goldenFileComparator is LocalFileComparator) {
    final testUri = (goldenFileComparator as LocalFileComparator).basedir;
    goldenFileComparator = ToleranceGoldenComparator(
      Uri.parse(path.join(testUri.toString(), "test.dart")), // dummy file name
      diffTolerance: globalTolerance,
    );
  }
  // run actual tests
  await testMain();
}

This function is run before every run of the test file and replaces the golden comparator with yours. Previous changes can be removed, if this is the way you want to go.

As you can see, the used test file is placeholder only, which is always removed during running the golden test.

Conclusion

Golden tests are a good way, how to make sure your UI stays intact. It is mainly useful when you change common components in the app, so you see, which screens get affected by the changes. It allows you to evaluate the changes and make further changes or accept the current state.

A couple of tips:

• use the golden testing even for simple widgets
• make your UI dumb intentionally, so the implementation of golden tests is simple
• test goldens against all themes in the app
• use different sizes and pixel densities, if you are worried about the flexibility of your UI
• keep golden images small - it improves the speed of tests
• create a specific folder for goldens, where you keep them organized. E.g. folder for every widget under goldens folder, where you mimic the structure of your project.
• Do not forget flutter test --update-goldens!

Thanks for reading and follow for more!