Mini Approval Testing

I always learn a lot from Emily Bache everytime I hear her speak or read her blog. Recently on an automation project we needed the idea of an assertion that wasn't really an assertion or the ability to make more than one assertion in a block so that a test wouldn't stop on the first error but would report on all failures or differences at the end of test was raised by the team.

After some digging, I found Approvals and Mutation Testing and the idea of approval testing. This is the idea that, even if you don't know anything about a "legacy" software system, you can create a kind of image of the output of the program at some level and treat that as a golden copy so that any changes you make to the code must still conform to that output and any deviations are reported as differences and ultimately errors.

This way, you are free to experiment and refactor and know that, even if you don't a proper set of unit tests, you have got a reference point to compare your changes against. Hopefully, the unit tests can evolve as you refactor the code to be more amenable to testing.

Mini Approval

Back to the automation project. This was a Selenium-based web automation project that used the Page Objects pattern to define screens and interact with them. One test scenario is to load a screen, pick an option and validate that all of the values on the screen in each of the controls is as it was in the past and any difference needs to be flagged up. That sounded like just the thing for approval tests.

Unfortunately, the only Nu-Get package I could find required .Net Core or v 4.6.1 or greater to be the target plaform for the application. Due to other constraints of the project, this wouldn't be suitable so I decided to hand roll a mini approval test suite to work alongside the NUnit framework we had built.

Code

This is what that looks like. First we need a way to record all the values of all the fields in a form's page object. The name of the InstanceSnapshot is the class name of the instance and the values are name-value pairs corresponding to the controls and the value of each one as provided by reflecting on the page object's gettable properties.

public class InstanceSnapshot
{
    // build a snapshot of data from another object's 
    // public property values
    public InstanceSnapshot(object instance)
    {
        this.Name = instance.ToString();
        this.Values = new SortedDictionary<string, string>();

        var type = instance.GetType();

        foreach (var property in type.GetProperties(BindingFlags.Public | BindingFlags.Instance))
        {
            if (property.CanRead)
            {
                string key = property.Name;
                string value = "<Unknown>";

                try
                {
                    object valueObj = property.GetValue(instance);

                    if (Object.ReferenceEquals(null, valueObj))
                    {
                        value = "<Null>";
                    }
                    else
                    {
                        value = valueObj.ToString();

                        if (String.IsNullOrEmpty(value))
                            value = "<Blank>";
                    }
                }
                catch
                {
                    value = "<Unknown>";
                }

                this.Values.Add(key, value);
            }
        }
    }

    public string Name { get; private set; }

    public IDictionary<string, string> Values { get; private set; }

    // simplest output format
    public override string ToString()
    {
        var builder = new StringBuilder();

        builder.AppendLine(this.Name);

        foreach (var key in this.Values.Keys)
            builder.AppendFormat("{0} : {1}\r\n", key, this.Values[key]);

        return builder.ToString();
    }
}

We also need a way to persist the snapshot so the ToString method gives us the simplest possible rendering.

Conversely, if we render to a file, we need to be able to reconstruct so I added another constructor to take a list of lines read from a file:

    public InstanceSnapshot(IEnumerable<string> lines)
    {
        this.Name = lines.FirstOrDefault();
        this.Values = new SortedDictionary<string, string>();

        foreach(string line in lines.Skip(1).Where (x => !String.IsNullOrEmpty(x)))
        {
            int colonPosition = line.IndexOf(":");
            string key = line.Substring(0, colonPosition).Trim();
            string value = line.Substring(colonPosition + 1).Trim();
            this.Values.Add(key, value);
        }
    }

Compare

Now we can create snapshots from objects and persist them, we should compare them. First something to capture the differences - the property name, expected value from the original object and the actual value from the new object:

    public class SnapshotDifference
    {
        public string Name { get; set; }

        public string Actual { get; set; }

        public string Expected { get; set; }
    }

And the compare method so we can compare an existing "golden" image of an instance with a new instance from the test.

    public ICollection<SnapshotDifference> CompareTo(InstanceSnapshot other)
    {
        var differences = new List<SnapshotDifference>();

        var missingKeys = this.Values.Keys.Except(other.Values.Keys);

        if (missingKeys.Any())
        {
            foreach (var missingKey in missingKeys)
            {
                differences.Add(new SnapshotDifference
                {
                    Name = this.Name + "." + missingKey,
                    Expected = this.Values[missingKey],
                    Actual = "<Missing>"
                });
            }
        }

        var addedKeys = other.Values.Keys.Except(this.Values.Keys);

        if (addedKeys.Any())
        {
            foreach (var addedKey in addedKeys)
            {
                differences.Add(new SnapshotDifference
                {
                    Name = this.Name + "." + addedKey,
                    Expected = "<Missing>",
                    Actual = other.Values[addedKey]
                });
            }
        }

        var commonKeys = this.Values.Keys.Intersect(other.Values.Keys);

        if (commonKeys.Any())
        {
            foreach (var commonKey in commonKeys)
            {
                string expectedValue = this.Values[commonKey];
                string actualValue = other.Values[commonKey];

                if (String.Compare(expectedValue, actualValue, StringComparison.InvariantCulture) != 0)
                {
                    differences.Add(new SnapshotDifference
                    {
                        Name = this.Name + "." + commonKey,
                        Expected = expectedValue,
                        Actual = actualValue
                    });
                }
            }
        }

        return differences;
    }

We use Intersect to determine the common set of properties by name and those that exist in only the original object or the new object using the Except method. We further examine the common properties to check for differences in value.

Assert

The output is then a list of differences. If there are no differences the list is empty, if there are, they can be submitted to your test framework of choice to generate test errors. In this case, NUnit.

    
    var alice = new Person("Alice");
    var goldenImage = new InstanceSnapshot(alice);

    var bob = new Person("Bob");
    var imageToApprove = new InstanceSnapshot(bob);

    var diffs = goldenImage.CompareTo(imageToApprove);

    if (diffs.Any())
    {
        Assert.Multiple(() =>
        {
            foreach(var difference in diffs)
            {
                Assert.That(difference.Actual, Is.EqualTo(difference.Expected), difference.Name);
            }
        });
    }

Tiny

That's the majority of this tiny framework. Still outstanding is some functionality to create a snapshot if there is not one already available and persist it to a file so that it can be checked in to source control as the golden image for that test scenario, ready for the next test run. This can all be wrapped up in a handy class called Approve so we can use it to approve and instance of a class for a given scenario.

    
public static class Approve
{
    public static void Instance(object instance, string hint = null)
    {
        // improve this path handling
        string workingFolder = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
        string approvalsFolder = Path.Combine(workingFolder, "Approvals");

        if (!Directory.Exists(approvalsFolder))
            Directory.CreateDirectory(approvalsFolder);

        string goldenImagePath = GeneratePath(approvalsFolder, instance.GetType(), hint, "expected");

        if (!File.Exists(goldenImagePath))
        {
            // no approval yet, nothing to compare against - 
            // save this as the golden copy and return
            var snapshot = new InstanceSnapshot(instance);
            File.WriteAllText(goldenImagePath, snapshot.ToString());
        }
        else
        {
            // load golden image
            List<string> goldenFileContent = new List<string>(File.ReadAllLines(goldenImagePath));
            var goldenImage = new InstanceSnapshot(goldenFileContent);

            // save actual image for investigation later.
            string actualImagePath = GeneratePath(approvalsFolder, instance.GetType(), hint, "actual");

            var actual = new InstanceSnapshot(instance);
            File.WriteAllText(actualImagePath, actual.ToString());

            // compare and report
            var diffs = goldenImage.CompareTo(actual);

            if (diffs.Any())
            {
                Assert.Multiple(() =>
                {
                    foreach (var difference in diffs)
                    {
                        Assert.That(difference.Actual, Is.EqualTo(difference.Expected), difference.Name);
                    }
                });
            }
        }
    }

    private static string GeneratePath(string baseFolder, Type type, string hint, string caseName)
    {
        StringBuilder fileName = new StringBuilder();

        fileName.Append(type.FullName);

        if (!String.IsNullOrEmpty(hint))
            fileName.AppendFormat(".{0}", hint);

        fileName.AppendFormat(".{0}.txt", caseName);
        
        return System.IO.Path.Combine(baseFolder, fileName.ToString());
    }
}
    
var alice = new Person("Alice");
Approve.Instance(alice, "After Create");

alice.GoToJob();
Approve.Instance(alice, "At Work");

I included an optional hint parameter to the Instance method so we can distinguish between difference scenarios for instances of the same type because, in this example, we don't want to confuse a person in an initial state with one where they may have changed state by reacting to a message.