Automated Approval Testing with Dynamic Data Sets

For some tests, you’ll need to check if files or documents are the same as they were before, or that they comply with a certain format. When something differs, you need to “reject” or “approve“ them, or even make them the new baseline if you are welcoming the changes. We call this Approval Testing. For more use cases, take a look at the nice presentation of Emily Bache about this subject with the catchy title Catching Dancing Ponies.

A nice package that can help you with this is ApprovalTests. You can import it as a package and use it right away. Approval Testing is not only limited to text documents, this is a little example on how to use AppovalTests on a scraped webpage:

[TestMethod]
public void ApproveTheHomePageHTML()
{
    Scraper scraper = new Scraper(@"https://qualityengineers.nl");
    var html = scraper.GetHTML();
    Approvals.VerifyHtml(html);
} 

The first time this runs, it will make a snapshot of the text and save it to a file. The second time you run the test, it will look for differences and once there is one, your favorite diff or merge tool will open and you can approve the change or reject them and raise a bug. This way you can, for example, track that your homepage is unchanged and when it changes, the test will fail until you approve the changes.

OK, but now the challenge we are facing: can we approve a dynamic amount of documents? How it is currently designed, I can only use one TestMethod for one approval test because the stored approved text gets the name of the method, like ApproveWebScrape.ApproveTheHomePageHTML.approved.html. So if I want to approve the contacts page too, I'll have to add another test.

But when new pages are added (like this new blogpost today), I also want them to be added to the approved pages collection. And I want it dynamically because I am lazy.

To dynamically get web content, we are embedding the webpage-scraper into a crawler. (Crawlers are using scrapers to collect all the links and will visit them until a whole domain is scraped. This is an example of a crawler:

var crawler = new Crawler(@"https://qualityengineers.nl");
var folder = new DirectoryInfo(@"c:\temp\storedPages");
var filePath = crawler.Crawl(folder);
Console.WriteLine($"{filePath.Count()} pages have been crawled."); 

This crawler enters a domain, and tries to follow all links in that domain and stores the pages it will find as text in the given folder.

We can loop through all files in the folder and try to approve them, but it will only be in one method! That is against the design of atomic tests…

We are going to use the following design:

1. Get all pages from a domain using a crawler

2. Store them in a folder as separate files

3. Make a list of all stored files

4. Inject each file into a data driven test

Let’s start with a data driven test and assume we already scraped the files with the crawler above and stored a list with these files in a .csv file.

[TestMethod]
public void ApproveEverythingFromFolderCSVFile()
{
    var fileToApprove = TestContext.DataRow<0>.ToString();
 
    // get the next file to approve from the csv and approve it
    using (var file = new StreamReader(fileToApprove))
    {
        var text = file.ReadToEnd();
        Approvals.Verify(text);
    }            
} 

(You’ll need to have a public TestContext TestContext {get; set;} property in your class as well if you haven’t one already)

But now, the file gets always verified against one verify file called something like ApproveWebScrape.ApproveEverythingFromFolderCSVFile.approved.html. resulting in lots of false positives (and consequentially an unworkable situation).

To solve this, we are going to assign a custom name for each file to test with the Approvals.RegisterDefaultNamerCreation(Func<IApprovalNamer> creator) method. This looks a bit complicated if you never burned your finger onto lambda expressions or delegates, but an example will follow.

All we need is a namer that implements the IApprovalNamer interface, so let’s do it:

// implements our own namer
class CustomNamer : IApprovalNamer
{
    public CustomNamer(string sourcePath, string name)
    {
        SourcePath = sourcePath;
        Name = name;
    }
 
    public string Name { get; private set; }
    public string SourcePath { get; private set; }
} 

Implementing an Interface is not very hard: Just type : ISomeInterface behind a new class definition and Visual Studio will automatically suggest to insert the missing members. You can do without a constructor, but this just looks better J.

And now, use it in our Approve Everything method:

[TestMethod]
public void ApproveEverythingFromFolderCSVFile()
{
    // get the filename that will also be used as unique id
    var fileToApprove = TestContext.DataRow<0>.ToString();
 
    // register a name creator that makes a unique name for this compare/approval
    Approvals.RegisterDefaultNamerCreation(() => 
         new CustomNamer(@"C:\temp\approved", Path.GetFileName(fileToApprove))
    );
 
    // get the next file to approve from the csv and approve it
    using (var file = new StreamReader(fileToApprove))
    {
        var text = file.ReadToEnd();
        Approvals.Verify(text);
    }            
} 

We are good to go, unless you want to have your data automatically collected just at the start of the test. We can use the attribute to gather the data and make the CSV file that will be used later on:

[ClassInitialize]
public void CrawlPagesAndPrepareDataSet()
{
    Crawler crawler = new Crawler(@"https://qualityengineers.nl");
    DirectoryInfo folder = new DirectoryInfo(@"c:\temp\storedPages");
 
    List<string> filePaths = crawler.Crawl(folder);
    
    using (StreamWriter file = new StreamWriter(@"C:\temp\files.csv"))
    {
        file.WriteLine("Dummy Header"); // csv files needs a header row
        foreach (string filePath in filePaths)
            file.WriteLine(filePath);
    }
} 

Conclusion

Approval Testing seems to be a useful way of testing in particular use cases. Especially if you want to compare and merge in a tool that is designed for that job. In this article, the focus is on one major drawback we experienced: not being able to test a dynamic amount of data.

Fortunately, after researching the inner workings of the ApprovalTests framework, we discovered the ability to customize names, allowing us to dynamically assign approval data. With the use of the Data Driven Test capabilities of the standard Microsoft UnitTest framework, we could create a pattern on how to dynamically collect, verify and approve data.