Computer Witchcraft

Rediscovered the opening few sentences from one of my favourite books, Danny Hillis’ “The Pattern on the Stone”.

I etch a pattern of geometric shapes onto a stone. To the uninitiated, the shapes look mysterious and complex, but I know that when arranged correctly they will give the stone a special power, enabling it to respond to incantations in a language no human being has ever spoken. I will ask the stone questions in this language, and it will answer by showing me a vision: a world created by my spell, a world imagined within the pattern on the stone.
A few hundred years ago in my native New England, an accurate description of my occupation would have gotten me burned at the stake. Yet my work involves no witchcraft; I design and program computers. The stone is a wafer of silicon, and the incantations are software.

W. Daniel Hillis

Test Driving PowerShell with Pester

Work in progress text and examples from an upcoming presentation.

Describe It and Should

(quotes from Christmas Carol) It is a ponderous chain!


Import-Module Pester

Describe 'Presenting in public' {
	
	It 'Emotion should be positive' {
	
		$Emotion = 'Kacking It'
		$Emotion | Should Be 'Joy'
	}
}

Ok, that fails, let’s fix the test to make it more representative of reality.


Import-Module Pester

Describe 'Presenting in public' {
	
	It 'Emotion may not be positive' {
	
		$Emotion = 'Kacking It'
		$Emotion | Should Not Be 'Joy'
	}
}


Import-Module Pester

Describe 'Presenting in public' {
	
	It 'should be like Macdonalds' {
	
		$Emotion = "I'm Loathing It"
		$Emotion | Should BeLike "I'm Lo*ing It"
	}
}

Refactoring

We can have multiple describe blocks or collapse into one describe with multiple context blocks


Import-Module Pester
Set-StrictMode -Version Latest

Describe 'My Blog' {

    It 'Serves pages over http' {
        Invoke-WebRequest -Uri 'http://deejaygraham.github.io/' -UseBasicParsing | 
		Select-Object -ExpandProperty StatusCode | 
		Should Be 200
    }

    It 'Serves pages over https' {
        Invoke-WebRequest -Uri 'https://deejaygraham.github.io/' -UseBasicParsing | 
		Select-Object -ExpandProperty StatusCode | 
		Should Be 200
    }
}


$ImportantLinks = @(
    'https://deejaygraham.github.io/2015/02/15/sketchnote-challenge/',
    'https://deejaygraham.github.io/img/posts/sketchnoting-challenge/mac-power-users.png'
)

Describe 'Externally Referenced Links' {

    $ImportantLinks | ForEach-Object {

        It "$_ is reachable" {

            Invoke-WebRequest -uri $_ -UseBasicParsing | Select-Object -ExpandProperty StatusCode | Should Be 200
        }
    }
}

Documenting Web APIs

We can setup a test context for a specific Rest API - get and post.


Param (
	[Parameter(Mandatory=$True)]
	[string]$BaseUri
)

[string]$Resource = "$($BaseUri)builds/"

Describe "Build API : $Resource" {

	Context "Get Requests" {

		$AllBuilds = Invoke-RestMethod -Method Get -Uri $Resource -UseBasicParsing
	
		It 'Finds all builds' {

			$AllBuilds.Count | Should BeGreaterThan 0
		}

		It 'Finds builds by Id' {

			$SpecificBuild = $AllBuilds | Select-Object -First 1 

			$Build = Invoke-RestMethod -Method Get -Uri "$($Resource)$($SpecificBuild.id)" -UseBasicParsing

			$Build.id | Should Be $SpecificBuild.id
		}

		It 'Finds build by Name' {

			$SpecificBuild = $AllBuilds | Select-Object -First 1 

			$Build = Invoke-RestMethod -Method Get -Uri "$($Resource)?name=$($SpecificBuild.name)" -UseBasicParsing

			$Build.id | Should Be $SpecificBuild.id
		}

		It 'Unknown build returns 404' {

			[int]$InvalidDbKey = 99999999

			Try {
				Invoke-RestMethod -Method Get -Uri "$($Resource)$InvalidDbKey" -UseBasicParsing
			} Catch {

				$Error[0].Exception.Message | Should Be 'The remote server returned an error: (404) Not Found.'
			}
		}

	}

	Context "Post Requests" {

		$AllBuilds = Invoke-RestMethod -Method Get -Uri $Resource -UseBasicParsing

		It 'Accepts new builds' {

			$SpecificBuild = $AllBuilds | Select-Object -First 1 

			$NewBuild = @{

				productid = $SpecificBuild.productid 
				name = 'Example build'
				definition = '99.99.99.99'
				url = 'http://example.com/'
				version = 'Example Build 99'
			}

			$JsonObject = $NewBuild | ConvertTo-Json

			$CreatedObject = Invoke-RestMethod -Method Post -Uri $Resource -UseBasicParsing -Body $JsonObject -ContentType 'application/json'

			$CreatedObject.name | Should Be $NewBuild.name
		}
	}
}

Test Suite

We can then invoke tests from a controlling script that will execute against all tests in a directory.


Param (
	[Parameter()]
	[string]$BaseUri = 'http://localhost:51089/api/'
)

Import-Module Pester

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

$here = Split-Path -Path $MyInvocation.MyCommand.Path

Invoke-Pester -Tags First -Script @{ 

	Path = "$here\*.Tests.ps1"

	Parameters = @{ 
		BaseUri = $BaseUri
	}
}

Christmas Driven Development

(red and green) Like scrooge (more on him later), I need to know when christmas is coming so I can make preparations. Let’s write two functions, one to get some content from a website, and another to parse the text and find the value.


Function Get-WebPageContent {

    Param(
        [Parameter(Mandatory=$True)]
        [string]$url
    )

    $response = Invoke-WebRequest -Uri $url -UseBasicParsing
    Write-Output $response.RawContent
}

Function Get-HowLongUntilChristmas {

    $response = Get-WebPageContent -Url "https://www.emailsanta.com/clock.asp"

    If ($response -match '<span class="XmasDayemph">(.*)</span>') {

        Write-Output $matches[1]
    }
}


Mocking


Import-Module Pester

Describe 'Email Santa Service' {

    Context 'Countdown to Christmas' {

        It 'Expressed in days' {
            Get-HowLongUntilChristmas | Should BeLike '* days'               
        }
		
        It 'Calculates correctly' {
            Get-HowLongUntilChristmas | Should Be '24 days'               
        }
    }
}

Fails. We need some way to make this test work for every day of the year. Mock the content function and return a known value. Works all of the time.


Import-Module Pester

Describe 'Email Santa Service' {

    Context 'Countdown to Christmas' {

        Mock Get-WebPageContent { return '<html><span class="XmasDayemph">24 days</span>' }

        It 'Expressed in days' {
            Get-HowLongUntilChristmas | Should BeLike '* days'               
        }
		
        It 'Calculates correctly' {
            Get-HowLongUntilChristmas | Should Be '24 days'               
        }
    }
}

SDD

It should be Christmas Day, I am sure,” said she, “What’s to-day?” cried Scrooge, calling downward to a boy in Sunday clothes, who perhaps had loitered in to look about him.

Albert Finney (and my pop-in-law whose birthday it is) are both interested in whether today is christmas day.

To save them shouting out of the window to a passing urchin, I have written a function.


Function Test-ChristmasDay {

    $Today = Get-Date

    If ($Today.Month -eq 12 -and $Today.Day -eq 25) {
        Write-Output $True
    } Else {
        Write-Output $False
    }
}

So I know I should test that difficult logic:


Describe 'Scrooge' {

    Context 'Before the Ghosts Visit' {
        
        It 'Doesn't care about Christmas day' {
			# when will this work, when will it not work
            Test-ChristmasDay | Should Be $false
        }
    }

    Context 'The Spirits have done it all in one night' {

        It 'It is Christmas Day' {
            Test-ChristmasDay | Should Be $True
        }
    }
}


Describe 'Scrooge' {

    Context 'Before the Ghosts Visit' {
        
        Mock Get-Date { New-Object DateTime (2018, 7, 10) }

        It 'Doesn't care about Christmas day' {
            Test-ChristmasDay | Should Be $false
        }
    }

    Context 'The Spirits have done it all in one night' {

		# Wizzard way of doing it.
		Mock Get-Date { New-Object DateTime (2018, 12, 25) }
	
        It 'It is Christmas Day' {
    
            Test-ChristmasDay | Should Be $True
            Assert-MockCalled Get-Date -Times 2 -Exactly
        }
    }
}

Made tests independent of dates and times. We can expect that our code is much more likely to succeed when put into a real environment. We can also make sure that the internals we are expecting are called the right number of times.

Modules

Testing modules and developing them side by side with the tests, it’s a good idea to make sure we are working with the most up to date version


Get-Module Scrooge | Remove-Module -Force
Import-Module $here\Scrooge.psm1 -Force


Set-StrictMode -Version Latest

Function Get-Ghost {
  [CmdletBinding()]
  Param()
  
  $Ghosts = @( 'Jacob Marley', 'Christmas Past', 'Christmas Present', 'Christmas Future', 'Patrick Swayze', '' )

  Write-Output $Ghosts
}

Export-ModuleMember -Function *


Describe 'Get-ChristmasCarolGhost' {

	$Ghosts = Get-ChristmasCarolGhost
    $FirstGhost = $Ghosts | Select-Object -First 1

    It 'Three spirits shall visit scrooge' {
        $Ghosts.Count | Should -Be 4
    }

    It 'First Ghost is Marley' {
        $FirstGhost | Should -Be Like '*Marley'
	}
}

Driving Home For Christmas


Describe 'Get-ChrisReaSong' {

	$Path = Join-Path $TestDrive -ChildPath 'ChrisRea.txt'
	Set-Content -Path $Path -Content "I'm driving home for Christmas, Oh, I can't wait to see those faces"

    It 'Can read lyrics from a local file' {
		Get-ChristmasSong -Path $Path | Should Contain 'faces'
    }
}


Set-StrictMode -Version Latest

Function Get-ChristmasSong {
  [CmdletBinding()]
  Param(
    [Parameter(Mandatory=$True)]
    [string]$Path
  )

  Write-Output (Get-Content -Path $Path)
}

Export-ModuleMember -Function *

CI


Invoke-Pester -OutputFile 'PesterResults.xml' -OutputFormat NUnitXml

Code Coverage


Invoke-Pester -Script Demo.Tests.ps1 -CodeCoverage Demo.ps1

Occasionally...

Running an Azure classic cloud service, which at the top level is a big while(true) block, one sometimes needs to run an action sometimes and not every time around the loop. We could solve this in a multi-threaded way by creating a thread per action and blocking for a set amount of time in each thread. A more low-tech and more “tell-dont-ask” approach might be something like this.

An occasional task runs as part of a while loop, single threaded whenever it feels like the next run is required:


public class OccasionalAction
{
	private DateTime nextRun;
	private TimeSpan frequency;

	private Action action;

	public OccasionalAction(Action a, TimeSpan freq)
		: this(a, freq, DateTime.MinValue)
	{
	}

	public OccasionalAction(Action a, TimeSpan freq, DateTime future)
	{
		this.action = a;
		this.frequency = freq;
		this.nextRun = future;
	}

	public void Run()
	{
		if (RunNow())
		{
			this.action();
			this.nextRun = DateTime.UtcNow + this.frequency;
		}
	}

	protected virtual bool RunNow()
	{
		return (DateTime.UtcNow > this.nextRun);
	}
}


Tests confirm the shape of the client code I was shooting for:


[Test]
public void OccasionalAction_Always_Executed_First_Time()
{
	bool actionCalled = false;

	OccasionalAction action = new OccasionalAction(() =>
	{
		actionCalled = true;
	},
	TimeSpan.FromDays(1));

	action.Run();

	Assert.IsTrue(actionCalled);
}

[Test]
public void OccasionalAction_Executes_Once_For_Long_Frequency()
{
	int callCount = 0;

	OccasionalAction action = new OccasionalAction(() =>
	{
		callCount++;
	},
	TimeSpan.FromDays(365));

	for(int i = 0; i < 10; ++i)
		action.Run();

	Assert.AreEqual(1, callCount);
}

[Test]
public void OccasionalAction_Executes_Multiply_For_Short_Frequency()
{
	int callCount = 0;

	OccasionalAction action = new OccasionalAction(() =>
	{
		callCount++;
	},
	TimeSpan.FromMilliseconds(1));

	for (int i = 0; i < 10; ++i)
	{
		action.Run();
		System.Threading.Thread.Sleep(TimeSpan.FromMilliseconds(10));
	}

	Assert.AreEqual(10, callCount);
}


[Test]
public void OccasionalAction_Future_Action_Not_Run_First_Time()
{
	bool actionCalled = false;

	OccasionalAction action = new OccasionalAction(() =>
	{
		actionCalled = true;
	},
	TimeSpan.FromDays(1),
	DateTime.Now.AddHours(1));

	action.Run();

	Assert.IsFalse(actionCalled);
}


And a collection of occasional tasks to round things off:


public class OccasionalActions
{
	private List<OccasionalAction> actions = new List<OccasionalAction>();

	public void Add(OccasionalAction action)
	{
		this.actions.Add(action);
	}

	public void Add(Action a, TimeSpan frequency)
	{
		this.Add(new OccasionalAction(a, frequency));
	}

	public void Run()
	{
		this.actions.ForEach(x => x.Run());
	}
}


Tracing Failed IIS Requests

Today, after some frustrating debugging IIS and rewrite rules not matching what was expected, I learned that IIS can log out the match process and give a break to someone trying to psychically debug a reverse proxy situation.

Here’s how I did it in code.


using Microsoft.Web.Administration;

...

	using (ServerManager serverManager = new ServerManager())
	{
		Configuration config = serverManager.GetApplicationHostConfiguration();

		ConfigurationSection traceFailedRequestsSection = config.GetSection("system.webServer/tracing/traceFailedRequests");
		ConfigurationElementCollection traceFailedRequestsCollection = traceFailedRequestsSection.GetCollection();

		if (traceFailedRequestsCollection.Count == 0)
		{
			ConfigurationElement addElement = traceFailedRequestsCollection.CreateElement("add");
			addElement["path"] = @"*";

			ConfigurationElementCollection traceAreasCollection = addElement.GetCollection("traceAreas");
			ConfigurationElement addElement1 = traceAreasCollection.CreateElement("add");
			addElement1["provider"] = @"WWW Server";
			addElement1["areas"] = @"Rewrite";
			addElement1["verbosity"] = @"Verbose";
			traceAreasCollection.Add(addElement1);

			ConfigurationElement failureDefinitionsElement = addElement.GetChildElement("failureDefinitions");
			failureDefinitionsElement["statusCodes"] = @"200-399";
			traceFailedRequestsCollection.Add(addElement);

			serverManager.CommitChanges();
		}
	}


ngrok

A new tool (to me) is ngrok a command line tool and online service to allow you to map an external url to a local server behind a NAT or firewall.

It does require sign up to the service but free accounts are available but these use random urls under the ngrok.io domain.

If you start a local developer machine web service or api you can run the command line ngrok and tell it to map it’s generated url to a local host url and port number.

NGrok can be started with just the details of the port number you want forwarded.

	
	ngrok http 14446
	

I found my asp.net application got a bit upset having ngrok forward the request to it. A quick google showed that it required a host header that was being omitted by ngrok. Fortunately, this is easily fixed:

	
	ngrok http 14446 -host-header="localhost:14446"
	

The console (with some details redacted) looks like this:

ngrok

Once ngrok is running, it works pretty much as you would expect, with a summary of the latest few hits to the service. If you want more detail and want to be able to dig into a request, you can use a local server instance created by ngrok to examine traffic.

This has made some awkward Azure-hosted website development in .Net core much easier when I can run the service locally and debug while responding to external requests. Triffic! :)

Unix Philosophy

I’ve been thinking recently about how learning PowerShell and Elixir have had an influence on my day job working with C# and was reminded of two quotes from the early Unix days that seem appropriate.

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

Doug McIlroy, Inventor of the Unix Pipe

Much of the power of the UNIX operating system comes from a style of program design that makes programs easy to use and, more important, easy to combine with other programs. This style has been called the use of software tools, and depends more on how the programs fit into the programming environment and how they can be used with other programs than on how they are designed internally.
This style was based on the use of tools: using programs separately or in combination to get a job done, rather than doing it by hand, by monolithic self-sufficient subsystems, or by special-purpose, one-time programs.

Brian Kernighan and Rob Pike, Program Design in the Unix Environment