Performance implications of cross-domain calls for unit tests

My latest post about classifying tests of code that spawns AppDomain instances as integration tests raised a discussion in our team that can be quickly summarized as “does it matter?” Running a test fixture has certain overhead, and even if AppDomain creation comes at considerable costs, will we ever notice it when running a suite of thousands tests spread around hundred different assemblies.

In order to check how much cross-domain calls may add to a unit test session, I created a simple project with the following class:

[Serializable]
public class ClassUnderTest
{
    public void Wait(int milliseconds)
    {
        System.Threading.Thread.Sleep(milliseconds);
    }

    public void WaitAcrossDomain(int milliseconds)
    {
        var setup = new AppDomainSetup();
        setup.ApplicationBase = AppDomain.CurrentDomain.BaseDirectory;
        setup.ConfigurationFile = AppDomain.CurrentDomain.SetupInformation.ConfigurationFile;
        var domain = AppDomain.CreateDomain("ClassUnderTest", AppDomain.CurrentDomain.Evidence, setup);
        var instance = domain.CreateInstanceAndUnwrap(Assembly.GetAssembly(this.GetType()).FullName, this.GetType().FullName) as ClassUnderTest;
        System.Threading.Thread.Sleep(milliseconds);
    }
}

Then I created tests for both Wait and WaitAcrossDomain methods and measured how much time unit test runner used on test execution. I ran these tests using the following assumptions:


  • Unit tests must run fast. As Michael Feathers pointed once, if a unit test takes more than 1/10 of a second to execute, it is a slow unit test. So I passed only two values to Wait methods: 10 and 1 milliseconds. Code that needs more usually falls into integration test category.
  • As long as tests belong to the same assembly, total time to execute them is approximately the same as if there was only one test containing all code from these tests. I verified this assumption, and it was valid. So it saved me from copy-pasting test code into multiple tests. I just created one method with [Test] attribute and executed all logical tests in a single loop from that method.
  • Test execution overhead will depend on different test runners may

Below are benchmark results from test sessions performed using TestDriven.NET.

 

1. Unit tests for a method that executes in 10 milliseconds

 

Number of tests

Calls within single domain

With cross-domain calls

1

1.3

1.4

10

1.3

1.6

100

2.5

5.1

1000

12.6

42.2

 

2. Unit tests for a method that executes in 1 millisecond

 

Number of tests

Calls within single domain

With cross-domain calls

1

1.4

1.4

10

1.4

1.5

100

1.6

3.8

1000

3.1

20.0

 

As you can see from these tables, tests with cross-domain calls take much longer time to execute – but only if a test project contains large number of such tests: 100 or more. In case test assembly contains only few tests or there are only few cross-domain calls in these tests, overhead of test execution makes cross-domain communication performance costs negligible.

I still believe that cross-domain communication is integration matter and developers should strive to write fastest possible unit tests. However, from practical point of view tests that execute code that spawns new domains most likely will not become a bottleneck in your development environment.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s