If you followed the last post you should have a TeamCity job set-up which will compile a Unity project and let you know if there was any errors, and this is pretty useful already, it will save a bunch of time and catch a few silly errors. But it would be much more useful if we could do some more automatic validation, thankfully, Unity makes this relatively easy.

Unity Test Tools

Unity provides a testing framework, Unity Test Tools which has both decent editor integration and the ability to run the tests from the command-line, it supports two different classes of test, unit tests, and integration tests.

Unit Tests

The unit test part of the Unity Test Tools is a wrapper around the popular NUnit framework, and as such it’s fairly comprehensive, the wrapper supports most of the things NUnit does, and the Unity Test Tools package comes with some good examples of tests and the various supported features.

I’ll take you through updating the TeamCity job to run these example tests.

When you run the unit tests, you’re running them against the Unity editor executable telling it to load your project, the command-line you need to run is something like this:

1
Unity.exe -batchmode -projectPath PATH_TO_YOUR_PROJECT -executeMethod UnityTest.Batch.RunUnitTests -resultFilePath=PATH_TO_RESULTS_FILE -logFile PATH_TO_LOG_FILE

This will run all the unit tests in the project, there is options available to filter tests by name or category, but given how fast unit tests are, I’ve not had cause to use them. The one new parameter there is -resultFilePath (which distinguishes itself by using an ‘=’ for some reason), this is where Unity will write the results of the tests, as an NUnit xml file.

The documentation also says that for running on windows, if you want to get the exit code correctly, you should use ‘start /WAIT’ before the command-line above. My TeamCity agent is running Windows, and we definitely want the return code (because TeamCity will class a non-zero return code as a failure).

So we can either create a new job or add all this to the job we created previously, at the moment I have my pipeline set-up so that these tests are run before the build, but that’s entirely up to you (you should put whichever one fails most often first, to minimise the time to catch a fail).

In the General Settings, we again want to specify two artifacts, but this time they’re both just single files:

1
2
%unit.test.log.file.path%
%unit.test.results.path%

The unit.test.log.file.path will be the log file that Unity writes whilst running the tests, this is similar to the log file produced during the build step. The unit.test.results.path is the results XML file that we’ll need to get TeamCity to parse, you don’t need to specify this as an artifact for the parsing to work but I like to include it for completeness.

Add a new build step, again it’s a command-line runner but this time you want ‘Custom script’ instead of ‘Executable with parameters’. (For some reason TeamCity doesn’t like running ‘start’ commands with the latter). The script you want to run is:

1
start /wait "Unity" "%env.UNITY_PATH%" -batchmode -logFile "%unit.test.log.file.path%" -projectPath "%project.path%" -resultFilePath="%unit.test.results.path%" -executeMethod UnityTest.Batch.RunUnitTests

Again we use the environment variable UNITY_PATH, and the project.path parameter that we’ll be specifying later.

Next we want to add a Build Feature, in particular, we want to add an XML report processing feature with the report type set to NUnit. What this will do is monitor a path for any matching files, and if it finds one, it will attempt to parse it as an NUnit XML file. Just set the Monitoring rules to be ‘%unit.test.result.path%’ and it will parse the output that Unity creates and give us a load of information about all the tests.

Finally we need to fill in these parameters that we’ve been using, this is the values I used:

Name Value
unit.test.log.file.path %env.TEMP%\UnitTestLog.txt
project.path %teamcity.build.checkoutDir%
unit.test.results.path %env.TEMP%\UnitTestResults.xml

With that all set-up, you can run the job, and you should get results that look something like ‘Tests failed: 3, passed 11, ignored: 6’. If you click on that text, you’ll be taken to the Overview tab for that run where you can find out even more detail about the tests, including why they failed.

It’s worth noting that by default TeamCity will stop running a job if one of the steps fails, so if you added this step to the same job as the compile, and have this step before the compile step, you won’t compile the game if the tests fail. This is normally the behaviour you want, but it’s possible to make a step run even if a previous step failed (check out the ‘advanced options’ for a step), or you can remove or mute the failing tests.

Integration Tests

The Integration Test framework is something the Unity team has created themselves, it allows you to write higher-level tests that use engine stuff (e.g. physics, rendering) that can be difficult to write unit test against. It does this by operating on test scenes, each containing one or more tests. You can read the docs for more info, but essentially each test in a scene is a node, when the scene is played all the test nodes start disabled, then one by one they (and their children) are enabled (and all other nodes disabled), the test happens, and the result is recorded. There really is quite a lot of depth to this framework so it’s worth having a poke around the examples.

You can run the Integration Tests against the editor with your project loaded, or against the compiled version of your project for a specific platform (the latter even allows you to send the results back over the network, perfect for platforms where writing files is iffy). I’m going to take you through setting up a TeamCity job to run against the compiled version on Windows.

Interestingly to run the Integration Tests against the compiled project, you still run a command against Unity.exe, the Integration Test Framework will compile to the target platform, and then run the tests. Initially I was worried about running then test against one set of compile options, and shipping another, but it actually makes a lot of sense. When you compile in Unity you basically pass a list of scenes and a couple of options to a function, for the Integration Tests the list of scenes is different (at-least I assume you don’t want to ship your tests), the options are basically the same, it will still be running the tests against the same code and prefabs etc.

You know the drill by now, either make a new TeamCity job or add this to the existing one, I have it after the compile (so if there’s a compile error I should find it in the compile part instead of in the test part).

In General Settings we (yet again) need to specify 2 artifacts:

1
2
%integration.test.log.file.path%
%integration.test.results.dir%/**/*=>%integration.test.retults.zip%

The first one is just the log file that Unity will helpfully output, it will contain details of the compile and the test run. The second one is more interesting, the Integration Test framework will produce an NUnit XML file for each test scene, so we need to recursively scan for anything in the results folder and add them to the results zip.

On to the build step, it’s pretty similar to the Unit Test one, again, it’s a Command Line with a Custom script:

1
start /wait "Unity" "%env.UNITY_PATH%" -batchmode -logFile "%integration.test.log.file.path%" -projectPath "%project.path%" -resultsFileDirectory="%integration.test.results.dir%" -executeMethod UnityTest.Batch.RunIntegrationTests -targetPlatform=StandaloneWindows

You can specify the list of scenes to compile and use by adding the ‘testscenes’ parameter like so:

1
-testscenes=TestScene1,TestScene2

If you omit this parameter (which is what I do) the Integration Test Framework, will search for any scene in the project that ends in ‘Tests’

We again need to add an XML report processing Build Feature, set the report type to NUnit, and put the following in Monitoring rules:

1
%integration.test.results.dir%/**/*.xml

Finally we need to fill in the parameters again:

Name Value
integration.test.log.file.path %env.TEMP%\IntegrationTestLog.txt
integration.test.results.dir %env.TEMP%\IntegrationTestResults
integration.test.retults.zip IntegrationTestsResults.zip
project.path %teamcity.build.checkoutDir%

You should now be able to run this job and get some results, the Unity Test Framework contains 2 scenes of example integration tests (found in the Examples/IntegrationTestsFrameworkExamples folder), some simple examples using the physics (ExampleIntegrationTests.unity), and a more involved example using assets from the ‘Angry Bots’ project (AngryBotsTests/ExampleABTests.unity).

Are You Missing Some Tests?

When I first ran the job, I was missing the tests from one of the scenes (if you’re running the tests against all the test scenes the framework provides, you should have 12, 9 from the ExampleIntegrationTests scene and 3 from the ExampleABTests scene. I was always missing the results from the first scene that ran. Watching it running on the agent, it looked as though the tests were running, but the editor window that collects the results was still initializing until the first scene-swap.

I didn’t want to get stuck digging around the Integration Test framework too much, so I came up with a simple work-around that you can also apply.

First I created a test scene named ‘___FirstTestSceneTests’ that contained a simple test that must pass, then in the IntegrationTestsFramework/TestRunner/Editor/Batch.cs file, I changed the ‘FindTestScenesInProject’ function to be this:

1
2
3
4
5
private static List<string> FindTestScenesInProject()
{
    var integrationTestScenePattern = "*Test?.unity";
    return Directory.GetFiles("Assets", integrationTestScenePattern, SearchOption.AllDirectories).OrderBy<string, string>(Path.GetFileNameWithoutExtension).ToList();
}

This should ensure that the __FirstTestSceneTests are always ran first, so we get all of the ‘real’ results. Obviously this is a bit of a hack that I’m not particularly happy with, and at some point I’ll go looking for a proper solution, but for now it will do.

Why Not Run Tests Against The Editor?

I tried for a few hours to get running the tests in editor to work (you do this by omitting the target platform command line argument), but for some reason it just wouldn’t. The build logs suggest that Unity is constantly regenerating things (most notably the UnityVS plugin), this in turn seemed to cause some sort of time-out in the Integration Test Runner before it even started running the tests. I’m not too worried about not running the tests against the editor, I was only going to include it for completeness, after-all it’s the compiled game we’re shipping.

Conclusion

You should now have an automated build pipeline set-up that will take care of all your build needs and run your tests (you are writing tests right?). Hopefully between the messages reported in the tests and the log files Unity produces, you should have enough information to fix any errors that are found.

If you’re working in a team you want to try and get as many people as possible to be familiar with the build pipeline, and setting up the notifications is a good next step, TeamCity will helpfully e-mail people who might have caused a break, but you can set up a whole host of other notifications too (and there are other options besides e-mail to look at).

If you have any questions or comments please get in touch