I’m trying to test some database access functions and need to have the tests run sequentially, rather than in parallel, so that the ‘earlier’ tests don’t get messed up by the ‘later’ tests, e.g. Update happening before or at the same time as create.
I’ve found out how to use Collections in C# Shared Context between Tests > xUnit.net but I can’t figure out how to use them in F# because the documentation uses classes rather than modules.
I’m open to using a different technique as long as it lets me run the tests in the order I want.
I think I can get round the problem for the basic tests by creating test data and running each test function against a different record but I don’t think that will work well when things get more complicated.
Note: I’m also using Unquote on top of xUnit, if that makes a difference, but I don’t need to if not doing so will help.
I don’t have any xUnit experience. We use Unquote on top of NUnit at work, and NUnit has some tags that lets you prescribe a test order at least within a single fixture.
That said, every time we’ve ordered tests for working with databases, we’ve later removed the ordering because it’s caused so many problems. In general, I don’t think it’s a good practice to have ordered tests, for a few reasons:
Failures typically cascade, where the first failure triggers many other tests to fail, and it can be challenging to discover the initial source of a failure, or to work through a series of failures since you can only ever pay attention to one failure at a time.
Tests can only be run via the whole suite. If you ever run a test by itself, you can’t be sure if a failure is due to a bug or just because of a missing prior test that was expected to have run.
Tests or functions being tested can more easily hide assumed preconditions that sometimes even the author is unaware of. Any such preconditions would be discovered and fixed over time when tests are being run in isolation or in a random order.
Refactoring can be harder because the test suite can be a lot more complex (where the whole suite requires understanding of how the state changes as you progress through the suite).
Probably others that aren’t off the top of my head.
Of course all of these problems can be overcome, but just oftentimes ends up not being worth it.
So while I can’t help you setup test ordering with xUnit, I’d recommend if you do order any tests, scope it to a small test fixture. Though I’d recommend if possible preferring tests that don’t need the database, or tests that can startup with a known state spun up from scratch (depending on your data access layer, perhaps by a Sqlite DB or EntityFramework InMemory test database).
I should have stated that I’m using an SQLite database, via Dapper, and the testing is being done with in-memory databases.
My original idea was to create a single database of test data which could be shared by many tests, simply in order to reduce the number of databases being spun up during testing.
After trying this for a while I’ve come to the conclusion that this was a silly thing to do because, as you say, it’s getting way too complicated and difficult to co-ordinate what’s happening, even for the simple tests that I’m using at the moment.
My new plan, for now at least, is to completely change direction and move to ‘targeted partial databases’, each containing just the table(s) needed for each user command being tested. I can then create very specific tests which can be run individually and give me confidence that what I want to test is actually being tested.
I’ll probably need to change my database migration processes to accommodate this – they needed some ‘streamlining’ anyway, so that gives me the ‘kick up the bum’ to do it – but I think it will be a better solution in the long-run – fingers crossed.