Are your unit tests becoming a Liability?
Does your project have a lot of unit tests? I don’t mean a good amount, I mean you really unit test the hell out of things. You add a feature, it gets unit tested to oblivion. I’m talking success cases, failure cases, corner cases, made up cases that are only possible in theory, made up cases that are impossible even in theory. Do you have all those unit tests? Well bad news, your codebase is probably in trouble.
I know what you are thinking: “You can’t be serious. Unit tests should make me feel confident my code will always do what I said it should do… right?” The likelihood is your code is doing what you tell it to do, not what you want it to do. I’ve seen a weird pattern at multiple companies, where the closer you are to 100% test coverage, the more often your codebase is immovable and inadequately tested. I’ll describe the scenario below which will make it more obvious why this happens. Note: I’m not saying to never use unit tests, but as a reminder, unit tests are meant for testing small units of very strict adherence. I give a string you give me a number, or vice versa. I pass in an enum and you give me back a simple object. These are great unit test scenarios. The problem is, at several of the companies I’ve worked for they unit test their API rather than use integration tests. That might not sound like a problem, but let me walk you through a typical API scenario.
Typical API Scenario
Say that I have a service that is a Rest API with an OpenAPI spec to back it (bonus points if it is gRPC). I have a Postgres Database for my data storage, and a Redis instance for my cache. How much of that should I unit test? My answer is any usage of the API should be backed by Integration tests, with unit tests being used for the “simple” work. Simple in this case would be things like string manipulation, model translations, enum management, etc. Anything that is in that OpenAPI spec should be integration (and system) tests only. The reason for this is because unit tests on an API require a mocking of your dependencies. In general I find mocking to be an anti-pattern which has limited uses. Once you start mocking all of your Postgres calls, what happens when you want to move off of it to something else? And how far down the rabbit hole do you get with the mocks? Do you mock an exact query, or just a call to the method that does the query? If you do the exact query, then any adjustments you make require updates in 2+ places going forward. Which really makes splitting tables or using new column structures a drag to implement. I’ve had several instances where I scrapped a large set of unit tests for API calls which had mocked the DB because they weren’t as valuable as the simple API call I can make/check in an integration test. My testing not only became easier to write/maintain, but I could later change the underlying implementation without fear of breaking 100s of tests that weren’t real issues in a deployment environment.
Teams use unit tests because they don’t create a proper integration test environment. This makes integration test setup/debugging difficult (CI/CD is important), and someone gets the idea to just mock their way through the DB and Cache calls because unit tests are easier to debug. Then you end up with 10s of 1000s of unit tests (not an exaggeration) that take hours to run locally. Code Coverage ends up near 100%, but actual testing usefulness drops down closer to 0 over time. Another killer side effect is if you ever want to change the implementation details of your API, you have to reconfigure your tests. Want to move to a different DB/Cache? Good luck as now you have to go change most/all of those tests. Also note that you are likely missing out on basic service management like auth, client usage, and other issues you only notice when you run the actual service. You don’t get that when you mock things out, and you will undoubtedly have missed one of those when you go to push your API out to production.
TL;DR: Mocking your internals will bring your development speed to a crawl. It is very likely to force you in to a single way of doing things whether you like it or not.
How to avoid this issue (or attempt to fix it in place)
Get a localized api setup that is quick/easy to run AND works in CI without any changes.
Create an integration test framework that is essentially copy/paste/replace to get a new API test going.
Create a spec for your API, deleting any that are no longer used/needed.
Add metrics by endpoint, deleting any that are no longer used/needed.
Convert unit tests to integration tests for your API endpoints.
A lot of how you get out of this is code deletion. There is very likely to be 20-50% of your code that you don’t really need at this point. That is your key to minimizing this problem.
The Hidden Step
In addition to the 5 things above, there is another thing to be thinking about if you are at the point of having several teams. You should probably create a team to take over a Domain and move it out of your monolith. I didn’t use the term until now, but it is really hard to get in to the situation above unless you are in a monolith. You can reduce the test burden with better practices, but there is a good reason companies eventually move away from a monolith. There are several terrible reasons as well, but in general your organization will be designed around Domains, and it makes sense your code is as well. You don’t need to go all in on SOA (Service Oriented Architecture) and microservice yourself in to oblivion, but you do need to be able to cut out a portion of your code that can stand alone. This will allow any split off domain to remove itself from the issues of other domains, and allow you to have a team of experts in an area rather than 1 or 2 SMEs. Ideally you start with an important, but self-encapsulated segment of the code base, possibly even with the API layer itself. Then, after you do your first domain, the next becomes easier and the next even easier. This will allow you to grow out a codebase and organization that scale to the demands of your business.
Having issues like the above, or others that you could use help with? Check out our Contact page and see our consultation Services.