Developer practices for long-term product maintainability
Rules of thumb vs data-driven development
It is all too easy to focus on feature X that has to be delivered in 2 weeks and lose sight of the long-term, especially when everybody in the room is fixated on that. My experience is that it's more important to focus on the future instead and not make sacrifices that will bite later.
But it's also a hard sell usually: why spend more time to implement feature X when we can do it quicker? It goes back to probability: the increase in development cost is a sure thing: if something takes 4 weeks instead of 2 then there is a near-100% chance that it costs the team 2 weeks of work.
But the benefits are less certain: sure, the more complicated way makes it easier to debug future issues, but how likely they are? Or maybe it results in a cleaner architecture that is easier to extend in the future, or less moving pieces that makes a security incident less likely.
How can you sum these low-probability beneficial events so that you can reason if the additional development time is worth it?
In my experience people vastly underestimate these probabilities and also the impact. This reminds me of the black swan events described by Nassim Taleb. In his book, he argues that these events are uncomputable not just because they are so rare any measurement is inherently inaccurate (what is the probability a credential committed to Git is found by a hacker?) but also because you can't even enumerate all the events that can happen (a novel security flaw found).
Of course, there are always clear cases. A feature that takes 6 months to implement instead of 2 weeks for a very narrow benefit will clearly not worth it. Similarly, spending 11 days instead of 10 for a clearly superior solution is also a no-brainer. But most of the real-world decisions are in the middle.
There are a couple of best practices that I follow that in my experience contribute to the long-term longevity and especially the long-term changeability of the product.
Maintain data consistency
The database should be without inconsistencies. If there are invoice line items and then a grand total, they should be always in sync, no exceptions. One of the worst developer experience is to get the same information from the database in two different ways and getting different results.
Developers should take extra caution to keep the database neat and tidy as that is the core of the system.
Focus on the edge cases
Related to data consistency, I like to spend time to handle edge cases: what if this user was just created and the last login date is null? Or the GPS position is unknown for an online device? Or if a system sends events out-of-order?
In my experience these low-probability edge cases add up and quickly result in a faulty product.
Separate components based on responsibility
Each component should have their responsibilities, and then any logic that is not related to those should be outside the component. The backend is sending real-time events to a client as they appear: is it the responsibility of the backend to maintain message ordering? If not, implement it on the client. One component handles clients and another purchases? Don't put "last buy date" into the former as it should not know that clients can buy things; it's not its responsibility.
Because of this, ease of implementation is irrelevant: I see a tendency to implement logic in the place where it is easiest to do so and not where it logically belongs. It is worth not doing that: respect the responsibilities of each component, put logic where it belongs and in the long term you'll end up with a maintainable product instead of a mess.
Develop against contracts
Using APIs is everywhere and for a good reason: they make thinking about systems easier as the boundary between components is well-defined.
Each component should have some sort of a contract: an API schema, a document describing how it works, or something similar. Then everything else is implementation detail, developers should be free to change it in any way they like.
I get two types of pushback against this. The first is that it's easier to not respect this separation: read directly from the database, or create resources that the other component should manage instead of enlarging the API with these operations.
The other pushback is along the lines of: "we are a small team and we can communicate efficiently to coordinate changing the implementation".
In my experience violating this bites the earliest. Even if I'm working alone I'm defining interfaces between components I write. If everything can access everything else then it takes no time for complexity to increase into unmanageable proportions.
Maintain invariants and define processes
Agile says "people over processes" but it's usually taken too far to the extreme. Processes give assurances: if everybody runs the tests before merging a PR then the develop branch is always passing. Or if the database documentation is always updated when the schema changes then I can always just go there and see how to get data from.
Generic features instead of narrow ones
When there is a request to "implement X" it's very easy to focus on that specific use-case and end up with a narrow implementation. It is better to focus on the longer term and come up with a solution that solves not just "today's" problem but also "tomorrow's": a more generic solution that handles a whole class of problems where "feature X" is just a specific case.
There is a concept of "over-engineering": why spend more time solving imaginary future problems when there is a good chance we won't need them? In practice I find these type of "wasted abstractions" rarely and the opposite more frequently: now we have this mechanism in the product and then people find novel uses for it. I think innovation is mostly fueled by finding these good "over-engineered" features that open many doors.
Make sweeping changes
I have a history of rewriting applications when the benefits were clear: I migrated a sizeable codebase to TypeScript, an AppSync project to JS resolvers, rewrote some core features that changed most of the files, and so on.
People generally don't like to make big changes and instead go for small additions: let's add an endpoint here, a field there. It is safe to do so as usually clients don't mind when they get an extra field in an object or have access to an additional API. But it also sacrifices future maintainability: if nothing can be removed or meaningfully changed then it's a road that ends with a product that is so complex nobody dares to change it in any meaningful way.
