Iterative Validation

If you’re reading this, you're likely familiar with Optimizely CMS, and you may even work with it. You might be considering upgrading your current instance. The benefits of upgrading are numerous, but that's not the focus of this article. Having conducted numerous upgrades for various clients over the past few years, we've encountered unique challenges that were addressed during the upgrade process itself. However, all the upgrades share common patterns and processes that remain consistent across different technical stacks. In this article, I'm discussing a constant aspect from a recent upgrade—an aspect that initially met some resistance, but one that we at Hidden Foundry feel strongly about.

For some clients, there is a pressing need to upgrade a codebase as quickly as possible. While this urgency is understandable, given the direct benefits such as readiness for new integrations and faster technology, there are no immediate business benefits. From the client's perspective, this is seen as a cost that should be minimized to allow a quick return to their product backlog and serving their stakeholders.

However, this approach has its downsides and can lead to issues such as a partner exceeding their promised delivery date. A simple solution is to introduce iterative validation. Iterative validation involves testing features to ensure they meet all business requirements as soon as possible. This does not mean upgrading the code in its entirety, but as soon as the solution is executable. Test it, document what’s breaking, and get it raised as a backlog item. The homepage is a common milestone for a successful first step migration. This tells the developers that all the frontend assets are being built and the backend is successfully communicating the domain to the front end. Everything else after is the business domain logic. This is the point at which developers should stop and switch focus to upgrading third-party integrations. Instead, test the pages, the blocks, the routing, and security—check all the base features that Optimizely offers out of the box first. Then move onto third-party integration.

The start of the project usually involves a single developer updating the core NuGet packages from NetFX to ASP.NET Core (the latest being .NET 8 at the time of writing). This is often a bottleneck for other developers on the team, so often we stagger the start time of the rest of the team. For most clients, this is quite straightforward. However, the real challenge arises in ensuring that features in the new codebase behave exactly as they did in the old one. We are not introducing new features; we are upgrading the underlying technology to the latest version. The digital solution should perform identically.

More often than not, the period after the homepage milestone is not straightforward. A common issue we see involves integrations using JSON. The underlying technology has changed, so updates are necessary to resolve this. These issues do not surface until you actively use the website and test the features from a customer's perspective. It’s crucial to "fail fast" by utilizing iterative validation. As soon as a section of code is updated to the latest pattern, the associated feature must be tested immediately. Issues with upgrades often only surface when tested in real-time, and because some clients wish to move quickly, they are only brought to attention near the project's end. This results in missed deadlines. You might argue that whether an upgrade has caused a feature to change behavior is irrelevant, whether discovered at the beginning or the end of the project. However, remember that in a user journey, features depend on one another. Not testing features as you go through the code can introduce an incorrect foundation for its dependent features.

Without going into specifics for NDA reasons, we recently encountered an issue with a third-party integration. The client decided to proceed in a particular direction with it. Without testing the feature associated with it, more work was built on top of this decision, and eventually, a problem was discovered with the underlying direction while we were testing the behavior of the feature. This led the team to revisit that original decision and take a different course, resulting in additional time needing to be spent.

If you are a customer, as a partner, I implore you to consider iterative validation as an approach in your upgrade development. Test as much as you can. Involve your testers and stakeholders as early as possible; do not wait until your development is complete to start your quality assurance.

If you appreciate my thoughts and our approach, please do get in contact with us for a chat. We’d love to discuss your plans and see if there is an opportunity for us to help you achieve your goals more efficiently.