December 4, 2024

Frameworkism: Senior Software Developers' Pit of Doom

Mistakes so insane and destructive that they require experience to make.

If you follow me on Twitter / X, you have likely seen several increasingly exasperated tweets from me about a legacy software project from hell. This project deserves its own series of blog posts and YouTube videos, but in service to a higher calling in the field of software I’ll reveal it now: these coding horrors all originate from our rewrite of Sdkbin.

More accurately: we decided the best path forward was to rescue Sdkbin 1.0 and not rewrite it after all, but we’re still conducting the software equivalent of tearing a building down to its foundation and starting over again.

For all of its faults, Sdkbin does generate a significant amount of revenue for us each year. We’ve kept it on life support for four years because its original author left it in utterly incomprehensible shape. In this post I’m going to talk about the plague that’s killed four years of opportunities and left this project swaddled in technical debt before it even served its first customer: frameworkism.

Bespoke Frameworks: a Telltale Sign of “Expert Beginners”

Here’s a tiny taste of what I’ve been finding during my excavation of Sdkbin, which really started in earnest on November 3rd 2024:

Generic repositories with built-in dependencies on AutoMapper and data transfer objects for an enum are symptoms of a project run by senior developers who’ve never stuck with a single project for long enough to learn from their mistakes. “Expert beginners.”

I addressed in “We’re Rewriting Sdkbin” what I was up to when Sdkbin’s code was originally written back in 2020 (i.e. trying to stop our company from getting killed as a result of COVID19 shut down measures 1-shotting 40-50% of our revenue) - thus the intricacies of Sdkbin’s plumbing remained beneath my notice for a few years and I tolerated the “working well enough” state of the project. I know this will amaze some software developers when they read this, but things like hiring people, improving our organization’s ability to run smoothly, firing myself from roles I shouldn’t have, and finding customers all rank much higher in precedence than servicing the technical debt for one small part of our over-arching business.

However, when it became apparent last year that we’d need to address Sdkbin’s considerable technical debt in order to expand our product portfolio in the near future, I started looking more seriously into it.

To give you an idea of how Sdkbin is currently constructed here’s how almost every user interaction in the system works:

sequenceDiagram
    participant Client
    participant RazorPage
    participant Repositories
    participant EFCore
    participant EntityMapper

    Client->>RazorPage: HttpRequest received
    RazorPage->>RazorPage: Start Query Loop
    loop Query for each repository
        RazorPage->>Repositories: Create Query Models
        Repositories->>Repositories: Build EF Core Query (progressively)
        Repositories->>EFCore: Execute Query
        EFCore-->>Repositories: Return Results
        Repositories->>EntityMapper: Map Entity to EntityDto
        EntityMapper-->>RazorPage: Return EntityDto Results
    end
    RazorPage->>RazorPage: Map all EntityDto to ViewModel
    RazorPage->>Client: Return ViewModel as HttpResponse

The inefficiencies created by this design are severe and high-impact - I’m going to have to save the technical inefficiencies (i.e. database performance, memory usage, HTTP response times) for their own separate post in the future because they are numerous.

For this post, let’s stick with the issue at hand: the “framework” our developer used to build this application made Sdkbin incredibly expensive to maintain, fix, and improve from a software developer-time cost.

So why was Sdkbin’s plumbing built this way in the first place?

While sizing Sdkbin’s technical debt last year I had a suspicion that our developer’s framework wasn’t something he’d come up with on the fly, but rather something that he’d copied from another project.

Looking through the commit history, my suspicions are confirmed: this developer copied and pasted his generic EF Core-powered repositories and AutoMapper + DI contraptions from a totally different project. This was a “framework” that wasn’t built for purpose - it was pre-ordained that this was how our technology stack was going to look regardless of our requirements. The design of the entities, view models, and slew of data transfer objects were all custom, but the way these were going to be processed and delivered was an automatic decision.

This is what I was referring to in my last post on whether or not Ruby on Rails-style package glue-gunning come to .NET - this is formula driven programming because that’s what the developer learned how to do on a single project and abruptly stopped there. This is the quintessence of expert beginners - get far enough along to launch a project off the ground but not far enough to learn the shortcomings of how you originally built it, so you’re doomed to repeat the same slop formula over and over again.

Preemptive Framework Creation is Premature Optimization

A question I find myself asking 20+ times a day when reviewing Sdkbin’s code - “why does this need to exist?”

/// <summary>
/// Creates element in child collection of the aggregate
/// </summary>
/// <typeparam name="TModel">Model type</typeparam>
/// <typeparam name="TEntity">Entity type</typeparam>
/// <typeparam name="TRootEntity">Aggregate type, which owns the collection</typeparam>
/// <typeparam name="TRootNotFoundException">Type of the exception thrown when parent entity is not found</typeparam>
/// <typeparam name="TElementAlreadyExistsException">Type of the exception thrown when child element already exists in collection</typeparam>
/// <param name="rootSelector">Preducate to find parent entity, which owns the collection</param>
/// <param name="createModel">Model to add</param>
/// <param name="dbSetAccessor">DbSet getter from DbContext</param>
/// <param name="collectionAccessor">Child collection getter from parent entity</param>
/// <param name="rootNotFoundException">Exception thrown when parent entity is not found</param>
/// <param name="elementAlreadyExistsException">Exception thrown when child element already exists in collection</param>
/// <returns>Created model</returns>
protected async Task<TModel> CreateChildCollectionItem<TModel, TEntity, TRootEntity, TRootNotFoundException, TElementAlreadyExistsException>(
    Expression<Func<TRootEntity, bool>> rootSelector,
    TModel createModel,
    Func<TContext, IQueryable<TRootEntity>> dbSetAccessor,
    Expression<Func<TRootEntity, List<TEntity>>> collectionAccessor,
    TRootNotFoundException rootNotFoundException,
    TElementAlreadyExistsException elementAlreadyExistsException
)
    where TRootNotFoundException : EntityDoesNotExistException
    where TElementAlreadyExistsException : Exception
    where TRootEntity : class
{
    // Find parent entity and include child collection to it with Include
    var rootEntity = await dbSetAccessor(Context).Include(collectionAccessor).FirstOrDefaultAsync(rootSelector);
    if (rootEntity == null)
        throw rootNotFoundException; // Throw when parent is not found

    // Create new entity
    var newEntity = MapperObject.Map<TEntity>(createModel);
    var logEntity = MapperObject.Map<TEntity>(createModel);
    try
    {
        // Add entity to child collection
        collectionAccessor.Compile().Invoke(rootEntity).Add(newEntity);

        // Save DbContext changes 
        await Context.SaveChangesAsync();

        // Return created model
        return MapperObject.Map<TModel>(newEntity);
    }
    catch (Exception ex) when (ex is DbUpdateException || ex is InvalidOperationException)
    {
        var serializedEntity = SerializeWithoutErrors(logEntity);
        var serializedModel = SerializeWithoutErrors(createModel);
        Logger.LogWarning($"Error during CreateChildEntity operation for entity {serializedEntity} from model {serializedModel}");
        Logger.LogDebug($"Error during CreateChildEntity operation for entity {serializedEntity} from model {serializedModel}: {ex}");
        throw elementAlreadyExistsException;
    }
}

This is a method on our the RepositoryBase<T> class I had in one of my earlier Tweets. It has zero usages in our code base. Yet, this intensely complicated method, exists just in case we need it. I know this is an artifact of the original developer copying stuff from another one of his projects, so maybe he needed it in one of those, but this code is a liability on our books from a testing, risk, and cost perspective. Why have it in the first place if we don’t need it?

The answer is simple:

  1. Building applications requires negotiating reality and making hard decisions about how to model it with software;
  2. Building frameworks requires little to no commitment - it’s pure abstraction. You get to feel productive while deferring the yeoman’s work of understanding and expressing raw business needs in code.

Writing frameworks feels good, productive, and best of all: it’s risk-free.

So before we start doing the hard work of actually having to write an application, why not spend some time considering what type of extension methods we’re going to need or what sort of data access patterns our application might use? We can submit lots of pull requests, get lots of green check marks, and even have unit tests to cover how our framework works so we get good test coverage score! Sure looks like productivity to me!

There’s a second major reason why developers lean towards building frameworks first: it’s easier to “understand” how the application works when all of its parts work the same way.

However, this argument is intrinsically stupid and self-falsifying: if two things in your domain naturally don’t behave identically in real life, i.e. a payment intent (transient activity) and a customer (permanent entity), why on Earth would you try to enforce standard behaviors and modes of expression upon them in software? Shouldn’t the software treat different things differently, each according to their needs?

Because this is all preemptive design, done before we fully understand how to model reality in our software, we are committing the cardinal sin of prematurely optimizing - expending blood and treasure on code that is likely going to need to be reconsidered or rewritten once it makes hard contact the requirements, users, and stakeholders.

If you’re going to build a framework, it should be a retroactive activity, not a preemptive one. If you find yourself having to write the same code to do identical things 3 or more times, you’ve probably found an area where a shared abstraction could be helpful. That’s a good heuristic. Trying to anticipate what an application might possibly need before you’ve really tried your hand at building it is not.

And here comes the bill - the failure of bespoke frameworks to do their jobs, ensuring the smooth development and maintenance of software, creates a tremendous accumulation of technical debt often before projects even ship.

Bespoke Frameworks: Rapid Technical Debt Accumulation Factories

In addition to importing this EF Core –> AutoMapper –> DTO –> AutoMapper –> ViewModel pipeline of death, our original developer also assumed he’d be able to ship this application the exact same way as his previous work for a different company: deliver via a set of ASP.NET Core WebApi controllers with some other, yet-to-be-hired developer doing the yeoman’s work of cobbling those HTTP APIs together into a usable interface for end users via a static HTML front-end.

Again - a preemptive design, and even worse: a design that required the cooperation of another person who didn’t even exist in our organization. Our original developer didn’t volunteer or share this assumption with anyone else on our team until he’d gotten reasonably far along on his implementation - at which point I vetoed the design and said it’d need to be done with SSR and Razor Pages.

Here’s the technical debt Sdkbin is living with today:

If you’re not familiar with this chart - this is a code coverage graph from dotCover inside JetBrains Rider.

I’ve historically not used code coverage tools much in my career; I experimented with code coverage on TurboMqtt earlier this year and found it very helpful on making sure the MQTT 3.1 spec was mostly covered in our packet-handling pipeline.

Sdkbin had, up until I deleted all of them two weeks ago, something like 85% code coverage - which is pretty good! However, as this chart shows - none of those tests are covering the Razor Pages that our end-users ACTUALLY USE to navigate around the site to do things like make payments.

Instead the tests are covering the old HTTP controllers our original developer implemented - he knew how to test HTTP controllers, but he didn’t know how or care to test the Razor Pages! Worst of all: other than these garbage repositories, there’s very little shared code between the Razor Pages and the controllers!

This is the most obvious technical debt - our test suite is worthless and tests a bunch of preemptively built code that never made it into production use. Awesome, great.

But, wait, there’s much more pernicious debt layered deeper in the application!

Nothing is obvious in this system - it’s meta-programming and side effects all the way down.

Lessons Not to Learn

Sdkbin is a source of tremendous professional embarrassment for me. I’m sharing the stories about its problems and my management failures not because I want to air grievances, but because I also see the exact same problems appear in many of my customers applications when I’m doing consulting work at Petabridge. I’m free to talk about failures that happened under my watch - that’s why I’m sharing them: to further my mission to help make great software.

A portion of people who’ve made it this far in the article will begin typing a comment along the lines of “well just because frameworks didn’t work well for Sdkbin doesn’t mean they won’t work well for me!”

That’s the wrong lesson to take here - that this is a matter of execution. The problem is that this is a matter of strategy: imagining and enforcing a universal model of expressing reality, before you’ve even made contact with it, is an utterly doomed approach to building software.

Frameworkism is an industry-wide problem within the .NET community and I suspect it’s also pervasive across most other platforms. The lesson to learn isn’t that there’s a right way or a wrong way to do frameworks, it’s to stop building them.

Conclusion

There’s merit in trying to enforce standards inside code. We don’t need 30 different ways of logging.

Where frameworks go off the rails is threefold:

  1. Preemption - trying to rigidly design how software should work before it makes contact with reality; this is an irredeemable sin.
  2. Rigidity - trying to enforce monolithic standards on a code base in areas where that approach ultimately can’t work. It’s fine to use a framework for standardizing the handling HTTP requests, concurrency, and other infrastructure-problems. It’s probably not a great idea to impose a universal model on entities, domains, user interactions, and areas where behavioral / temporal / data access divergence occurs naturally.
  3. Predetermination - I have some software patterns and libraries that I prefer to use, but here is what you won’t ever see me do: blindly recycle the same approach and code I used on the last project for the next project regardless of its requirements. If you are doing this, at least pick a template that is simple and not an opinionated mess of data transfer object spam.

The best alternative to frameworks, as I laid out in “DRY Gone Bad: Bespoke Company Frameworks”, is to re-use patterns rather than code. I won’t rehash all of the arguments there, but in-essence:

  1. You can find generalizable models of behavior that are good enough to express most of your entities and domains effectively - this should be done with as few shared abstractions as possible; Sharing patterns and methodologies can still provide reuse without anchoring your application to golden abstractions;
  2. If these models are free to diverge to accommodate natural differences (i.e. short-lived vs. long-lived entities), you will get a better result just letting the code do that vs. having a universal model with lots of configuration toggles; and
  3. Shared abstractions are still fine - what you want to avoid is being dictatorial about having “one true religion” for how things should be done in the code base. Patterns give you repeatable ways of doing things and the option to not use them when not a good fit - layered frameworks, once they’re fully baked into a given feature area, are designed to make not using them difficult to do.

As a final parting thought - frameworkism is a pit most developers fall into during the middle parts of their careers. For some reason, it seems to happen after we discover meta-programming for the first time. I’ve made these mistakes too.

Frameworkism is a simultaneous act of both experience and inexperience: taking the lessons learned the hard way from under-engineering in early career days and trying to gird against those failures by swinging too far in the other direction.

Inexperience is the hubris of frameworkism: that you can build a set abstractions smart enough to succinctly unify reality AND rigid enough to prevent other software developers on your team from making a mistake.

This is where the learning must happen - identifying the hubris and not allowing it to take root. This is where your path can diverge into expert beginner land, copying and pasting the same AutoMapper EF Core garbage into each project, or you can go the more humble route and accept that there is no universal model of behavior and trying to build one is pointless.

Discussion, links, and tweets

I'm the CTO and founder of Petabridge, where I'm making distributed programming for .NET developers easy by working on Akka.NET, Phobos, and more..