July 3, 2014

The Profound Weakness of the .NET OSS Ecosystem

I’m in the process of writing up a lengthy set of blog posts for MarkedUp about the work that went into developing MarkedUp In-app Marketing, our real-time marketing automation and messaging solution for Windows desktop applications (and eventually WP8, WinRT, iOS, Android, Web, etc…)

During the course of bringing this product to market, I personally made the following OSS contributions:

I really enjoy contributing to OSS.

But here’s the kicker – MarkedUp is a very small company; we’re operating a business that depends heavily on doing distributed computing in .NET; and I have lots of tight deadlines I have to hit in order to make sales happen.

I didn’t make any of these contributions because they made me feel all tingly and warm inside – I don’t have the free time for that any more, sadly. I did it because it was mission-critical to our business. So here’s my question: in the ~15 year history of .NET, no one built a reactive, server-side socket library that’s actively maintained? It’s 2014 for fuck’s sake.

Over the course of working on this product, I was constantly disappointed by a .NET OSS landscape littered with abandoned projects (looking at you, Kayak and Stact) and half-assed CodeProject articles that are more often factually wrong than not.

This week when I started work on HyperLogLog[footnote:here’s the only known HyperLogLog implementation in C#, which appears factually correct but out of date and not production-usable] I fully expected that I was going to have to implement it myself. But I was incredulous when I couldn’t find a developer-friendly implementation of Murmur3 in C# already. The algorithm's been out for over three years – I could find a dozen decent implementations in Java / Scala and two barely-comprehensible implementations in C#, neither of which had an acceptable license for OSS anyway. I had an easier time porting the algorithm from the canonical C++ Murmur3 implementation than I did following the two C# examples I found online.

So I had to ask myself:

Do .NET developers really solve any hard problems?

We came to the conclusion early on that the right architecture for MarkedUp In-app Marketing depended on a successful implementation of the Actor model. We thought to ourselves “this is a programming model that was invented in the early 70s – surely there must be a decent actor framework in .NET we could leverage for this.”

We weren’t just wrong, we were fucking wrong. We found a graveyard of abandoned projects, some half-assed Microsoft Research projects that were totally unusable, and a bunch of conceptual blog posts. Nothing even remotely close to what we wanted: Akka, but for .NET.

So we spent two weeks evaluating migrating our entire back-end stack to Java – we already depend heavily on Cassandra and Hadoop, so being able to take advantage of 1st party drivers for Cassandra in particular really appealed to us. However, we decided that the cost of migrating everything over to the JVM would be too expensive – so we went with an slightly less expensive option: porting Akka to .NET ourselves.

Why in the hell did it fall on one developer working at a startup in LA and one independent developer in Sweden to port one of the major, major cornerstones of distributed computing to .NET? Where the hell was Project Orleans (released by Microsoft AFTER Akka.NET shipped) five years ago?

Did none of the large .NET shops with thousands of developers ever need a real-time distributed system before? Or how about a high-performance TCP socket server? No? What the hell is going on?

.NET is the tool of choice for solving rote, internal-facing, client-oriented problems

There’s a handful of genuinely innovative OSS projects developed by .NET programmers – and they largely deal with problems that are narrow in scope. Parsing JSON, mapping POCOs, dependency injection, et cetera.

The number of projects like MassTransit, i.e. projects that address distributed computing, are rare in .NET – I can’t even find a driver for Storm in C# (which should be easy, considering that it uses Thrift.)

The point I made four years ago this very day about why .NET adoption lags among startups is still true – .NET is the platform of choice for building line of business apps, not building customer-facing products and services. That’s reflected strongly in it’s OSS ecosystem.

Need a great TCP client for Windows Phone? Looks like there are plenty of those on NuGet. Or a framework for exporting SQL Server Reporting services to an internal website? A XAML framework for coalescing UI events? .NET OSS nails this.

The paramount technical challenge facing the majority of .NET developers today looks like building Web APIs that serve JSON over HTTP, judging from the BUILD 2014 sessions. Distributed computing, consistent hashing, high availability, data visualization, and reactive computing are concepts that a virtually absent from the any conversation around .NET.

And this is why we have a .NET OSS ecosystem that isn’t capable of building and maintaining a socket server library, despite being the most popular development platform on Earth for nearly 10 years[footnote: no, not going to bother citing a source for this. Get over it.].

Compare this to the Java ecosystem: virtually every major .NET project is a port of something originally evented for the JVM. I’m looking at you, NAnt, NUnit, NuGet (Maven), NHibernate, Lucene.NET, Helios, Akka.NET, and so on.

You know what the difference is? There’s a huge population of Java developers who roll hard in the paint and build + open source hard shit. The population of people who do this in .NET is miniscule.

We are capable of so much more than this

The tragedy in all of this is that .NET is capable of so much more than the boring line of business apps Microsoft’s staked it’s business on and CRUD websites.

The shortcomings of .NET’s open source ecosystem are your fault and my fault, not Microsoft’s. Do not point the finger at them. Actually, maybe blame them for Windows Server’s consumer-app-hostile licensing model. That merits some blame.

But the truth is that it’s our own intellectual laziness that created a ghetto of our OSS ecosystem – who’s out there building a lightweight MMO in .NET? Minecraft did it with Java! How about a real-time messaging platform – no, not Jabbr. I mean something that connects millions of users, not dozens.

We don’t solve many hard problems – we build the same CRUD applications over and over again with the same SQL back-end, we don’t build innovative apps for Windows Phone or Windows 8, and there’s only a handful of kick-ass WPF developers out there building products for every day consumers. Looking at you, Paint.NET – keep up the good work.

Don’t make another shitty, character-free monstrosity of a Windows Phone application – no, it’s not“new” because you can write it using WinJS and TypeScript instead of C#.

Do something mind-blowing instead – use Storm to make a real-time recommendation app for solving the paradox of choice whenever I walk into a 7-11 and try to figure out what to eat.

There are brilliant developers like the Akka.NET contributors, the Mono team, Chris Patterson, and others who work on solving hard problems with .NET. YOU CAN BE ONE OF THEM.

You don’t need Ruby, Node.JS, or even Java to build amazing products. You can do it in .NET. Try a new way of thinking and use the Actor model in Akka.NET or use the EventBroker in Helios. Figure out how to fire up a Linux VM and give Cassandra or Storm or Redis a try. Learn to how work with raw byte streams and binary content. Learn how to work with raw video and audio streams.

The only way to clean up our acts and to make an ecosystem worth keeping is to stop treating our tools like shit shovels and start using them to lay alabaster and marble instead. .NET is more than capable of building scalable, performant, interesting, consumer-facing software – and it falls on us, individual and independent developers, to set the example.

Discussion, links, and tweets

I'm the CTO and founder of Petabridge, where I'm making distributed programming for .NET developers easy by working on Akka.NET, Phobos, and more..