Please do not use Python for tooling

Programmers like to argue over programming languages and tooling. If we remove all the pretense, it usually comes down to people defending their personal choices. We have a tendency to try to justify and protect our investment in whatever languages and tools we know and use. This makes sense. But it isn’t always rational behavior.

This blog posting is mainly about the tooling for platforms like Zephyr (west) and ESP-IDF (idf.py). But many of the observations are valid for a lot of one-off and ad-hoc tooling as well.

Sunk cost

Despite what we like to think, it takes years to learn a programming language properly. I typically spend 3-4 years using a language basis before I’m comfortable with saying I know the language. Some people probably spend a lot less to build the same proficiency. Learning a programming language takes a lot of time and effort and represents a considerable investment of your personal time.

Because learning a programming language is expensive it is to be expected that people become very defensive about their choices. If you have invested 4-5 years in language X, it is going to take a lot to convince you to switch to a different language. It is easy to fall prey to the sunk cost fallacy.

Fashionable languages

A lot of people are learning Python these days. Python is highly fashionable and people tell you that if you learn Python, you will have an easy time getting a job. This is probably true. Python presents low barriers for entry, a rich flora of decent libraries for everything from web programming to machine learning, and a lenient language that lets you be a bit vague in terms of types and whatnot. Or to be more precise: a language that allows you to be a bit sloppy.

Two and a half decades ago there was a language that played a similar role. Perl was the Python of its time. For a while, much of the internet industry ran on Perl. It was the language of choice for web programming and system administration. You would also find lots of Perl programmers in places like the finance sector and in intelligence services. Places where people had lots of data to analyze or transform.

If you shave off the cosmetic differences, Perl wasn’t all that different from Python in terms of software development practice. It allowed you to write sloppy code fast and it had a lot of goodies for quickly hacking together solutions.

Perl and Python have some of the same problems. The most obvious being that neither is a compiled language and neither has static typing. Which isn’t a big deal when you are slapping together a prototype quickly, but it does become a significant problem when your project grows in complexity, you are addressing a large audience, and people start to depend on your code. Maintainance with zero help from a compiler, and no easy way to distribute software has a high cost at scale.

You can of course try to compensate by writing tests that try to cover what static typing and a compiler would have given you, but this is wishful thinking. Be honest: you are not going to be that thorough. If you were, you probably wouldn’t be using a dynamic language in the first place.

Anti-social behavior

This might provoke Python programmers, but Python is an anti-social language. It focuses primarily on the needs of developers and not as much on the user of the software. That’s not very nice. In fact, it is pretty arrogant.

It is okay to feel provoked by this statement. As pointed out previously: you have probably invested a lot of time in Python. You will be inclined to justify and defend that investment. I would urge you to take some time to think about this and try to calm your urge to come up with counter-arguments. Let it sink in and try to be open to the possibility that this is how many users experience software written in Python.

So what do I mean when I say that Python is anti-social?

If you use a language that can produce binaries the job of ensuring you have all the dependencies in all the right versions is a one-time job: it happens at build time. It does not happen every time you are running the program. What’s more: it can be automated so that the result can be distributed. You can produce a statically linked binary that merely has to be downloaded and put in the appropriate location.

This ought to be the job of the party making the software. Not the user.

Every time you run a Python program, all the dependencies have to be there. If one of them are suddenly gone, have been upgrade or if the correct Python version is not available, your program will usually fail. And worse yet, it may fail in non-obvious ways.

This happens a lot. In fact, every year we lose weeks of productivity due to Python programs that suddenly stop working. Every single toolchain upgrade for every embedded toolchain we use comes with a considerable risk of breaking our build environment so that we have to spend time repairing the tooling before we can get back to doing productive work. In fact, it is so bad that if I check out an embedded codebase a few weeks after it was checked in, and try to build it, there is a 50/50 chance that it isn’t going to build.

A lot of embedded toolchains are unacceptably brittle. If you are a professional software engineer and this doesn’t worry you, you should probably re-evaluate your quality standards. Reproducibility and repeatability matters. Especially if you want to consider yourself a professional.

Re-thinking tooling

It is understandable that people will not be particularly inclined to consider dumping their current implementation language and re-write software that kind of works. You have made an investment in learning a language and then in writing tools or applications in that language. This represents an investment, and often a considerable sunk cost. Starting over may not sound very appealing.

But you should consider it seriously. Where do you want to spend your time in the long run? Do you want to solve problems that arise from choosing Python, figuring out ways to make Python applications run predictably, reliably and without the need for constant vigilance by the user, or do you want to focus on creating value?

If you are using Python today, I think you owe it to yourself to try to do some smaller projects in languages more suited for creating hassle-free tooling that works independently of how your system is configured. Start by picking a sensible compiled language. Two excellent candidates are Rust and Go. But any language that has static typing, a decent standard library, and which compiles to (preferably statically linked) binaries will do.

In a pinch, even Java provides a better alternative as you have the ability to build all-in-one-jar files that contain all the dependencies. Not fashionable, but objectively a far less brittle option.

Rewrite some smaller piece of tooling in one of these languages. You may have to do it a couple of times: first re-creating the functionality, and then perhaps seeing if there are more idiomatic ways of structuring the code. The main objective being to learn how to produce more professional tooling.

When you start to feel a bit more comfortable with whatever language you choose, start to think about how you can create “dual use” codebases: try to structure whatever tooling you write as a library that can be used to build new tools that include this functionality. Try to think about how you can amplify the productivity of others. (If you are looking for an opportunity to become a 10x programmer, this is how you do it: you enable other people).

Closing thoughts

Having empathy for your users is important. Writing code can be a very self-absorbed activity where it is easy to only focus on your own needs. If other people struggle to use your programs, then that’s on you. Don’t tell them that they are using it wrong. Don’t waste time on figuring out baroque ways of coping with your brittle code. Don’t expect them to be understanding of you not wanting to make their lives easier.

And you can be totally selfish about this: If you create something that will make your users happier, it reflects well on you.

I can understand that asking people not to use Python is provoking. But I don’t say this out of spite or to hurt anyone. I’m saying it because I have wasted many weeks struggling to get poor tooling to work, I haven’t seen much improvement over the years and I think it is time we start to take this a bit more seriously.

A helping hand

Since I run a startup I don’t have a lot of time on my hands for side projects. But if you are a toolchain maintainer for one of the embedded toolchains I use and you want to make an effort to do something about this, I’ll see if I can set aside some time to help. I really want to see tools for embedded development improve.

If you want more comprehensive help, my startup sometimes takes on consulting gigs and we can talk about how we might be able to dedicate more effort to help you out. However I’m not really fishing for consulting gigs here.

Update

Whoa. The response to this blog posting seems to be extremely polarized. On one hand the feedback I get from a lot of embedded engineers is “thanks for highlighting the pain of Python based tooling” and statements along the lines of “it would be nice to get rid of Python”. Which is understandable since people actually experience a lot of pain from the tooling being brittle and peculiar, and being met with unhelpful attitudes.

On the other hand the feedback from people who see this as a Python hit piece is almost vicious at times. There’s also a lot of “helpful advice” that isn’t really helpful because people haven’t really taken the time to understand a) that this is a very real problem, and b) it shouldn’t be the consumer’s job to fix software that is so brittle you have to spend a lot of time trying to make it work.

It is worth observing that both sides in the polarized response validates what I’m trying to bring attention to. And it also validates the inclusion of the second paragraph under the “Anti social behavior” section. If this paragraph provokes you it kind of means that you are part of the problem and you aren’t doing Python any favors by manifesting the problem. I’m sorry, if you feel offended, but being offended isn’t helpful. For anyone.