Solving “accidents” and “essences” of programming with better languages

Note: this post was written in January 2007, but it has stood the test of time.

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds castles in the air, from air, creating by exertion of the imagination.

Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures. Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

… The computer resembles the magic of legend in this respect, too. If one character, one pause, of the incantation is not strictly in proper form, the magic doesn’t work. Human beings are not accustomed to being perfect, and few areas of human activity demand it. Adjusting to the requirement for perfection is, I think, the most difficult part of learning to program.

Frederick Brooks, author of the The Mythical Man-Month, distinguished between two problems in software development: the accidental problems and the essential ones. You can read Brooks’ watershed article on this topic, “No Silver Bullet”, freely available online in HTML or PDF, or as part of his book.

Those in the industry like to argue that we have made huge leaps in terms of the “accidental” problems in recent years. Programmer productivity is better with the help of better software revision systems, but more importantly there are RAD tools, like user interface builders and powerful IDEs, like Eclipse and Visual Studio. Do I think we’re headed in the right direction?

In a sense. I think those tools are useful, but only because I think all we can do is shrink the accidental problem as small as possible, and then do our best to tackle the essential. The major win in terms of the accidental and even the essential is high-level programming, and I mean really high level, like Python.

Why is this important? Well, Brooks once talked about how PL/I was a great software engineering language because the statements written to the compiler are “pretty close” to the “thought stuff” the programmer is working with in his head. But PL/I is nowhere near there (neither is C++, or even Java or C#). However, Python heads in that direction.

In my head, unless I really am immersed in C code all the time, I don’t think of things like types, memory management, or, God, pointer arithmetic. When we solve problems, we solve it in something computer scientists have used for years: pseudocode. Pseudocode is nothing more than code that expresses an algorithm or approach without worrying about the gory details of the underlying hardware.

Python isn’t quite pseudocode, but it’s closer than anything else I’ve seen. And that’s a huge help to productivity not just from the point of view of the single developer, but, I’d argue, also from the point of view of a team of developers. Code readability is hugely important in team environments. C and Perl can be very unreadable, which makes them ill-suited to team development. Java and C++ are better, but even they suffer from some readability problems, where the actual solutions can be masked by “best practices” and strange ways of exploiting constructs of the language. Python can be hacked too, but it tends not to be, and it tends to have very high readability.

But what about the tools mentioned above (Eclipse & Visual Studio)? I often wondered about all the *nix programmers (including many of my past Computer Science professors) who still load up terminal with vim or emacs for their programming needs. Wouldn’t they benefit from the latest and greatest in IntelliSense, Refactoring, CallTips, SaneOnlineDocumentation, and any other CamelCase ideas I can think of for a development environment?

Well, most *nix users would say that stuff is unnecessary–and in many senses, they are right. The essential problem remains, no matter how fancy your IDE.

A lot of *nix developers shrink their accidental problem on a “as-needed” basis, by coding scripts or plugins or configurations for their highly-extensible text editors. Or incorporating tooling that is freely available online in the wider F/OSS community.

But nowadays, productivity is starting to be a concern even in the F/OSS world, where users have traditionally stuck with the old “mortar and pestle,” or, should I say, “gcc and gdb”.

But Python offers a nice, alternative path, I think. The accidental problem is worth shrinking. But I think it can be done simply by one major refocus:

(1) make the language usable, not the tool. IMO, this is already done with a high-level language like Python.

…and…

(2) create source code analyzing tools that integrate with development environments to make the language-bearer more productive.

You can see enormous success in (2) with the Eclipse IDE project for Java programmers. My main problem: Eclipse’s focus is on a language that isn’t very usable, that is, Java. Java is certainly better than C, but still, it has significant problems that stop it from connecting me (the programmer) with my problem-solving ideas — that is, the “thought-stuff” of programming, as Brooks called it, above.

At my new job, I work with Java eight hours a day, but my code just doesn’t read and work like my Python code, which is much closer to the underlying algorithms and approaches. Java certainly pushed forward OOP on the masses of computer programmers, but it did so without incorporating other non-OOP concepts, whether from the dynamic language, scripting language, or functional language traditions. One simple example: having Lists and Hashes as built-in types, with tons of syntactic sugar, for crying out loud! Most problems break down to List and Hash problems, after all.

What’s more, programmers have learned that OOP, per se, is no panacea. I’ve seen many convoluted OOP designs that I wish were written functionally, or even just procedurally, and which might have been more performant (and more comprehensible) if they had been written in a non-OOP style.

As the Python community says, “Practicality beats purity”. The “legacy” OOP software engineering community thinks that there will be a silver bullet (OOP! AOP! XP! MDE!). But: I think just getting programmers closer to the “thought-stuff” — via high-level languages like Python, and lightweight tooling around the editor and the command line — will be an effort with better rewards.

---

Update from the future in June 2020, 13 years later: I was right. I’ve incorporated all my years of experience writing professional Python into The Elements of Python Style, which you can read online. If you’re still stuck in a programming language like Java, C#, or C++ which is far away from the “essential”, you’re well advised to check out my Idiomatic Python Resources compendium. I personally can’t believe just how successful Python has become in the intervening 13 years. Its massive success surprised all of us, the early champions and fans, included. And I don’t personally miss my RAD tools one bit. I only ever open an IDE when working with Java code, specifically, IntelliJ. But for every other language, vim and the “UNIX IDE” suffices. Plus, with projects like VS Code, Onivim 2, Language Server Protocol (aka langserver, LSP), and Jupyter, you can “have your cake and eat it, too”. Also: Brooks was right. Despite the tooling improvements (most notably, cloud computing), the essential problem of programming remains.