Shipping the Second System

In 2015-2016, the team embarked upon the task of re-envisioning its entire backend technology stack. The goal was to build upon the learnings of more than 2 years delivering real-time web content analytics, and use that knowledge to create the foundation for a scalable stream processing system that had built-in support for fault tolerance, data consistency, and query flexibility. Today in 2019, we’ve been running this new system successfully in production for over 2 years. Here’s what we learned about designing, building, shipping, and scaling the mythical “second system”.

The Second System Effect

But why re-design our existing system? This question lingered in our minds a few years back. After all, the first system was successful. And I had the lessons of Frederick Brooks accessible and nearby when I embarked on this project. He wrote in The Mythical Man-Month:

Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs.

When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one. The result, as Ovid says, is a “big pile.”

Were we suffering from engineering hubris to redesign a working system? Perhaps. But we may have been suffering from something else altogether healthy — the paranoia of a high-growth software startup.

I discuss’s log-oriented architecture at Facebook’s HQ for PyData Silicon Valley, with’s VP of Engineering, Keith Bourgoin.

Our product had only just been commercialized. We were a team small enough to be nimble, but large enough to be dangerous. Yes, there were only a handful of engineers. But we were operating at the scale of billions of analytics events per day, on-track to serve hundreds of enterprise customers who required low-latency analytics over terabytes of production data. We knew that scale was not just a “temporary problem”. It was going to be the problem. It was going to be relentless.

Continue reading Shipping the Second System

Expanding my mind, once more, with functional programming

The Structure and Interpretation of Computer Programs (SICP) is a classic computer science text written by Gerald Jay Sussman and Hal Abelson. It is widely known in the computer science community as the “wizard book”. It intends to teach the foundations of computer programming from “first principles”, illustrating programming language design using Scheme, a dialect of the Lisp language.

In this context, from Aug 26 – 31 2018, I am taking a “think week” to reflect on my relationship to computer programming.

I am spending this week in Chicago with David Beazley (@dabeaz), where we will be spelunking through the land of this famed SICP textbook via Racket, a modern functional programming environment one can use to program in — and even extend — Scheme and many other languages.

The course will also (of course) involve some Python. This will be a fun follow-up to an earlier course I took with Beazley in 2011, “Write a Compiler (in Python)”. I can’t believe I wrote the code for that course over 7 years ago.

Back in 2011, I took “Write a Compiler (in Python)” with David Beazley. A handful of long-time professional programmers and Pythonistas, locked in a room together for 5 days, hacking away on a Python compiler for a Go-like language. It was so much fun. It proved to me that I loved programming! I’m the one whose head is exploding on the left.

How I’m thinking about this course

I have long identified primarily as a computer programmer. I studied Computer Science at NYU, and I currently read about programming languages, paradigms, and design patterns all the time. I have read way more technical programming books than any other category or genre of book.

But, I’m also someone who is interested in the business of software, and leadership of software teams, in a sort of secondary way to my love of software itself. Business books — and particularly books about high-growth startups and their teams — make up my other big obsession. But, in the last several months, I’ve seen my relationship with software change in a number of ways.

Continue reading Expanding my mind, once more, with functional programming

Flow and concentration

From Good Business, by Mihaly Csikszentmihalyi, the author of Flow.

Another condition that makes work more flowlike is the opportunity to concentrate. In many jobs, constant interruptions build up to a state of chronic emergency and distraction.

He goes on:

Stress is not so much the product of hard work, as it is of having to switch attention to from one task to the other without having any control over the process.

Continue reading Flow and concentration

Public technical talks and slides

Over the years, I’ve put together a few public technical talks where the slides are accessible on this site. These are only really nice to view on desktop, and require the use of arrow keys to move around. Long-form notes are also available — generated by a sweet Sphinx and reStructuredText plugin. I figured I’d link to them all here so I don’t lose track:

Continue reading Public technical talks and slides

Software planning for skeptics

Engineers hate estimating things.

One of the most-often quoted lines about estimation is “Hofstadter’s Law”, which goes:

Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law.

If you want to deliver inaccurate information to your team on a regular basis, give them a 3-month-out product development timeline every week. This is a truism at every company at which I have worked over a varied career in software.

So, estimation is inaccurate. Now what?

Why do we need a product delivery schedule if it’s always wrong?

There is an answer to this question, too:

Realistic schedules are the key to creating good software. It forces you to do the best features first and allows you to make the right decisions about what to build. [Good schedules] make your product better, delight your customers, and — best of all — let you go home at five o’clock every day.

This quote comes from Joel Spolsky.

So, planning and estimation isn’t so much about accuracy, it’s about constraints.

Continue reading Software planning for skeptics

Lenovo and the new Linux desktop experience

I am a longtime Thinkpad and Lenovo user as my preferred laptop for Linux computing and programming.

The Lenovo X1C 2016 4th Generation Model is my latest Linux laptop

For some context, I’ve been running Linux on my desktop and laptop machines since ~2001, and started using Thinkpads in this role starting with the famous Thinkpad T40 (2003), one of the first laptops that provided good Linux support, a rugged design, portability, power, and an excellent keyboard.

I then moved through a few different Lenovo models: the T400 (2008), the T420s (2011), and the X220 (2011).

I spent a couple of short stints in-between — which I always regretted — on other PC laptop models, including HP and Asus. I upgraded from the T420s to the X220 after coming to the realization that portability and power consumption mattered more to me than the 14″ form factor, and that I could easily expand the X220’s limited hard drive with a 512 GiB SSD.

Since 2013 or so, the X220 has been my main programming/Linux machine. The X220 was my favorite Thinkpad model of all time, despite some flaws. I’ll discuss my Linux desktop experience with the X220 briefly, and then go on to my experience with my current model, the Lenovo X1 Carbon 2016 model (4th Generation).

Continue reading Lenovo and the new Linux desktop experience

Specialists in medicine: not the problem

The NYTimes published a woefully misguided piece on the “specialist stranglehold” on modern US medicine.

So, doctors are to blame for setting the prices for reimbursement by private health insurance companies? And they are also to blame for pursuing extra years of low-pay resident/fellowship training so that they can do advanced procedures and be paid decently for their work? And, they are also to blame for “defending their turf” within a specialty — that is, for specializing at all?

And, this article is even written by a doctor?! Why, yes, of course, it is!

I think medicine may be the world’s most self-hating profession. Trapped inside a system that takes advantage of their altruism, ridiculous work ethic, and decades of training, they can’t help but blame themselves even as the capitalists around them exploit them.

Continue reading Specialists in medicine: not the problem

In support of net neutrality

I wrote a letter in support of net neutrality and Title II classification of Internet Service Providers to the FCC. For background on this FCC vote, you can read this Arstechnica explainer.

You can add your own comment in support of net neutrality to the FCC at the URL To clarify some terms:

  • “net neutrality” is a term coined by Tim Wu (author of “The Master Switch” and “The Attention Merchants”) which describes a legal principle that “Internet service providers and governments regulating the Internet should treat all data on the Internet the same, not discriminating or charging differentially by user, content, website, platform, application, type of attached equipment, or mode of communication.”
  • “title II” is a part of the Communications Act of 1934 that establishes that certain forms of communication infrastructure are “common carriers”, which means that in delivering Internet service, the ISPs “cannot discriminate [content/services], that is refuse the service unless there is some compelling reason.”

Continue reading In support of net neutrality Culture: Ethics & Identity

In September 2013, my startup,, had just raised Series A capital, and had just begun growing its team rapidly, from a small group of fewer than 10 to over 40 employees now. In the past several years, I have run’s fully remote engineering, product & design team.

Back in 2013, we had achieved initial product/market fit, initial revenue, and had already established a kernel of a product and engineering culture. I knew the company would change, but I wasn’t sure exactly how. Meanwhile, I had just recently read “Reasons & Persons”, a book on ethics and identity by the philosopher Derek Parfit. Though his ideas focused primarily on individuals, they influenced the way I thought about my business, my team, and its evolution over time.

What follows are my speaker notes from a talk I gave to my team to discuss the issues of Ethics and Identity central to’s culture:

Origin of this talk

  • turned 4 years old in May 2013
  • I reflected after our Series A round
  • I read a book about ethics/identity, Reasons & Persons
  • Realized some interesting concepts apply to firms, too, different takes

  • “An analytics platform for large media companies?”
  • “A startup founded originally in 2009 at Dreamit Ventures?”
  • “A team of employees?”
  • “A specific configuration of tech and code?”

What is, really?

Are we:

  • our history?
  • our appearance to customers / press?
  • our employees (or founders)?
  • our technology / product?
  • our shareholders? (huh?)

Ship of Theseus

What is the Ship of Theseus?

  • They took away the old planks as they decayed
  • … putting in new and stronger timber in their place
  • One side held that the ship remained the same,
  • … and the other contended that it was not the same.


Continue reading Culture: Ethics & Identity

The value of money in a technology career

Michael O. Church wrote an essay awhile back called “Why programmers can’t make any money.” The post is no longer on his website — for some strange reason — but you can have a look at the archived version here.

If you don’t wish to read his post, this quote will give you the summary.

When the market favors it, junior engineers can be well-paid. But the artificial scarcities of closed allocation and employer hypocrisy force us into unreasonable specialization and division, making it difficult for senior engineers to advance. Engineers who add 10 times as much business value as their juniors are lucky to earn 25 percent more; they, as The Business argues, should consider themselves fortunate…!

I empathize with his thoughts, but I have struggled — for years, now — to understand the author’s conclusion.

If we want to fix this, we need to step up and manage our own affairs. We need to call “bullshit” on the hypocrisy of The Business, which demands specialization in hiring but refuses to respect it internally. We need to inflict […] artificial scarcity.

I decided to (finally) publish this response today because I have seen artificial scarcity play out in another industry; my wife is a medical doctor in the US. Are we to believe that programmers should establish artificial scarcity in the same way that doctors have — with political organizations like the American Medical Association and credentialing via something equivalent to medical school and board certification?

Continue reading The value of money in a technology career