N-way parallel mail retrieval with getmail and bash

I wrote a pretty sweet script tonight. It parallelizes the getmail retrieval process, while still printing prefixes so I know which accounts download which messages. This means that instead of my mail fetching process taking sum(i1,…,in), where i is the length of time for a given mail retrieval, my fetching process now takes max(i1,…,in).


#!/bin/sh
GETMAIL='python2.3 -Wignore /usr/bin/getmail'
unwanted() {
  grep -E -v '(Copyright|getmail|Simple)';
}

echo "N-WAY GETMAIL RETRIEVER SCRIPT:"
$GETMAIL \
  --rcfile=/etc/getmail/account1 \
  2>&1 | sed -e \
  's/.*/account1................: &/g' \
  | unwanted &

$GETMAIL \
  --rcfile=/etc/getmail/account2 \
  2>&1 | sed -e \
  's/.*/account2................: &/g' \
  | unwanted &

...

$GETMAIL \
  --rcfile=/etc/getmail/accountN \
  2>&1 | sed -e \
  's/.*/accountN................: &/g' \
  | unwanted &

wait

Server “secured”

I stopped by my web host, Peer1, in order to check out my server and see if I could come up with an explanation to yesterday’s downtime. Nothing looked fishy, but it seems likely it was the stupid power cable again. So, to completely eliminate that variable, I hooked two metal ties into the grill of the nearby fans and wrapped them around the power cable’s plug. Now when I yank on the cable, instead of it coming out, it pulls the whole fucking server across the racking slide. That’s right, sysadmin soup du jour: metal ties as power cable securer.

PIDA: Python Integrated Development Application

PIDA 0.2.2 was released recently. This is truly a novel development in the Python/OSS world. What PIDA provides is a nice plugin system and the “makings” of an IDE. So, in a nice IDE you have a class browser, an integrated debugger, a profiler, maybe even a RAD-like GUI builder, an interpreter console, etc. The one piece that tends to be most controversial in every IDE is the text editor. This is one-part because UNIX people are really crazy about their text editors, but two-parts because text editors are very important programmer tools, and no one wants to learn a different text editor for every language one uses.

vim happens to be awesome for C programming, which is probably why a lot of UNIX hackers use it. But for Python, more advanced support would be nice. PIDA can run and connect to a vim server instance in order to allow you to have an “add-on IDE” for vim.

But even more interesting to me is the culebra plugin, which provides a code-completion-savvy GtkSourceView inheritor, which has the initial support for fancy Intellisense-like features.

I’ve already spoken to the developers of PIDA, and they said they would very much be interested in seeing Python Intellisense features brought to VIM. When I started thinking about different approaches to doing this, I realized that the whole OSS community could benefit from a general Python module that enhances the Python introspection features (and perhaps combines them with source code parsing) to make available nice productivity-enhancing features. I was thinking of calling this beast “Pyductivity.”

More on that later. For now, check out PIDA.

CAFTA: Is this democracy?

Check out a few links about CAFTA: 1 2 3 4 5.

Disgusting. My take? Globalization in its most ruthless form is going to be seen in the next few years in each of the countries affected by CAFTA. Then there’s going to be a long track record of failure for the theories of globalization, from NAFTA and Mexico, to WTO effects on Jamaica, to CAFTA in Central America and Dominican Republic. And then what are we all going to say? We’re still supporting these theories on “faith”? Or will we wake up and smell the coffee, and realize that “globalization,” the way it is sold now by politicians and even theoreticians, is nothing more than “handing public capital over to corporations, basically for free”.

Or, perhaps more pointedly: restoring the world order of imperial powers and their subjugated colonies.

My only fear is that by the time we realize this, globalization will have so evolved that the “imperial powers” won’t be rich governments, but rich transnationals, and the subjugated colonies won’t be third-world countries, but all countries and all people who don’t have major stakes in the ruling corporations.

Exa: a new architecture for Xorg

This is exciting news. A Trolltech developer has modified KAA, the acceleration architecture used in Keith Packard’s experimental “kdrive” Xserver, to work with the traditional Xorg tree. He announced this new development in an e-mail that makes it clear it is extremely easy to get drivers to use Exa to gain Apple/Windows-like graphics performance.

I know that the unichrome project generally doesn’t bother itself with these very desktop-oriented features (their focuses are more on MediaPCs, etc.), but I think this may be an excellent way for me to begin hacking the new modular Xorg tree I mentioned last time. If I added Exa support to the unichrome driver, would that mean transparency and full-on graphics acceleration for my X desktop, what I’ve long been waiting for? We’ll see.

I am Jack’s raging anger

I find myself studying in the Kimmel Center for my Linear Algebra final, which is this Thursday. There I am, minding my own business on the second-floor study lounge.

But then, a group of three girls shows up in the seating area right near me, and starts chatting about whatever inane topic comes to mind. They are then followed by a German foreign student whom they know, who asks them if they know words like blitzkrieg and rammstein. They respond with blank stares and Southern accents conceding they never heard the terms. The German boy responds, “Surely you must know blitzkrieg, I heard you use the term when you study German military history.” Indeed, they probably know the words, but have never heard them pronounced properly by a German, and thus cannot make the connection.

Well enough, however, slowly more and more girls file in. A few guys show up as well. They ask to have the extra chairs around the desk at which I’m working, and make a lot of noise chatting and chatting. After a few minutes, there are probably twenty people, 15 girls and five guys. And they begin their club meeting. What’s the club? Bible study.

Is there a floor dedicated to clubs at Kimmel? Yes, floor seven. Is the second floor meant for study? Yes. Could I possibly focus on Linear Algebra with talk of Leviticus and Genesis in the air? Certainly not. What the hell is wrong with these people?

Xorg goes modular, is now approachable

It seems that Xorg, as of the 7.0 release, has been split into monolithic and modular development trees. The modular model allows you to compile individual components related to Xorg separately from the whole X server, so that you don’t need to do a two-hour compilation just to work on this or that driver or this or that library.

This is good news for me. My biggest craving lately was to put my C skills to use by diving into a big project that has effects on the Linux desktop, and Xorg is certainly the biggest in that sense. However, in the past I was always put off by the huge amount of groking one needs to do just to understand the Xorg compilation process. After class is over, I’m gonna start diving in.

The Potentiality Argument

In my first philosophy class, I remember putting forth an argument for why abortion cannot be defended on the “potentiality principle,” that is, that the human fetus has the potential to be a human being, and thus must be saved.

Someone on /. recently replied to an article about stem cell research, with some nice observations about the abusers of this principle:

Now, someone might argue that a process is started at conception which would end up with a functioning human. The potential is critical. There are a few problems with that position:

  • When a fertile woman smiles back at me, there is a potential for a new human.
  • The use of condoms stops the potential for a new human being.
  • Soon, all our cells will be potential humans with a little “twist”…
  • Half of all conceptions ends soon with a spontaneous abortion. That means, according to the bible belt, that half of all people dies at an age of a few days. To be consistent, the believers should argue that half of all medical research should try to stop this mass death!

The retort often flung back is that a fetus doesn’t merely have potential to be a human being, but it is a human being. This, however, is interesting when taking this poster’s last point about “spontaneous abortions.” If you’re willing to save a fetus from a willing abortion, shouldn’t you also see spontaneous abortions (and miscarriages, for example) as equally tragic, and thus deserving of serious medical research?

Ah, the arguments against early-term abortions really amaze me. How can you be so philosophically inconsistent?

Amdahl’s Law

Gene Amdahl once applied the law of diminishing returns to computation. He pointed out that when optimizing part of a computer program or computer system, one must take into account what percent of the overall task at hand that optimization affects.

I recently read some articles comparing the speed of Python to Java, most of which concluded that about the only place that Java beats Python is in actual interpreter speed (i.e. how fast statements and parsed and executed), and that since Python opts to provide thin wrappers for standard C libraries, Python performance ends up being really good.

A good comparison of the language features between Java and Python can be found online, along with a nice comparison of code simplicity and efficiency.

I think I agree with the first author: Python is a better high-level language, and should thus be used for higher-level tasks, and especially for one-offs. What’s interesting is that a lot of people look at Python, say “Man, Python is slow, I could do this better in C,” but then forget about Amdahl’s Law. If your program is accessing the network, the disk, or any other non-CPU/non-memory resource, no amount of optimization through lower-level languages will save you an order of magnitude on performance. So why waste the programmer time, when it can be done in a few lines of Python?

(I think one forgotten benefit of Python in both these articles is SWIG: if you’re truly a performance-oriented engineer, you can always profile your code, find the bottleneck algortihm/code fragment, rewrite it in C, and wrap it with SWIG so that you can access it as an object in Python. Not hard to do, and potentially huge performance gains. OTOH, you can even write Python-accessible code directly in C, using the same abstractions the Python interpreter uses.)