Improving a surface interpretation of “big data”

A silly little piece appeared in The New York Times discussing a hypothesis of a Harvard economics professor that Apple might slow down its operating system ahead of major product releases in an attempt to encourage consumers to upgrade.

One of his students used Google Trends data to investigate this hypothesis. In the article, two graphs are compared — one that shows Google Trends search volume for “iPhone Slow” and the other for “Samsung Galaxy slow”.

iphone_slow

It is shown that the spikes in searches for slow operation of Apple’s products seem to correlate with new iPhone release dates, whereas there are no search spikes in the data for the Samsung Galaxy.

samsung_galaxy_slow

These graphs are horribly misleading on their own. Both products have grown in popularity over the years, so the increase in search volume over time reflects nothing more than their widespread mainstream popularity. This could have easily been removed from the graphs by adjusting these trendlines relative to the “base” searches, e.g. “iPhone” and “Samsung Galaxy”. In the graphs as shown, it’s hard to tell whether little spikes are actually hidden within the compressed and precise trendline for the Samsung Galaxy.

Continue reading Improving a surface interpretation of “big data”

Are “tweetstorms” a new form of publishing?

Fred Wilson has just published a #tweetstorm about #tweetstorms. It’s my view that in order for a new media concept to “become a thing”, someone needs to complete The Strange Loop — discussing the concept using the concept.

Strange Loop

Listicles became a thing for me when the NYTimes made one about BuzzFeed, 15 Crazy Facts About BuzzFeed That Will Blow Your Mind. GIFs became a thing when Clickhole wrote the hilarious satire article, 16 Incredible GIFs We Would Make If We Knew How.

Fred Wilson’s #tweetstorm isn’t quite as meta or funny, but it’s clear it’s now a thing, too.
Continue reading Are “tweetstorms” a new form of publishing?

Truth on tap

Some people have put together an alternative to Wikipedia called Conservapedia. But, I won’t grace it with a link. I’d rather not let the Internet become more dangerous as a form of mind control.

The site is meant to provide explanations of world-wide phenomena in conservative terms. This brings full circle the blurring notion of truth in the Internet Era, as was described quite well by Clay Shirky in his essay, “Truth without scarcity, ethics without force.”

For example, the many-thousand word article on “Public Schools” includes a section entitled “Gender Disparity”. It explains that “Public schools as of late have seen girls’ scores soar above boys’ because schools have been geared toward the needs of girls”. It goes on:

Schools seek to emasculate boys by preventing healthy roughhousing and having psychologists put boys on drugs such as Ritalin. Then boys often come to hate school because radical feminists seek to prevent men from being men and forcing males to go through counseling to “discuss their feelings” and other liberal hogwash treating all students as if they were female. Colleges, because of this trend, see a trend of 60/40 female to male ratio because of feminist drivel such as romance novels in literature and ineffective therapy and attempts to push feminine traits on boys and young men making them frustrated and fed up with the system unless they agree to the school’s desire to become effeminate.

Now, certainly, there are valid conservative arguments against public schools. You don’t have to look far to find them. You might feel that a public school is a poor use of taxpayer dollars, is a violation of parental child-rearing rights, or is a form of mass indoctrination.

But, a feminist conspiracy?

Continue reading Truth on tap

The content trading desk

Note: At the time of this post’s publication, I was the co-founder & CTO of Parse.ly, the company behind a a real-time analytics platform used by top publishers such as The Wall Street Journal, Bloomberg, Slate, Fortune, TechCrunch, and over 2,500 others. This post was informed by that work experience.

My first job out of college was as a software engineer for Morgan Stanley. This 50,000 employee firm employed tens of thousands of software engineers. Though the most important employees were the traders, it was clear that software ran the place.

For anyone who has worked on Wall Street in the last couple of decades, this is no surprise. It may be surprising to people from outside the industry, who perhaps still associate Wall Street with traders wearing funny jackets calling out orders across a busy NYSE trading floor. That still goes on (though, probably not for long). Most of the trading activity on Wall Street is heavily computerized. Orders are placed and fulfilled mostly by machines.

Wall St Trading Floor

Once the markets went digital, data analysis became a much more rigorous discipline on Wall Street. Whereas before, a trader’s primary edge was being personally connected to the most important players (think Gordon Gecko), today’s traders seek edge in systematic prediction models. In other words, other machines analyze the orders being placed and fulfilled on the market, and yet others try to detect patterns or make predictions based on all of this data.
Continue reading The content trading desk

The End of Philosophy?

David Brooks has written a column for the NYTimes entitled, “The End of Philosophy”. The basic thrust of the article is that moral reasoning is less about reasoning and more about intuition. In other words, morality is more like aesthetics than logic.

A representative section:

Think of what happens when you put a new food into your mouth. You don’t have to decide if it’s disgusting. You just know. You don’t have to decide if a landscape is beautiful. You just know.

Moral judgments are like that. They are rapid intuitive decisions and involve the emotion-processing parts of the brain. Most of us make snap moral judgments about what feels fair or not, or what feels good or not. We start doing this when we are babies, before we have language. And even as adults, we often can’t explain to ourselves why something feels wrong.

In other words, reasoning comes later and is often guided by the emotions that preceded it.

The major hole I see in Brooks’ article — and argument — is what he himself recognizes here:

Moral intuitions have primacy, Haidt argues, but they are not dictators. There are times, often the most important moments in our lives, when in fact we do use reason to override moral intuitions, and often those reasons — along with new intuitions — come from our friends.

It’s true that moral intuitions may have evolutionary (or other) roots distinct from reason, but that’s why they’re called “intuitions.” Brooks recognizes that at the “most important moments in our lives”, we cast those intuitions aside. Well, doesn’t that suggest that there exists a moral “right answer” outside our intuitions? Perhaps people should use reason to override impulse at more mundane moments of their lives, too. For example, when deciding whether one deserves those alligator skin shoes, or whether the dying children in Africa might be better candidates for that money.

There have been many attempts in recent years to justify the less rational sloppy moral thinking of individuals by pointing to evolution and saying that an individuals’ beliefs are just derived from their primordial roots. I simply disagree with this line of reasoning. The fact that you can override your moral impulses means that at times you must! I much prefer to frame my decisions in terms of Jean-Paul Sartre’s concept of “radical” or “unlimited” freedom. And with that freedom comes responsibility.

Brooks quotes Haidt,

The emotions are, in fact, in charge of the temple of morality, and … moral reasoning is really just a servant masquerading as a high priest.

My analogy is that moral intuitions are more like the inmates in a psychotic ward. In people who don’t think their moral choices through, “the inmates are running the asylum.”

America’s “first black president”

Elizabeth Alexander of Salon writes about the mis-used Toni Morrison quote that Bill Clinton was America’s “first black president.” This quote was repeated during the Democratic Presidential Debates — which was the first time I heard it. You can read Toni Morrison’s original article from the New Yorker, but Alexander’s analysis concisely illuminates the key point. Surprise, surprise: this quote is always used out of context, and never in the way Morrison intended.

Her words have been used frequently and almost always out of their original context, as a way of signaling Bill Clinton’s supposed comfort with and advocacy for black people, to the extent that Hillary Clinton even attempted to joke that she was “in this interracial marriage.” …

[Instead, Morrison] questioned the pitch of Starr-fueled hysteria, and said: “Years ago, in the middle of the Whitewater investigation, one heard the first murmurs: white skin notwithstanding, this is our first black President. Blacker than any actual black person who could ever be elected in our children’s lifetime … The always and already guilty ‘perp’ is being hunted down not by a prosecutor’s obsessive application of law but by a different kind of pursuer, one who makes new laws out of the shards of those he breaks.” …

Morrison was not saying that Bill Clinton is America’s first black president in a cute or celebratory way, nor was she calling Clinton an “honorary Negro.” Rather, she was comparing Clinton’s treatment at the hands of Starr and others with that of black men, so often seen as “the always and already guilty ‘perp.'”

I have to ask the obvious question: does our media even do its basic job anymore? Can we rely upon it to do anything right? Or will it continue to take quotations out of context and mis-represent ideas like these?

American Thinker?

I just stumbled across this online newsletter called “The American Thinker.”  I will not link to it because I refuse to give this piece of trash a boosting in any search engine ranking.

I read an article on there (the first one I saw) called “Cultural Marxism.”  Its thesis is that though self-proclaimed communists hardly exist in America, the “new left” is organized around Marxist principles and is just a form of “masked communism.”  Here’s a nice quote:

Both communism and the New Left are alive and thriving here in America.  They favor code words: tolerance, social justice, economic justice, peace, reproductive rights, sex education and safe sex, safe schools, inclusion, diversity, and sensitivity.  All together, this is Cultural Marxism disguised as multiculturalism.

Hah hah hah hah hah hah hah hah hah Hah hah hah hah hah hah hah hah hah Hah hah hah hah hah hah hah hah hah Hah hah hah hah hah hah hah hah hah …. it would be funny, if only it were made up.

Search for it yourself.  This is what the American Right reads and how they frame the progressive movement.  Absolutely stunning.

Danish Cartoon Follow-Up

I was going to do a follow-up post on the Danish cartoon situation, but didn’t get to it yet. Instead, I replied to roman’s post on the subject, and realized it was the length of a post on this site anyway. He asked:

did maheen ever give you a reply?

i know that their protest went ahead anyway, and i heard she started tearing when she addressed the faithful assembled in front of her.

i also heard, admittedly from one of the islamic center kids, that the way the objectivist club went about discussing their cartoons was pretty racist…

but i’d be interested to know how she replied.

To answer his question, no, Maheen never replied to that e-mail. If she does, I’d be glad to post it on my site.

I saw the reporting on the protest, but as you probably know, the Objectivist club was also asked to not display the cartoons by the administration. They displayed blank panels instead, and so from what I heard a lot of the discussion of free speech issues got drowned out by the anger over the administration’s blanket censorship. Unfortunately, the administration has every right to censor an event held on its campus, it being a private university, so there isn’t much to say there.

If they did discuss the cartoons in a racist way, then shame on them. From WSN, I heard that the basic gist was that a couple of the Ayn Randian intellectuals claimed that all religions are basically bogus, and that they ascribe to the only true religion, which they called “rationality”, or somesuch. Of course, this is utter tripe that you can feed to the dogs — “Objectivism” hardly even attempts to answer the existential questions that religions address, so that any form of atheism arising from it is hardly stronger than any fundamentally faith-based belief system.

My main issue in that e-mail and over this situation in general is a principle. The basic principle is that you can’t just claim that you have a right to see something censored because it offends your religion. I think that the freedom of information flow — no matter what kind of information that is — definitely trumps maintaining tolerance for religious sensibilities. If someone publishes things that can be taken as offensive by Christians, Muslims, Buddhists or Hindus, I don’t want the precedent set that we, as a society, will simply remove that text from our literature. It’s true that here, with the cartoons, we have something that is very clearly anti-Islam and racist, but where do we stop in the protection we grant to speech we don’t like? What if the next time, it is a cartoon portraying Jesus as blinding his followers, or an evolutionary analysis of religion? We have to protect the speech we don’t like, so that the canon of “acceptable things to publish” doesn’t shrink everyday.

I Just Didn’t Know: Republicans invented the reality then, and are doing it now

I just didn’t even know this happened.

I knew about the Swift Boat ads, and I knew about a bunch of POWs trashing Kerry’s war record only because he spoke out against the war when he returned back to the states. But I simply did not know the extent of this. Take a look at this great student-written article about it. Apparently a broadcasting company named Sinclair pushed to have a documentary named “Stolen Honor” broadcast on TV just days before the 2004 election. To see what the documentary is about, check out NewsMax’s “news report” on it. I put that in quotes because it begins sounding like an AP article but ends just accepting the thesis of the documentarian hook, line and sinker, and making the creators of this documentary seem heroic.

The basic premise of the film isn’t that Kerry did bad things during war. The basic premise is that he was a traitor because he chose to be against the war while the fighting was still going on. The film includes all sorts of references to POWs who say that their torturers used to refer to Kerry, that Kerry is lionized in modern Vietnam as a hero for the enemy, and that the museums in Vietnam worship Kerry as helping fight the good fight for the Vietnamese.

Of course, this is all bullshit, and even if it were true, it doesn’t mean Kerry’s a traitor. It is true that Kerry was against the Vietnam war, but guess what–historical revisionism is with him on that one. The Vietnamese can idolize whomever they want, the important thing was that Kerry had the courage to come home and admit that this war was wrong. Did he benefit politically from it? Maybe. But that’s a better way to advance your political career than George W. Bush, who was just handed his career by his daddy.

Almost all historians who have studied Vietnam and written anything about see it as nothing more than a war in error, whose nature caused soldiers to act in a completely amoral way simply due to their yearning to survive. No one blames the soldiers for being amoral–the circumstances bred that. There was no fucking law in Vietnam, it was kill or be killed. But it is the task of a government and its generals to prevent those situations–to plan strikes on concentrated enemy forces. We just dropped soldiers into a foreign jungle and expected them to only kill the bad guys. Some plan we had.

Anyone who thinks that Vietnam was a just war, and that someone is a traitor to be against it, is seriously living in a dream world. It was a god-damn mess, and we raped and killed people, gave them cancer with Agent Orange, pillaged villages and lost many soldiers, and all for a war that was very much political from the start (part of the grand national obsession with communism, which has only been replaced by terrorism as of late).

But more than that, this documentary was solely based on a quote of Kerry’s taken out of context, as usual for these Republican scumbags. Take a look at FactCheck’s analysis.

I know Kerry is long off the radar and the 2004 was a long time ago at this point, but this still baffles me. Just look at how the documentarians refer to it as the documentary “that made history.” It even quotes a NYTimes article which says “”Stolen Honor”…should be shown in its entirety on all the networks, cable stations and on public television,” yet another quote taken out of context. The NYTimes review was actually very negative on the film in general, and said its only value was to show how those being tortured in POW camps felt betrayed when people came out against the war. But the review goes on at length about how various things are blown out of proportion in order to try to make the film a political propaganda piece against Kerry. When the author of the review wrote it should be shown on all news stations, he was being a bit ironic. He said that because he would then say, “This histrionic, often specious and deeply sad film does not do much more damage to Senator John Kerry’s reputation than have the Swift Boat Veterans for Truth’s negative ads, which have flooded television markets in almost every swing state.” In other words, the political damage has been done, so at this point may as not worry about it.

Why am I worrying about it? Because this points in general to the state of our media. Conservatives don’t like objective analysis. so they have a real easy answer: we’ll just invent the reality. I still remember one of the scariest, and craziest quotes I had ever come across:

“The aide said that guys like me were ‘in what we call the reality-based community,’ which he defined as people who ‘believe that solutions emerge from your judicious study of discernible reality.’ I nodded and murmured something about enlightenment principles and empiricism. He cut me off. ‘That’s not the way the world really works anymore,’ he continued. ‘We’re an empire now, and when we act, we create our own reality. And while you’re studying that reality – judiciously, as you will – we’ll act again, creating other new realities, which you can study too, and that’s how things will sort out. We’re history’s actors . . . and you, all of you, will be left to just study what we do.'”

Conservatives don’t like global warming, so they pretend it isn’t happening. They don’t like evolution, so they pretend that isn’t happening. They don’t like Kerry, so they pretend he was a traitor to his country. They don’t like opposition to the war in Iraq, so they pretend it is doing lots of good. They don’t like blame for Katrina, so they pretend that “no one is to blame”, or that the poor people Left Behind in that state are to blame for settling down there, or that Democratic politicians are to blame for not doing something about it.

In short, they are creating the reality. NewsMax, Fox, Weekly Standard, Think Tanks, etc. When a conservative says to me the media is liberal, I’m no longer going to respond with the laugh I usually do, saying something like “I wish that the media were liberal. That would make us all a lot better off.” Instead, I am just going to talk about this, and ask them: tell me, what liberal media won John Kerry the 2004 election? Oh, that’s right. The one that pushed out propaganda on public airwaves calling Kerry a traitor. That liberal media.

Arghh, this stuff gets me so angry sometimes!

(p.s. I never really supported Kerry anyway, but for completely different reasons. I didn’t support him because he wasn’t at all radical, he was as close to the center as democrats go. He would have held off the Bush onslaught of 2004-2008, but he wouldn’t have brought our country back to a place I’d like to be–a place where social justice and fairness enter at all into the role of government. But with a President as bad as Bush, Kerry looks like Mahatma Ghandi.)

Cindy Sheehan smeared by O’Reilly

I really would expect nothing less of my unfortunate neighbor, Bill O’Reilly. Apparently on last night’s show he smeared Cindy Sheehan, the grieving mother that’s been glavanizing the Left as of late, on his wonderful show, the O’Liar Factor. Apparently we still live under McCarthy, where it’s not who you are, but with whom you associate, that determines whether you are a “radical,” or “commie bastard.”

How do people still watch his show?