Wednesday, May 27, 2009

3 Skills of Sexy Data Geeks

Blog post at Dataspora

Skill #1: Statistics (Studying). Statistics is perhaps the most important skill and the hardest to learn. It’s a deep and rigorous discipline, and one that is actively progressing (the widely used method of Least Angle Regression was only recently developed in 2004).

Skill #2: Data Munging (Suffering). The second critical skill mentioned above is “data munging.” Among data geek circles, this refers to the painful process of cleaning, parsing, and proofing one’s data before it’s suitable for analysis. Real world data is messy. At best it’s inconsistently delimited or packed into an unnecessarily complex XML schema. At worst, it’s a series of scraped HTML pages or a thoroughly undocumented fixed-width format.

Related to munging but certainly far less painful is the ability to retrieve, slice, and dice well-structured data from persistent data stores, using a combination of SQL, scripting languages (especially Python and its SciPy and NumPy libraries), and even several oldie-but-goodie Unix utilities (cut, join).

And when data sets grow too large to manage on a single desktop, the samurai of data geeks are capable of parallelizing storage and computation with tools like 96-nodes of Postgres, snow and RMPI, Hadoop and Mapreduce, and on Amazon EC2 to boot.

Skill #3: Visualization (Storytelling). This third and last skill that Professor Varian refers to is the easiest to believe one has. Most of us have had exposure to basic chart-making widgets of Excel. But a little knowledge is a dangerous thing: these software tools are often insufficient when faced with the visualization of large, multivariate data sets.

Here it’s worth making a distinction between two breeds of data visualizations, which differ in their audience and their goals. The first are exploratory data visualizations (as named by John Tukey), intended to faciliate a data analyst’s understanding of the data. These may consist of scatter plot matrices and histograms, where labels and colors are minimally set by default. Their goal is to help develop a hypothesis about the data, and their audience typically numbers one.

A second kind of data visualization are those intended to communicate to a wider audience, whose goal is to visually advocate for a hypothesis. While most data geeks are facile with exploratory graphics, the ability to create this second kind of visualization, these visual narratives, is again a separate skill — with separate tools.

The ability to visualize and communicate data is critical, because even with good data and rigorous statistical techniques, if the results of an analysis are poorly visualized, they will not convince: whether it’s an academic discovery or a business proposal.

Tuesday, May 26, 2009

Rise of the Startups

My gut feeling has been that history is cyclical, generally speaking, and that true revolutionary change is highly unlikely in the long term. Little did I know that there is an incredible amount of information on this topic just a Google search away.

Wikipedia article on "Social cycle theory"

Anyways, the following excerpt from this article about the "new new economy" caught my attention:
As venture capitalist Paul Graham put it, "It turns out the rule 'large and disciplined organizations win' needs to have a qualification appended: 'at games that change slowly.' No one knew till change reached a sufficient speed."

The result is that the next new economy, the one rising from the ashes of this latest meltdown, will favor the small.
The article points out the recent decline of large corporations and points out their disadvantages in our current economy, while highlighting the strengths of small, nimble startups. The article also links to an article by Paul Graham, expanding on the whole "old versus new" debate.
But in the late twentieth century something changed. It turned out that economies of scale were not the only force at work. Particularly in technology, the increase in speed one could get from smaller groups started to trump the advantages of size.

Large organizations will start to do worse now, though, because for the first time in history they're no longer getting the best people. An ambitious kid graduating from college now doesn't want to work for a big company. They want to work for the hot startup that's rapidly growing into one. If they're really ambitious, they want to start it.
Reading that article led to another one, which talks about the declining importance of credentials and how startups are much more meritocratic - nobody cares where you went to school or who your parents are, all that matters is your performance.
History suggests that, all other things being equal, a society prospers in proportion to its ability to prevent parents from influencing their children's success directly. It's a fine thing for parents to help their children indirectly—for example, by helping them to become smarter or more disciplined, which then makes them more successful. The problem comes when parents use direct methods: when they are able to use their own wealth or power as a substitute for their children's qualities.

Large organizations can't do this. But a bunch of small organizations in a market can come close. A market takes every organization and keeps just the good ones. As organizations get smaller, this approaches taking every person and keeping just the good ones. So all other things being equal, a society consisting of more, smaller organizations will care less about credentials.

Googlenomics and Auctions article about Hal Varian, Google's Chief Economist

Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.

This is an example of a disruptive innovation - a gutsy move for Google, but one that ultimately paid off.

The problem with an all-at-once auction, however, was that advertisers might be inclined to lowball their bids to avoid the sucker's trap of paying a huge amount more than the guy just below them on the page. So the Googlers decided that the winner of each auction would pay the amount (plus a penny) of the bid from the advertiser with the next-highest offer. (If Joe bids $10, Alice bids $9, and Sue bids $6, Joe gets the top slot and pays $9.01. Alice gets the next slot for $6.01, and so on.) Since competitors didn't have to worry about costly overbidding errors, the paradoxical result was that it encouraged higher bids.

By turning over its sales process entirely to an auction-based system, the company could similarly upend the world of advertising, removing human guesswork from the equation.

The move was risky. Going ahead with the phaseout—nicknamed Premium Sunset—meant giving up campaigns that were selling for hundreds of thousands of dollars, for the unproven possibility that the auction process would generate even bigger sums. "We were going to erase a huge part of the company's revenue," says Tim Armstrong, then head of direct sales in the US. (This March, Armstrong left Google to become AOL's new chair and CEO.) "Ninety-nine percent of companies would have said, 'Hold on, don't make that change.' But we had Larry, Sergey, and Eric saying, 'Let's go for it.'"

The article asks if we can imagine using auctions in our everyday lives? Does this make our free market economy much more agile, responsive, and transparent? Take game consoles, for example. An auction-based system would very quickly determine the value of consoles, creating a true free market economy, rather than our current system of retail price management (RPM) agreements (sometimes called vertical price-fixing).

Google even uses auctions for internal operations, like allocating servers among its various business units. Since moving a product's storage and computation to a new data center is disruptive, engineers often put it off. "I suggested we run an auction similar to what the airlines do when they oversell a flight. They keep offering bigger vouchers until enough customers give up their seats," Varian says. "In our case, we offer more machines in exchange for moving to new servers. One group might do it for 50 new ones, another for 100, and another won't move unless we give them 300. So we give them to the lowest bidder—they get their extra capacity, and we get computation shifted to the new data center."

Let the Little Guys Drive (Disruptive Innovation)

Article at

If a domestic auto industry is to survive, it will have to incorporate and encourage breakthroughs from outsiders like Transonic. Automakers will need to transition from a vertical, proprietary, hierarchical model to an open, modular, collaborative one, becoming central nodes in an entrepreneurial ecosystem. In other words, the industry will need to undergo much the same wrenching transformation that the US computer business did some three decades ago, when the minicomputer gave way to the personal computer. Whereas minicomputers were restricted to using mainly software and hardware from their makers, PCs used interchangeable elements that could be designed, manufactured, and installed by third parties. Opening the gates to outsiders unleashed a flood of innovation that gave rise to firms like Microsoft, Dell, and Oracle. It destroyed many of the old computer giants—but guaranteed a generation of American leadership in a critical sector of the world economy. It is late in the day, but the same could still happen in the car industry; it just has to harness our national entrepreneurial spirit to develop the next wave of auto breakthroughs.

By seeking to match the likes of Toyota, Detroit has been trying to come from behind in a game where its adversaries set the rules. To Klepper, the Carnegie Mellon economist, the Big Three today resemble the American television-receiver industry in the 1970s and 1980s, pioneered by US corporations that, after decades of domination, were suddenly confronted by foreign innovation. Companies like RCA and Zenith were slow to incorporate new technologies until it was too late; all exited or sold out to foreign firms. "Every time American companies catch up to the competition," Klepper says, "the competition already has moved on and instituted new things. In that situation, it's extremely difficult to get ahead."

The only escape from this conundrum is to pursue what Harvard Business School professor Clayton Christensen has called disruptive innovation—the kind of change that alters the trajectory of an industry. As Christensen argued in his 1997 book, The Innovator's Dilemma, successful companies in mature industries rarely embrace disruptive innovation because, by definition, it threatens their business models. Loath to revamp factories at high cost to make products that will compete with their own goods, companies drag their feet; perversely, financial markets often reward them for their shortsightedness. Good as they are, the European and Japanese automakers are established companies. At this point, they are as unlikely to pursue disruptive innovation as Detroit has been. That gives the US auto industry an opening. To take that opportunity, it will have to behave differently—it will have to step far outside the walls of the Rouge.

I very strongly believe in the idea of disruptive innovation. Other innovations that I consider disruptive are "netbooks" and casual gaming (i.e. PopCap Games). Getting sucked into a never-ending cycle of competition between established companies encourages incremental improvements, is reactionary, and ultimately drags down all players involved. Much better to break free and create a new paradigm/product.

An interview with Tina Seelig

Posted by Guy Kawasaki at the OPEN Forum

Some excerpts I liked:

Question: What is the secret to successful negotiation?
Answer: Make sure that you understand the other person’s point of view. If you make assumptions, you will very likely be wrong. When I bought a car for my son. I assumed that the salesperson wanted us to pay the highest price. That wasn’t the case! After asking a bunch of questions, I learned that his commission wasn’t based on the price of the car—it was based on the scores he got on the customer evaluation form we filled out afterward. Of course, I was happy to give him a great score in return for a great price. This is how win-win negotiations come about.

Question: How does one balance work and “life”?
Answer: You copied a quote from my book into one of your recent blogs. That quote, attributed to the Chinese philosopher Lao-Tzu, is very powerful.
“The master of the art of living makes little distinction between his work and his play, his labor and his leisure, his mind and his body, his education and his recreation, his love and his religion. He simply pursues his vision of excellence in whatever he does, leaving others to decide whether he is working or playing. To him, he is always doing both.”

Thursday, May 21, 2009


via Seth Godin's solicitation for updates to Purple Cow

Two things are guaranteed at the remarkable DinTaiFung restaurant in Taipei: the extremely long line outside and the size/weight of their world famous steamed juicy pork dumplings. Each dumpling uses only the freshest ingredients, weighs a precise 0.74 oz, and has exactly 18 folds. In 1993, NY Times named DinTaiFung as one of the top 10 restaurants in the world. Even with many outlets worldwide today, thousands of tourists still visit Taipei every year just to eat at its original location. One of the stories told about the restaurant owner is that he takes the tour buses to hear what people say about his restaurant. One day, he found that bus stopped before reaching its destination and tourists were encouraged to use the restrooms so that they can avoid using the ones at his restaurant. He went back and installed the most advanced toilets available in the restrooms and made sure that they were cleaned every 15 minutes. Since then, the restrooms at DinTaiFung also became one of the most talked about topics for tourists.

Tuesday, May 19, 2009

Traits of successful CEOs

Op-ed column at the New York Times

They relied on detailed personality assessments of 316 C.E.O.’s and measured their companies’ performances. They found that strong people skills correlate loosely or not at all with being a good C.E.O. Traits like being a good listener, a good team builder, an enthusiastic colleague, a great communicator do not seem to be very important when it comes to leading successful companies.

What mattered, it turned out, were execution and organizational skills. The traits that correlated most powerfully with success were attention to detail, persistence, efficiency, analytic thoroughness and the ability to work long hours.

These results are consistent with a lot of work that’s been done over the past few decades. In 2001, Jim Collins published a best-selling study called “Good to Great.” He found that the best C.E.O.’s were not the flamboyant visionaries. They were humble, self-effacing, diligent and resolute souls who found one thing they were really good at and did it over and over again.

That same year Murray Barrick, Michael Mount and Timothy Judge surveyed a century’s worth of research into business leadership. They, too, found that extroversion, agreeableness and openness to new experience did not correlate well with C.E.O. success. Instead, what mattered was emotional stability and, most of all, conscientiousness — which means being dependable, making plans and following through on them.

The C.E.O.’s that are most likely to succeed are humble, diffident, relentless and a bit unidimensional. They are often not the most exciting people to be around.

Monday, May 18, 2009

Two Buck Chuck

CNN article about Fred Franzia, CEO of Bronco Wine

...the CEO of Bronco Wine, the nation's fourth-largest wine company, tells me repeatedly that only a sucker would pay more than $10 for a bottle of wine - including his own $35 Domaine Napa. And that Napa's and Bordeaux's claims about their special soils are bogus: "We can grow on asphalt. Terroir don't mean sh*t."

In 2002, Franzia persuaded Trader Joe's to sell a low-end label called Charles Shaw (after the winemaker who sold the tony label to Franzia, and dubbed Two Buck Chuck by consumers) that waged war on domestic wines in the $4 to $10 range - and was named best chardonnay in a blind taste test at July's California State Fair over far pricier competition. The label is one of America's fastest-growing, selling 5 million cases per year, all through one chain of stores.

"There's not a doubt in my mind that the two biggest things that have happened to the wine industry in the last 10 years are the movie Sideways and Two Buck Chuck," says Gary Vaynerchuk, who reviews wines on his popular video blog,

While he expects a lot, Franzia is known for listening to his employees, even if he has to berate them into talking. "The thing we do better than anyone is we listen," Franzia says. And despite Bronco's size, he's still willing to take big risks.

As we speak, enormous swaths of his fields are being ripped up to switch from cabernet sauvignon and merlot vines to pinot noir and pinot grigio, which Franzia expects to be big sellers because they're easy to drink.

"Success is easy if you think of it like rust: It's inevitable if you keep at it. You look for magic moments, but they're not there," Franzia says. "Guys can claim they are, but that's bullshit."


Interview with Matthew Mays by Guy Kawasaki

Question: How do you define elegance?
Answer: Something is elegant if it is two things at once: unusually simple and surprisingly powerful... At first glance, elegant things seem to be missing something.

Question: Which companies are your favorite examples or elegance?
Answer: Toyota is one. With Scion, they refused to advertise, and they drastically reduced the number of standard features to allow Generation-Y buyers to make a personal statement by customizing their cars. The Scion xB flew off the lot when it came out.Another example is the British bank, First Direct. It is branchless and became the most highly recommended bank in the United Kingdom. Then there’s the French manufacturing company FAVI that realized better employee relations when they eliminated their human resources department. W. L. Gore and Associates completely eliminated job titles and typical corporate hierarchy in order to release the creativity of its staff employees. And finally there’s always the usual suspects like the Google interface and Apple’s clean design. But my all-time favorite is In ‘N Out Burger. a freakishly popular hamburger chain that started in Los Angeles a half century ago, that has built its brand on the "less is more" approach with an interesting twist. The menu offers only five items: a hamburger, cheeseburger, double burger, French fries, and a short list of beverages. By keeping things simple, founder Harry Snyder says he is able to provide the highest quality food in a sparkling clean environment.

Question: What’s the first step a CEO should take to get her company on the right track?
Answer: ...Steve Jobs revealed that a "stop-doing" strategy figured centrally into Apple’s approach. What he said was: "We tend to focus much more. People think focus means saying yes to the thing you’ve got to focus on. But that’s not what it means at all. It means saying no to the hundred other good ideas that there are. You have to pick carefully. I’m actually as proud of many of the things we haven’t done as the things we have done." That’s the mindset. And step one? Create a solid stop-doing list. Sounds simple, but few do it. Guru Jim Collins says you absolutely must have a "stop-doing" list to accompany your to-do list. As a practical matter, he advises developing a strong discipline around first giving careful thought to prioritizing goals and objectives, and then eliminating the bottom 20 percent of the list. If as CEO you do that, and demand that everyone do that, including designers and engineers with respect to the stuff they’re building, your ugly crap quotient goes way down.

Question: Why do you think the Japanese have such a way with elegance?
Answer: [First is Zen, and one of the fundamental Zen aesthetic themes is emptiness, and second] ...kaizen — continuous improvement. It means "no best, only better."... To stop improving was to stagnate—which was to die. It was a war on all the things that make for crap: overproduction, overprocessing, defects, conveyance, unneccessary motion, inconsistency, and inventory. In short, Japan HAD to get elegant. They’ve never forgotten how they did it, and they’ve institutionalized it.

Tuesday, May 12, 2009

No utility, high price due to artificial scarcity

Article in the New York Times

Russia quietly passed a milestone this year: surpassing De Beers as the world’s largest diamond producer.

Quote from Aleksandr A. Malinin, an adviser to the president of Alrosa (Russia's largest diamond producer [97% of domestic rough diamond production]).

But what we are doing is selling an illusion,” meaning a product with no utility and a price that depends on the continued sense of scarcity where there is none.

“If you don’t support the price,” Andrei V. Polyakov, a spokesman for Alrosa, said, “a diamond becomes a mere piece of carbon.”

Monday, May 4, 2009

Escaping the Cubicle Nation

An interview with Pamela Slim

Question: How do you decide which business to start?

Answer: Business ideas are a dime a dozen. From my perspective, which is firmly rooted in the idea that the purpose of a business is to allow you to live the kind of life that makes you happy, healthy, wise, and wealthy—or at least well-fed, a good business idea has four components. First, it is rooted in something you are passionate about and which energizes you. Entrepreneurship is too darn hard to manufacture enthusiasm. Second, you have the skill and competence to make it happen—or at least a really great contact list of smart and enthusiastic friends to help you figure it out. Third, you need to do enough business planning to know whom you are trying to serve, and how you are going to make money. Finally, you want a business model that you have the resources to support and that delivers the life you want to live.

Question: What is the most common mistake the “escapees” make?

Answer: The most common mistake is thinking that they have to get all their plans absolutely perfect before launching. I have listened to people explain why they spent two months crafting an introductory email to a potential client. Perfectionism will cripple your business and thwart your plans faster than anything. Get used to pushing things out that feel not quite ready and then be completely responsive to fix them as you go. There will never be a perfect product, service, market or economy, so the most passionate, enthusiastic and responsive entrepreneur will win.

Friday, May 1, 2009

The greatest mathematical discovery - compound interest

Applied, in this case, to start-ups.

Article at ReadWriteWeb

While I thought the discussion of compound interest was interesting, I think the following excerpt is useful for entrepreneurs

Typical Thinking
"I've got an idea that no one has ever thought of. I'd better not tell anyone because as soon as I launch it, everyone will copy me. I'll develop it in secret, and then when my vision is ready, I'll launch it with great gusto and have a big sales and marketing campaign ready to go to snap up customers, who are all going to flock to me."

Typical Reality
You've got an idea that 100 people have had, 10 have already started working on, and probably a few have failed at. If you don't tell anyone or are so excited that you don't do your research, you won't find out about the 10 who have started working on it. You also don't invite 100 experienced people to help you refine your product, all of whom are far too busy to steal your idea anyway. You develop it in secret, and then 30 seconds after you finally launch, 50 people tell you about little things that are broken or need tweeking (a "tweek" being any change that you think will take two hours but ends up taking two weeks: tw(o) (w)eeks, get it?). You spend the next 6 months scrambling to get the product up to scratch. Customers don't come flocking. They come dribbling in like glue-covered sloths, whinging about every little thing you've missed.