Blog post at Dataspora
Skill #1: Statistics (Studying). Statistics is perhaps the most important skill and the hardest to learn. It’s a deep and rigorous discipline, and one that is actively progressing (the widely used method of Least Angle Regression was only recently developed in 2004).
Skill #2: Data Munging (Suffering). The second critical skill mentioned above is “data munging.” Among data geek circles, this refers to the painful process of cleaning, parsing, and proofing one’s data before it’s suitable for analysis. Real world data is messy. At best it’s inconsistently delimited or packed into an unnecessarily complex XML schema. At worst, it’s a series of scraped HTML pages or a thoroughly undocumented fixed-width format.
Related to munging but certainly far less painful is the ability to retrieve, slice, and dice well-structured data from persistent data stores, using a combination of SQL, scripting languages (especially Python and its SciPy and NumPy libraries), and even several oldie-but-goodie Unix utilities (cut, join).
And when data sets grow too large to manage on a single desktop, the samurai of data geeks are capable of parallelizing storage and computation with tools like 96-nodes of Postgres, snow and RMPI, Hadoop and Mapreduce, and on Amazon EC2 to boot.
Skill #3: Visualization (Storytelling). This third and last skill that Professor Varian refers to is the easiest to believe one has. Most of us have had exposure to basic chart-making widgets of Excel. But a little knowledge is a dangerous thing: these software tools are often insufficient when faced with the visualization of large, multivariate data sets.
Here it’s worth making a distinction between two breeds of data visualizations, which differ in their audience and their goals. The first are exploratory data visualizations (as named by John Tukey), intended to faciliate a data analyst’s understanding of the data. These may consist of scatter plot matrices and histograms, where labels and colors are minimally set by default. Their goal is to help develop a hypothesis about the data, and their audience typically numbers one.
A second kind of data visualization are those intended to communicate to a wider audience, whose goal is to visually advocate for a hypothesis. While most data geeks are facile with exploratory graphics, the ability to create this second kind of visualization, these visual narratives, is again a separate skill — with separate tools.
The ability to visualize and communicate data is critical, because even with good data and rigorous statistical techniques, if the results of an analysis are poorly visualized, they will not convince: whether it’s an academic discovery or a business proposal.
Wednesday, May 27, 2009
Tuesday, May 26, 2009
Rise of the Startups
My gut feeling has been that history is cyclical, generally speaking, and that true revolutionary change is highly unlikely in the long term. Little did I know that there is an incredible amount of information on this topic just a Google search away.
Wikipedia article on "Social cycle theory"
Anyways, the following excerpt from this Wired.com article about the "new new economy" caught my attention:
Wikipedia article on "Social cycle theory"
Anyways, the following excerpt from this Wired.com article about the "new new economy" caught my attention:
As venture capitalist Paul Graham put it, "It turns out the rule 'large and disciplined organizations win' needs to have a qualification appended: 'at games that change slowly.' No one knew till change reached a sufficient speed."The article points out the recent decline of large corporations and points out their disadvantages in our current economy, while highlighting the strengths of small, nimble startups. The article also links to an article by Paul Graham, expanding on the whole "old versus new" debate.
The result is that the next new economy, the one rising from the ashes of this latest meltdown, will favor the small.
But in the late twentieth century something changed. It turned out that economies of scale were not the only force at work. Particularly in technology, the increase in speed one could get from smaller groups started to trump the advantages of size.Reading that article led to another one, which talks about the declining importance of credentials and how startups are much more meritocratic - nobody cares where you went to school or who your parents are, all that matters is your performance.
Large organizations will start to do worse now, though, because for the first time in history they're no longer getting the best people. An ambitious kid graduating from college now doesn't want to work for a big company. They want to work for the hot startup that's rapidly growing into one. If they're really ambitious, they want to start it.
History suggests that, all other things being equal, a society prospers in proportion to its ability to prevent parents from influencing their children's success directly. It's a fine thing for parents to help their children indirectly—for example, by helping them to become smarter or more disciplined, which then makes them more successful. The problem comes when parents use direct methods: when they are able to use their own wealth or power as a substitute for their children's qualities.
Large organizations can't do this. But a bunch of small organizations in a market can come close. A market takes every organization and keeps just the good ones. As organizations get smaller, this approaches taking every person and keeping just the good ones. So all other things being equal, a society consisting of more, smaller organizations will care less about credentials.
Googlenomics and Auctions
Wired.com article about Hal Varian, Google's Chief Economist
Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.
This is an example of a disruptive innovation - a gutsy move for Google, but one that ultimately paid off.
The problem with an all-at-once auction, however, was that advertisers might be inclined to lowball their bids to avoid the sucker's trap of paying a huge amount more than the guy just below them on the page. So the Googlers decided that the winner of each auction would pay the amount (plus a penny) of the bid from the advertiser with the next-highest offer. (If Joe bids $10, Alice bids $9, and Sue bids $6, Joe gets the top slot and pays $9.01. Alice gets the next slot for $6.01, and so on.) Since competitors didn't have to worry about costly overbidding errors, the paradoxical result was that it encouraged higher bids.
By turning over its sales process entirely to an auction-based system, the company could similarly upend the world of advertising, removing human guesswork from the equation.
The move was risky. Going ahead with the phaseout—nicknamed Premium Sunset—meant giving up campaigns that were selling for hundreds of thousands of dollars, for the unproven possibility that the auction process would generate even bigger sums. "We were going to erase a huge part of the company's revenue," says Tim Armstrong, then head of direct sales in the US. (This March, Armstrong left Google to become AOL's new chair and CEO.) "Ninety-nine percent of companies would have said, 'Hold on, don't make that change.' But we had Larry, Sergey, and Eric saying, 'Let's go for it.'"
The article asks if we can imagine using auctions in our everyday lives? Does this make our free market economy much more agile, responsive, and transparent? Take game consoles, for example. An auction-based system would very quickly determine the value of consoles, creating a true free market economy, rather than our current system of retail price management (RPM) agreements (sometimes called vertical price-fixing).
Google even uses auctions for internal operations, like allocating servers among its various business units. Since moving a product's storage and computation to a new data center is disruptive, engineers often put it off. "I suggested we run an auction similar to what the airlines do when they oversell a flight. They keep offering bigger vouchers until enough customers give up their seats," Varian says. "In our case, we offer more machines in exchange for moving to new servers. One group might do it for 50 new ones, another for 100, and another won't move unless we give them 300. So we give them to the lowest bidder—they get their extra capacity, and we get computation shifted to the new data center."
Varian believes that a new era is dawning for what you might call the datarati—and it's all about harnessing supply and demand. "What's ubiquitous and cheap?" Varian asks. "Data." And what is scarce? The analytic ability to utilize that data. As a result, he believes that the kind of technical person who once would have wound up working for a hedge fund on Wall Street will now work at a firm whose business hinges on making smart, daring choices—decisions based on surprising results gleaned from algorithmic spelunking and executed with the confidence that comes from really doing the math.
This is an example of a disruptive innovation - a gutsy move for Google, but one that ultimately paid off.
The problem with an all-at-once auction, however, was that advertisers might be inclined to lowball their bids to avoid the sucker's trap of paying a huge amount more than the guy just below them on the page. So the Googlers decided that the winner of each auction would pay the amount (plus a penny) of the bid from the advertiser with the next-highest offer. (If Joe bids $10, Alice bids $9, and Sue bids $6, Joe gets the top slot and pays $9.01. Alice gets the next slot for $6.01, and so on.) Since competitors didn't have to worry about costly overbidding errors, the paradoxical result was that it encouraged higher bids.
By turning over its sales process entirely to an auction-based system, the company could similarly upend the world of advertising, removing human guesswork from the equation.
The move was risky. Going ahead with the phaseout—nicknamed Premium Sunset—meant giving up campaigns that were selling for hundreds of thousands of dollars, for the unproven possibility that the auction process would generate even bigger sums. "We were going to erase a huge part of the company's revenue," says Tim Armstrong, then head of direct sales in the US. (This March, Armstrong left Google to become AOL's new chair and CEO.) "Ninety-nine percent of companies would have said, 'Hold on, don't make that change.' But we had Larry, Sergey, and Eric saying, 'Let's go for it.'"
The article asks if we can imagine using auctions in our everyday lives? Does this make our free market economy much more agile, responsive, and transparent? Take game consoles, for example. An auction-based system would very quickly determine the value of consoles, creating a true free market economy, rather than our current system of retail price management (RPM) agreements (sometimes called vertical price-fixing).
Google even uses auctions for internal operations, like allocating servers among its various business units. Since moving a product's storage and computation to a new data center is disruptive, engineers often put it off. "I suggested we run an auction similar to what the airlines do when they oversell a flight. They keep offering bigger vouchers until enough customers give up their seats," Varian says. "In our case, we offer more machines in exchange for moving to new servers. One group might do it for 50 new ones, another for 100, and another won't move unless we give them 300. So we give them to the lowest bidder—they get their extra capacity, and we get computation shifted to the new data center."
Let the Little Guys Drive (Disruptive Innovation)
Article at Wired.com
If a domestic auto industry is to survive, it will have to incorporate and encourage breakthroughs from outsiders like Transonic. Automakers will need to transition from a vertical, proprietary, hierarchical model to an open, modular, collaborative one, becoming central nodes in an entrepreneurial ecosystem. In other words, the industry will need to undergo much the same wrenching transformation that the US computer business did some three decades ago, when the minicomputer gave way to the personal computer. Whereas minicomputers were restricted to using mainly software and hardware from their makers, PCs used interchangeable elements that could be designed, manufactured, and installed by third parties. Opening the gates to outsiders unleashed a flood of innovation that gave rise to firms like Microsoft, Dell, and Oracle. It destroyed many of the old computer giants—but guaranteed a generation of American leadership in a critical sector of the world economy. It is late in the day, but the same could still happen in the car industry; it just has to harness our national entrepreneurial spirit to develop the next wave of auto breakthroughs.
By seeking to match the likes of Toyota, Detroit has been trying to come from behind in a game where its adversaries set the rules. To Klepper, the Carnegie Mellon economist, the Big Three today resemble the American television-receiver industry in the 1970s and 1980s, pioneered by US corporations that, after decades of domination, were suddenly confronted by foreign innovation. Companies like RCA and Zenith were slow to incorporate new technologies until it was too late; all exited or sold out to foreign firms. "Every time American companies catch up to the competition," Klepper says, "the competition already has moved on and instituted new things. In that situation, it's extremely difficult to get ahead."
The only escape from this conundrum is to pursue what Harvard Business School professor Clayton Christensen has called disruptive innovation—the kind of change that alters the trajectory of an industry. As Christensen argued in his 1997 book, The Innovator's Dilemma, successful companies in mature industries rarely embrace disruptive innovation because, by definition, it threatens their business models. Loath to revamp factories at high cost to make products that will compete with their own goods, companies drag their feet; perversely, financial markets often reward them for their shortsightedness. Good as they are, the European and Japanese automakers are established companies. At this point, they are as unlikely to pursue disruptive innovation as Detroit has been. That gives the US auto industry an opening. To take that opportunity, it will have to behave differently—it will have to step far outside the walls of the Rouge.
I very strongly believe in the idea of disruptive innovation. Other innovations that I consider disruptive are "netbooks" and casual gaming (i.e. PopCap Games). Getting sucked into a never-ending cycle of competition between established companies encourages incremental improvements, is reactionary, and ultimately drags down all players involved. Much better to break free and create a new paradigm/product.
If a domestic auto industry is to survive, it will have to incorporate and encourage breakthroughs from outsiders like Transonic. Automakers will need to transition from a vertical, proprietary, hierarchical model to an open, modular, collaborative one, becoming central nodes in an entrepreneurial ecosystem. In other words, the industry will need to undergo much the same wrenching transformation that the US computer business did some three decades ago, when the minicomputer gave way to the personal computer. Whereas minicomputers were restricted to using mainly software and hardware from their makers, PCs used interchangeable elements that could be designed, manufactured, and installed by third parties. Opening the gates to outsiders unleashed a flood of innovation that gave rise to firms like Microsoft, Dell, and Oracle. It destroyed many of the old computer giants—but guaranteed a generation of American leadership in a critical sector of the world economy. It is late in the day, but the same could still happen in the car industry; it just has to harness our national entrepreneurial spirit to develop the next wave of auto breakthroughs.
By seeking to match the likes of Toyota, Detroit has been trying to come from behind in a game where its adversaries set the rules. To Klepper, the Carnegie Mellon economist, the Big Three today resemble the American television-receiver industry in the 1970s and 1980s, pioneered by US corporations that, after decades of domination, were suddenly confronted by foreign innovation. Companies like RCA and Zenith were slow to incorporate new technologies until it was too late; all exited or sold out to foreign firms. "Every time American companies catch up to the competition," Klepper says, "the competition already has moved on and instituted new things. In that situation, it's extremely difficult to get ahead."
The only escape from this conundrum is to pursue what Harvard Business School professor Clayton Christensen has called disruptive innovation—the kind of change that alters the trajectory of an industry. As Christensen argued in his 1997 book, The Innovator's Dilemma, successful companies in mature industries rarely embrace disruptive innovation because, by definition, it threatens their business models. Loath to revamp factories at high cost to make products that will compete with their own goods, companies drag their feet; perversely, financial markets often reward them for their shortsightedness. Good as they are, the European and Japanese automakers are established companies. At this point, they are as unlikely to pursue disruptive innovation as Detroit has been. That gives the US auto industry an opening. To take that opportunity, it will have to behave differently—it will have to step far outside the walls of the Rouge.
I very strongly believe in the idea of disruptive innovation. Other innovations that I consider disruptive are "netbooks" and casual gaming (i.e. PopCap Games). Getting sucked into a never-ending cycle of competition between established companies encourages incremental improvements, is reactionary, and ultimately drags down all players involved. Much better to break free and create a new paradigm/product.
An interview with Tina Seelig
Posted by Guy Kawasaki at the OPEN Forum
Some excerpts I liked:
Question: What is the secret to successful negotiation?
Answer: Make sure that you understand the other person’s point of view. If you make assumptions, you will very likely be wrong. When I bought a car for my son. I assumed that the salesperson wanted us to pay the highest price. That wasn’t the case! After asking a bunch of questions, I learned that his commission wasn’t based on the price of the car—it was based on the scores he got on the customer evaluation form we filled out afterward. Of course, I was happy to give him a great score in return for a great price. This is how win-win negotiations come about.
Question: How does one balance work and “life”?
Answer: You copied a quote from my book into one of your recent blogs. That quote, attributed to the Chinese philosopher Lao-Tzu, is very powerful.
Some excerpts I liked:
Question: What is the secret to successful negotiation?
Answer: Make sure that you understand the other person’s point of view. If you make assumptions, you will very likely be wrong. When I bought a car for my son. I assumed that the salesperson wanted us to pay the highest price. That wasn’t the case! After asking a bunch of questions, I learned that his commission wasn’t based on the price of the car—it was based on the scores he got on the customer evaluation form we filled out afterward. Of course, I was happy to give him a great score in return for a great price. This is how win-win negotiations come about.
Question: How does one balance work and “life”?
Answer: You copied a quote from my book into one of your recent blogs. That quote, attributed to the Chinese philosopher Lao-Tzu, is very powerful.
“The master of the art of living makes little distinction between his work and his play, his labor and his leisure, his mind and his body, his education and his recreation, his love and his religion. He simply pursues his vision of excellence in whatever he does, leaving others to decide whether he is working or playing. To him, he is always doing both.”
Thursday, May 21, 2009
DinTaiFung
via Seth Godin's solicitation for updates to Purple Cow
Two things are guaranteed at the remarkable DinTaiFung restaurant in Taipei: the extremely long line outside and the size/weight of their world famous steamed juicy pork dumplings. Each dumpling uses only the freshest ingredients, weighs a precise 0.74 oz, and has exactly 18 folds. In 1993, NY Times named DinTaiFung as one of the top 10 restaurants in the world. Even with many outlets worldwide today, thousands of tourists still visit Taipei every year just to eat at its original location. One of the stories told about the restaurant owner is that he takes the tour buses to hear what people say about his restaurant. One day, he found that bus stopped before reaching its destination and tourists were encouraged to use the restrooms so that they can avoid using the ones at his restaurant. He went back and installed the most advanced toilets available in the restrooms and made sure that they were cleaned every 15 minutes. Since then, the restrooms at DinTaiFung also became one of the most talked about topics for tourists.
Two things are guaranteed at the remarkable DinTaiFung restaurant in Taipei: the extremely long line outside and the size/weight of their world famous steamed juicy pork dumplings. Each dumpling uses only the freshest ingredients, weighs a precise 0.74 oz, and has exactly 18 folds. In 1993, NY Times named DinTaiFung as one of the top 10 restaurants in the world. Even with many outlets worldwide today, thousands of tourists still visit Taipei every year just to eat at its original location. One of the stories told about the restaurant owner is that he takes the tour buses to hear what people say about his restaurant. One day, he found that bus stopped before reaching its destination and tourists were encouraged to use the restrooms so that they can avoid using the ones at his restaurant. He went back and installed the most advanced toilets available in the restrooms and made sure that they were cleaned every 15 minutes. Since then, the restrooms at DinTaiFung also became one of the most talked about topics for tourists.
Tuesday, May 19, 2009
Traits of successful CEOs
Op-ed column at the New York Times
They relied on detailed personality assessments of 316 C.E.O.’s and measured their companies’ performances. They found that strong people skills correlate loosely or not at all with being a good C.E.O. Traits like being a good listener, a good team builder, an enthusiastic colleague, a great communicator do not seem to be very important when it comes to leading successful companies.
What mattered, it turned out, were execution and organizational skills. The traits that correlated most powerfully with success were attention to detail, persistence, efficiency, analytic thoroughness and the ability to work long hours.
These results are consistent with a lot of work that’s been done over the past few decades. In 2001, Jim Collins published a best-selling study called “Good to Great.” He found that the best C.E.O.’s were not the flamboyant visionaries. They were humble, self-effacing, diligent and resolute souls who found one thing they were really good at and did it over and over again.
That same year Murray Barrick, Michael Mount and Timothy Judge surveyed a century’s worth of research into business leadership. They, too, found that extroversion, agreeableness and openness to new experience did not correlate well with C.E.O. success. Instead, what mattered was emotional stability and, most of all, conscientiousness — which means being dependable, making plans and following through on them.
The C.E.O.’s that are most likely to succeed are humble, diffident, relentless and a bit unidimensional. They are often not the most exciting people to be around.
They relied on detailed personality assessments of 316 C.E.O.’s and measured their companies’ performances. They found that strong people skills correlate loosely or not at all with being a good C.E.O. Traits like being a good listener, a good team builder, an enthusiastic colleague, a great communicator do not seem to be very important when it comes to leading successful companies.
What mattered, it turned out, were execution and organizational skills. The traits that correlated most powerfully with success were attention to detail, persistence, efficiency, analytic thoroughness and the ability to work long hours.
These results are consistent with a lot of work that’s been done over the past few decades. In 2001, Jim Collins published a best-selling study called “Good to Great.” He found that the best C.E.O.’s were not the flamboyant visionaries. They were humble, self-effacing, diligent and resolute souls who found one thing they were really good at and did it over and over again.
That same year Murray Barrick, Michael Mount and Timothy Judge surveyed a century’s worth of research into business leadership. They, too, found that extroversion, agreeableness and openness to new experience did not correlate well with C.E.O. success. Instead, what mattered was emotional stability and, most of all, conscientiousness — which means being dependable, making plans and following through on them.
The C.E.O.’s that are most likely to succeed are humble, diffident, relentless and a bit unidimensional. They are often not the most exciting people to be around.
Subscribe to:
Comments (Atom)
