Phil doesn't just go on one date with Rita, he goes on thousands of dates. During each date, he makes note of what she likes and responds to, and drops everything she doesn't. At the end he arrives at -- quite literally -- the perfect date. Everything that happens is the most ideal, most desirable version of all possible outcomes on that date on that particular day. Such are the luxuries afforded to a man repeating the same day forever.
But at the end of this perfect date, something impossible happens: Rita rejects Phil.
Phil wasn't making these choices because he honestly believed in them. He was making these choices because he wanted a specific outcome -- winning over Rita -- and the experimental data told him which path he should take. Although the date was technically perfect, it didn't ring true to Rita, and that made all the difference.
While I think the analogy between Groundhog Day and A/B testing is perfect, I disagree with the conclusion that "A/B testing is like sandpaper. You can use it to smooth out details, but you can't actually create anything with it."
I would argue that A/B testing is useful because it reveals what people actually prefer, rather than what they say they prefer. Often there is quite a gap between the two. Also, as many of the commenters pointed out, A/B testing is just a tool - you don't have to limit yourself to testing tiny, incremental tweaks. You can creatively explore the world of possibilities, while basing your final choice on hard data.
As one commenter pointed out, the analogy to Groundhog Day breaks down because at the end of the day, "It's a movie. She rejects him, not because of some inherent failure in the method he uses, but because it was written that way in the script."
No comments:
Post a Comment