Coca-Cola conducted 200,000 taste tests before launching New Coke. The results indicated consumers did in fact like the new formula. Only months later, after significant backlash, the old formula was brought back. Coca-Cola failed to consider two things. As Malcom Gladwell pointed out, the taste tests didn’t represent how people normally consumed Coke. A small taste test doesn’t represent what it’s like to drink a whole glass or 2-liter bottle. Second, taste wasn’t the only variable that consumers considered in their brand allegiance. There were elements of history and nostalgia that were not considered. They only tested one variable with the assumption that taste alone was the determining factor in purchasing decisions.
Make sure the variable or variables you are testing are actually relevant to the results you are trying to determine.
H/T @BekahKnowsBS