United States: Consumers Prefer My Product!
A Primer on the Use of Consumer Preference Surveys to Support Preference Claims

You are the CEO of a corporation. Your company is in the midst of developing a new ad campaign to make a push for one of its products. In the course of developing this campaign, the marketing department comes to you with a presentation. They want to introduce a new tagline for the product. After doing some research, they’d like to say, "ABC is the most preferred brand of moisturizer" or "No brand of moisturizer is preferred more than ABC." You know that these preference claims are likely to result in legal challenges by your competitors, but your marketing team does have some evidence to support their claims, and you can conduct a more detailed consumer preference survey to confirm that the preference claims are substantiated. How can you be confident that your survey and its results will be accorded significant evidentiary weight if you encounter a legal challenge?

While a plaintiff must prove several elements to succeed on a Lanham Act § 43(a) false advertising claim, the threshold question under consideration is whether the advertisement contains a false statement of fact. See Abbott Lab. v. Gerber Prods. Co., Inc., 979 F. Supp. 569, 571 (W.D. Mich. 1997) (determining whether defendant’s preference claims were false based on defendant’s supporting consumer survey). A plaintiff proves that the statement is false by showing either "that the advertising is literally false as a factual matter, or . . . although the advertisement is literally true, it is likely to deceive or confuse customers." Abbott Lab., 979 F. Supp. at 571. Where a plaintiff disputes a statement that is based on a consumer survey, the generally accepted legal standard that most courts apply requires the plaintiff to prove that the claim is literally false by showing that " 'the tests [or surveys relied upon] were not sufficiently reliable to permit one to conclude with reasonable certainty that they established the proposition for which they were cited.' " Abbott Lab., 979 F. Supp. at 573 (quoting Rhone-Poulenc Rorer Pharms., Inc. v. Marion Merrell Dow, Inc., 93 F.3d 511, 514 (8th Cir.1996)). In other words, the plaintiff must prove either that the survey is fundamentally or scientifically flawed or that it does not support the statement contained in the advertisement.

Whether a survey is proffered as evidence to prove a plaintiff’s claim of false advertising or as support for a defense, courts and expert witnesses alike draw from the same set of factors to evaluate whether a survey is admissible, the evidentiary weight it should be given, and whether it supports an advertising claim that one brand is superior to or preferred over another. The court in Cumberland Packing Corp. v. Monsanto Co. enumerated a set of factors that may be used as a guide, although this list is not exhaustive. Cumberland Packing Corp. v. Monsanto Co., 140 F. Supp. 2d. 241, 245 (E.D.N.Y. 2001) (citing Toys "R" Us, Inc. v. Canarsie Kiddie Shop, Inc., 559 F. Supp. 1189 (E.D.N.Y.1983)). These factors are:

• The "universe" or product market is properly defined and examined.

• A representative sample of that universe is selected.

• The questions to be asked of the interviewees are framed in a clear, precise, and nonleading manner.

• Sound interview procedures are followed by competent interviewers with no knowledge of the litigation or the purpose for which the survey was conducted.

• The data gathered is accurately reported.

• The data is analyzed in accordance with accepted statistical principles.

• The objectivity of the entire process is ensured.

In addition to those factors listed above, the case law suggests that courts have evaluated surveys based upon such other factors as (1) whether the survey is properly "filtered" through use of controls to screen out irrelevant data; (2) whether the questions are directed to the real issues; and (3) whether the questions are ambiguous or biased. Millennium Imp. Co. v. Sidney Frank Imp. Co., Inc., No. Civ. 03-5145, 2004 WL 1447915, at *8 (D. Minn. June 11, 2004). However, court opinions generally address only factors that are raised by the parties or that the court relies upon to criticize a survey.

The "Universe" or Product Market Must Be Properly Defined. The "universe" is often scrutinized by the courts and expert witnesses. This is because the universe of survey respondents must reflect "[t]he segment of the population whose perceptions and state of mind are relevant to the issues in the case." J. Thomas McCarthy, McCarthy on Trademarks and Unfair Competition § 32:159 (4th ed. 2004). Universes that stray from the relevant segment of the population are considered either overinclusive or underinclusive, i.e., the defined universe reflects more than the relevant segment or it contains less, respectively. Errors in defining the universe of a study are more likely to prove fatal than errors in the content of the questions, for there is some value in a slanted question asked of the right witness, but no value in asking the right question of the wrong universe.

An overinclusive universe "generally presents less of a problem" than an underinclusive universe if responses from a relevant subset of respondents can be separated out and examined. Reference Manual on Scientific Evidence, 237 (Federal Judicial Center 1994). A universe may be defined as underinclusive by designating a group narrower than the ideal universe, thus leaving out a group of persons whose perception is relevant. If the issue involves the state of mind of consumers, a survey of dealers or members of the trade may or may not be probative, depending on the legal issue. In some cases, the propriety or impropriety of a survey is a function of the geographic area covered. It has been said that an informal rule of thumb is that four testing sites are a minimum number to ensure a reasonable degree of projection to the universe of a larger area. Jacoby, "Survey and Field Experimental Evidence," at 184 in Kassin & Wrightsman, The Psychology of Evidence and Trial Procedure (1985).

In false advertising cases, to define the relevant universe, the segment of the population must be narrowed geographically, commercially, according to buying habits, or by any other meaningful criteria that the law sets down as limiting or defining the class of persons whose state of mind is at issue, to exclude inappropriate, unknowledgeable, or unconcerned respondents. Therefore, if a product is sold only on the East Coast, a universe that includes West Coast consumers in a brand preference survey would be overinclusive because the West Coast consumers would not have the relevant state of mind necessary to formulate a preference with respect to the product at issue. Similarly, a universe can be narrowed commercially by limiting it to purchasers in a particular subclass equivalent to the products at issue. A survey comparing preference for Mercedes and BMW cars would not define a universe to include purchasers of Geo Metros or even Honda Accords because these cars are not commercially or economically similar. Lastly, the issue of buying habits—how or where the relevant consumers purchase the products at issue—though perhaps less relevant to preference surveys, nonetheless is an additional consideration that further illustrates the importance of a well-defined survey.

Once the universe is defined, a threshold inquiry must be conducted of each survey respondent to determine which respondents actually fall within the defined universe. This requires that the survey interviewer or the questionnaire ask questions that identify relevant respondents and weed out those who do not fall within the defined universe. It is not enough to "define the universe" appropriately, although not doing so may be a determinative factor. But the respondents must also be members of that relevant segment of the population for the survey to have weight and to support a claim.

Cases involving preference claims and other trademark survey cases find that the appropriate universe reflects the relevant consumers, who are purchasers and prospective purchasers of the products at issue.

Another criticism of survey universes is that they do not accurately reproduce the state of mind of consumers "in a buying mood" and therefore do not replicate marketplace conditions. However, the better and more common view is that "the closer the survey context comes to marketplace conditions, the greater the evidentiary weight it has. Survey interviews conducted in a store or in a shopping center should reach persons ‘in a buying mood,’ even assuming that this is a necessary ingredient." McCarthy, supra at § 32:163 (footnotes and citations omitted). "Surveys taken at home in person or by telephone should not be discounted or denigrated, but accepted as probative evidence if properly conducted. . . . The only inherent problem in a telephone interview is that the visual component is missing." Id. Telephone surveys are accepted by most courts, and the Federal Judicial Center 1994 Reference Manual on Scientific Evidence, recognizing the usefulness of telephone surveys, "recommends that the expert’s report specify three elements: (1) the procedures that were used to identify potential respondents; (2) the number of telephone numbers where no contact was made; and (3) the number of contacted potential respondents who refused to participate." Id. (citing Reference Manual on Scientific Evidence, 254 (Federal Judicial Center 1994)).

A Representative Sample of the Defined Universe Must Be Selected. Because it is cost-prohibitive to survey the entire universe, a representative sample of the universe must be drawn fairly and impartially to render the results and any claim based on them statistically reliable. There are two types of samples—probability samples and nonprobability samples. Probability samples use a mathematically random selection of persons and give a known margin of error, which describes how stable the mean response in the sample is likely to be. The most common use of probability sampling is in telephone surveys, in which telephone numbers in a given territory can be randomly selected. By contrast, nonprobability samples do not require a mathematically random selection of persons to question and do not provide known margins of error. This sampling is used in mall-intercept surveys, where the selection of respondents is done on the basis of the availability and characteristics of the people approached in a mall or on a street. This approach is more likely to be used where the universe or its subgroups are broadly defined, for example, among age, gender, income, or education levels. Nonprobability samples are used if you need results reflecting only the sample itself, whereas probability samples are used if you want to project the survey results to the universe at large. Avon Prods., Inc. v. S.C. Johnson & Son, Inc., No. 94 Civ. 3958, 1994 WL 267836, at *6 (S.D.N.Y. June 15, 1994). While either method of sampling is appropriate, depending upon the type of survey and the advertising claim to be made, nonprobability sampling is gaining acceptance among the courts and is now the most commonly performed type of in-person survey done for trademark and unfair competition litigation.

In addition, the representative sample requires a sufficient number of respondents. While the sample size does not necessarily impact the accuracy of the estimates, courts are quick to criticize surveys with small sample sizes, i.e., under 100 participants, because they cast into doubt the general applicability of the results. However, a larger sample size will reduce the size of the confidence interval (statistical error value). One survey expert has outlined three rules of thumb regarding sample size. First, each test group should include at least 200 to 300 participants. Second, if comparing subgroups, at least 50 participants should make up each subgroup. Third, if the percentage of respondents supporting the advertising claim is small, a larger sample size is recommended, because a smaller confidence interval is desired. Jacoby, supra, at 574-5.

The Interview Questions Must Be Clear, Precise, Nonleading, Unambiguous, and Unbiased. The manner in which survey questions are asked may be a significant factor in a court’s evaluation of the survey. Ambiguous questions—those open to multiple interpretations—must be avoided. They may be confusing and lead to irrelevant responses, as may questions that require the respondent to speculate or recollect events that took place in the distant past. Additionally, the survey interviewers should be mindful of the interview location because it could affect a respondent’s recognition or awareness of brands. For example, if the question is meant to measure preference, the interview should not take place in plain view of a competitor’s store or immediately after a purchase of the products at issue. Furthermore, the respondents should not be made aware of the survey’s purpose, for example, a pending litigation, because they may be sympathetic to one of the parties as a result.

To avoid a leading and biased survey, survey questions may take either a closed-ended (multiple choice) or an open-ended format. Researchers should avoid putting words into a respondent’s mouth. While closed-ended questions are not inherently leading, they are inherently suggestive because they provide a limited set of answers. Consumers Union of United States, Inc. v. New Regina Corp., 664 F. Supp. 753, 769 (S.D.N.Y. 1987). Further, aided questions may be leading or overly suggestive. For example, in an awareness survey, an aided question might place the "desired response" in a prominent position among the choices. This technique may not only affect the evidentiary weight of the survey, but it could also be a fatal defect of the survey. In an effort to overcome the inherent infirmities of closed-ended and aided questions, it is necessary, at a minimum, to provide a "Do not know" or "Other" answer choice, depending upon the question. In this way, the survey does not force a response or encourage a respondent to choose indiscriminately and randomly among the answers that are provided.

Open-ended questions have shortcomings as well. These questions may fail to elicit responses that the respondent does in fact have but are not "top of mind." For example, a question that asks "What brand do you prefer?" by itself, does not account for responses attributable to brand recognition, awareness, or the respondent’s last purchase. Therefore, open-ended questions require probing follow-up questions such as "Why?" or "What do you mean?" to clarify or expand upon an incomplete, ambiguous, or confusing answer. These questions may identify that a respondent is confused or simply that the response is irrelevant because, for example, it was brand awareness, not brand preference, that generated the response. Failure to adequately probe at incomplete, ambiguous, or confusing answers can be a factor in a court’s decision to discount survey evidence.

The order in which questions are posed may affect the survey’s objectivity and become an aspect of inquiry. A prior question should not provide or suggest an answer to a later question. Thus, aided-awareness questions should not precede unaided awareness. Also, closed-ended questions listing specific brands as answer choices should not precede open-ended questions about preference or awareness to avoid any risk that the answer choices will suggest the responses to the open-ended questions.

Accordingly, when an open-ended question is asked, such as "When you think of moisturizer, what brands come to mind?" the participant should also be asked the probing follow-up question "Why?" so that the interviewer can elicit the reasons for the previous answer. Similarly, after an open-ended question measuring preference is posed, the same probing question "Why?" should be asked. If the answer identifies a reason for the response other than an attribute for preference, such as "I just bought moisturizer by Brand X," it may reveal that the question is not clear or is ambiguous to the respondent. The probing questions can also be used to filter out irrelevant responses from the data. Cumberland, 140 F. Supp. 2d. at 245. For example, the response "I just bought Brand X" may be an indication that the answer to the open-ended preference question was the result of the respondent’s awareness, not necessarily his preference for that brand. Without incorporating these safeguards to ensure responses that are clear, unambiguous, and relevant, the survey will be open to attack.

It is standard practice for the survey expert to conduct a pilot test or a pretest of a proposed survey in order to evaluate the clarity and usefulness of the survey method, structure, and questions. A pilot test is a small field test replicating the procedures planned for the full-scale survey. A pretest tests the questionnaire. In such tests, a proposed survey is administered to a small sample of about 25 to 75 respondents, and the procedures and responses are analyzed. The final survey is often changed and questions reworded as a result of feedback from the pilot test. It is not considered biased or unfair to make changes in either the survey methodology or the questions as a result of the information learned from the pilot test. In fact, pilot tests can improve the quality of the survey. While it is not usual practice to disclose or discuss the existence or scope of a pilot test, the Reference Manual on Scientific Evidence notes cryptically, "The Federal Rules of Civil Procedure, however, may require that a testifying expert disclose pilot work that serves as a basis for the expert’s opinion." McCarthy, supra at § 32:163.1.

Incomplete, ambiguous, or otherwise flawed questions surface often enough to be seen as an indicator of courts’ increasing vigilance regarding question design. Two slightly different questions on the same topic may elicit two entirely different answers, and courts are increasingly suspicious of the design as well as the motives behind the research plan. Therefore, it is important to take a hard look at the question design format prepared by the research firm prior to launching the survey to make sure that it is appropriate for purposes of the survey, thereby eliminating the possibility of a fatal defect in the survey.

Sound Interview Procedures Must Be Followed by Competent Interviewers With No Knowledge of the Litigation or the Purpose for Which the Survey Was Conducted. To avoid potential biased administration of the survey, the interviewers should not have knowledge of the client, the litigation, or the purpose for conducting the survey. Consumers Union, 664 F. Supp. at 770. The concern is not necessarily the interviewer’s intentional impropriety, but rather the avoidance of any risk that an interviewer may inadvertently and unconsciously suggest or lead responses by adding emphasis to the questions, giving unsolicited clarification of questions, or moving on to another question once the anticipated response is given, without additional probing. On the other hand, the survey designers should know the underlying issues and the survey’s purpose so they can properly design the survey. And, while attorneys should have no part in carrying out the survey, some attorney involvement in the survey design is necessary to ensure that relevant questions are directed to a relevant population.

The Data Gathered Must Be Accurately Reported. For the survey to be reliable, the data must be accurately reported. Thus, responses such as those elicited by an open-ended question must be recorded verbatim and correctly categorized or coded. If this is not achieved, the survey may be regarded as biased or unreliable.

The Data Must Be Analyzed in Accordance With Accepted Statistical Principles. It is a basic tenet of survey practice and scientific evidence that accepted statistical principles be used to analyze and manipulate the data. If all the factors are adhered to—namely, a relevant universe, a proper representative sample, appropriate questions that are not susceptible to attack—an opposing expert may resort to an examination of the calculations used to analyze the raw data. This would likely entail producing all raw data to the other side and the opposing expert recalculating and evaluating the results and scrutinizing the original calculations.

The Objectivity of the Entire Process Must Be Ensured. This factor resonates throughout all the other factors. As discussed above, the interviewers must remain objective, as must those designing the survey. However, the survey designers must have some understanding of the underlying issues and reasons for conducting the survey, as well as the client’s goals. Every survey has a taste of subjectivity and bias. The purpose for conducting the survey—for example, to create evidence for a litigation—does not negatively affect or lessen its weight. See Consumers Union, 664 F. Supp. at 770.

The Survey Must Be Properly "Filtered" Through Use of Controls to Screen Out Irrelevant Data, and the Questions Must Be Relevant and Directed to the Real Issues. In combination, the focus of these last two factors is to isolate the relevant issues that the survey is intended to measure and to isolate the relevant data for analysis. This two-part inquiry asks, first, whether the question is relevant and, second, whether the response to it is relevant. Here, for example, the relevant issue for the survey to measure is consumer preference for a brand of moisturizer. Assuming that brand awareness is not equivalent to or a measure of preference, a question regarding awareness, or brand recognition, is not relevant. Data generated by such questions cannot be used to support an advertisement that claims consumer preference. However, simply asking "What brand do you prefer?" is not sufficient because a respondent may have any number of reasons for answering "Brand ABC" but not all reasons are attributes of preference for a brand.

The second part of the inquiry attempts to filter out the unresponsive and, thus, irrelevant responses. Cumberland, 140 F. Supp. 2d. at 253. This requires a definition of what "prefer" means and identifies. Abbott Lab., 979 F. Supp. at 573. In other words, what are the determinative factors for overall preference and which relevant attributes of the product or brand are necessary to establish superior preference of consumers? Id. Therefore, if, after probing, the underlying reason for a respondent’s selection of Brand ABC is something other than a relevant attribute of preference, that response must be thrown out of the analysis completely. Cumberland, 140 F. Supp. 2d. at 253.

A survey is an attempt to paint a picture of relevant purchasers’ states of mind. That picture, at the least, will portray the state of mind of those surveyed, and it may even reasonably depict the state of mind of the entire universe of relevant purchasers. Either way, if the survey is fairly and scientifically conducted, the results can be offered as evidence of and support for a corporation’s proposed advertising claim. Therefore, when survey data is relied upon to make an advertising claim, it is important to follow the guidelines set out above so that the survey and its results are accurate and reliable and will be accorded significant evidentiary weight if the survey is later attacked.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.