Friday, July 30, 2010

Review: (Orme 2009): Which Conjoint Method Should I Use?

Bryan K. Orme, Sawtooth Software, Inc.
Copyright 2009

Objective
- provides greater depth in understanding issues involved with choosing a conjoint approach

Useful Information
- Ratings-Based Systems
* respondents are asked to rate (or rank) a series of concept cards describing product concepts using multiple attributes
* creator, Paul Green, suggested six attributes and 12 to 30 cards to avoid simplification strategies and that increased attributes meant increased cards
* early version referred to as "card-sort conjoint"
* Adaptive Conjoint Analysis
^ Sawtooth developed software to adapt cards to previous answers
^ possible to study a dozen to two-dozen attributes while keeping respondent engaged 
^ had varying sections of interview with one or few attributes presented at a time
^ led through systematic investigation over all attributes
^ resulted in full set of preference scores for levels of interest (part-worth utilities)
^ require computer administration
^ main-effects model
^ "all else equal," without inclusion of attribute interactions
^ understates importance of price, understatement increased with number of attributes
What were the key methods or approach?
- Choice-Based Conjoint (CBC)
* closely mimic purchase process in competitive contexts
* instead of rating or ranking, asked to choose from a set of products
* some products show products, about a dozen, on a screen as if on store shelves
* recommend researchers show more rather than fewer product concepts per choice task
* contain less information than rating per unit of respondent effort
^ do not learn degree of preference among products
^ do not learn relative preference among rejected alternatives
* Sawtooth can include up to 10 attributes, with 15 levels each (advanced design: 30 attributes with 254 levels)
* traditionally analyzed at aggregate level, but now individual level can be assessed using latent class and hierarchical Bayes (HB) estimation methods (majority of Sawtooth users use HB for final market simulation models)
^ Aggregate Choice Analysis
+ argued that it permits estimation of subtle interaction effects
+ market is not homogeneous, consumers have unique preferences
+ suffers from Independence from Irrelevant Alternatives (IIA) assumption (red bus/blue bus problem: very similar products in competitive scenarios receive too much net share), fail when there are differential cross-effects between brands
^ Latent Class Analysis
+ simultaneously detects homogeneous respondent segments and calculates segment-level part-worths
+ if market is truly segmented can reveal much about market structure and improve predictability over aggregate choice models
+ subtle interactions can be modeled
^ HB (Hierarchical Bayes Estimation)
+ "borrows" information from each respondent to improve accuracy and stability of each individual's part-worth
+ consistently proven successful in reducing IIA problem and improves predictive validity of individual-level model and market simulation share results
+ can employ main effects or models that include interaction terms
+ main effects models with HB are sufficient to model choice
+ HB outperforms aggregate logit for predicting shares for holdout choices and actual market shares even when there was very little heterogeneity in data
- Partial-Profile CBC
* used to increase number of attributes
* each choice question includes subset of total number of attributes which are randomly rotated into tasks so that each respondent considers all attributes and levels
* data are spread quite thin because each task has many attribute omissions
* require larger sample sizes to stabilize 
* individual-level estimation under HB does not always produce stable individua-level part-worths
* if main goal is to achieve accurate market simulations (and large enough samples are used), individual-level stability can be sacrificed
* subject to similar price bias as ACA (though not as pronouced)
* respondents can ignore omitted attributes and base choice solely on partial information presented in each task
* researchers and academics prefer full-profile conjoint techniques that display all attributes within each choice task to avoid bias of final part-worth utilities
- Adaptive CBC (ACBC)
* respondents first identify ideal product using configurator
* software builds a couple dozen similar product concepts for respondent to indicate which they would consider
* considered products are taken to a choice tournament to identify overall best concept where the choice tasks look like standard CBC tasks
* respondents find more engaging and realistic, but takes longer than CBC
* sample size is smaller than standard CBC because more information is captured from each individual
* more information at the individual level leads to better segmentation
* validity is slightly better than CBC
* capture percent of respondents find each attribute level to be "must have" or "unacceptable"
* not as useful for packaged goods with only a few attributes
* appropriate for problems involving more complex products and services with five attributes of more
- Sample size
* if dealing with relatively small sample sizes (especially less than 100), be cautious about using CBC unless respondents answer more than the usual number of choice tasks
* ACBC and ratings-based approaches
* If interview must be done on paper and small sample size is norm, consider CVA

Tuesday, July 27, 2010

Review: (Wessells, 2002) The Economics of Information: Markets for Seafood Attributes

Cathy Roheim Wessells, University of Rhode Island

Marine Resource Economics, Volume 17, pp. 153-162

Objectives
- Presents economic theory of information reflected by differing demand for and supply of seafood products in the presence or absence of certain attributes
- present theoretical framework for economics of market for information regarding product attributes
- reveal use of framework in empirical literature related to seafood

Methods/Approach
- This paper was a literature review of:
* the economics of attributes, specifically for seafood
* empirical literature of studies that broke seafood into attributes specifically using hedonic analysis, conjoint analysis and conjoint choice experiment

Results/Conclusions
- demand for attributes and information about those attributes exist, specific attributes are valuable to consumer
- there is a demand for information on product quality, seafood safety, environmental friendliness, and even the name of broker in the case of bluefin tunas sold in Tsukiji market
- demand is subject to nature of attribute, for cases where there is no demand there is no market for information
- if the government perceives there is a social benefit to the provision of product attributes or information, then there may be regulatory steps taken that force producers to expend resources to do so
- if consumers continue to demand attributes there is a potential for confusion if there are too many labels
* an extreme case of confusion is a lack of interest on the part of the consumer which could possibly lead to the disappearance for the market for attributes or information

Assumptions
- currently a competitive market for seafood
- consumers' interest in knowing more about seafood products will evolve and people will want desired attributes
- fishery sector only recently began paying attention to quality vs. aquaculture sector which has historically focused on quality attributes (ex. Pacific wild-caught salmon market has declined)
- seafood producers (farmed and wild) differentiate products with country of origin, brand, nutritional or environmental  labels
- producers will supply information as long as marginal cost to provide is less than marginal benefit
- some consumers concerned with dolphin-safe practices, but not shark-safe practices
- the difficulty of determining product quality in the market is even greater than that of determining price levels, since information about quality is usually more difficult to obtain.
- producers will make explicit claims for all positive aspects of goods which makes consumers suspicious of goods without claims (Aldrich 1999)

Opinion
According to Google Scholar this article has been cited 15 times, 3 are about seafood supply chains. The author never stated a methodology for how the empirical literature was chosen, so I am not certain the literature review is exhaustive. The stated objectives of discussing the economics of information about seafood attributes was concisely discussed, and the literature was informative as it discussed hedonic, conjoint and conjoint choice methodologies. The conclusions were not directly tied to or supported by the information in the paper, but it  still contributed to the general discussion. Overall the article had a lot of useful information for me regarding terminology. Specifically the breaking down of good or attributes into search, experience and credence will come in handy for choosing attributes for a conjoint analysis. The discussion of attributes in economic terms can contribute to the framing of our work.


One point of confusions is the last line of of the paper:


- " The extreme case of confusion is a lack of interest on the part of the consumer. Then, the framework presented above tells us that there will no longer be a market for attributes or information. Based on the empirical analysis discussed above, this would truly be unfortunate. "


I guess I just don't think that it will be "unfortunate" based on the empirical analysis of the discussion. I can see it would be unfortunate if industry and government could not align themselves with consumer information demand and therefore confusing the entire population to the point of not caring. The gap may arise in that industry and government do not know what information the consumer demands which is why papers such as this need to deliver this information to the proper recipients.

Useful Information
-  Another example of consumer demand for information on product attributes was the nationwide attempt by consumer advocate groups in the U.S. requesting that the federal government require labeling milk as to whether it came from hormone-treated dairy cows. In this case, the consumers wished for a regulatory solution, rather than a voluntary solution, by the milk industry. However, the U.S. Food and Drug Administration (FDA) decided that there was no public benefit in providing that information on milk cartons, as there are no known health risks of drinking milk from bST-treated cows. Thus the initiative failed, even though there was consumer demand for this information." (p. 154)
- search cost (Stigler 1961) = time and energy expended by consumer in determining seller with lower price (or any other attribute), marginal benefit of additional information is equal to marginal cost of obtaining additional information; there exists a willingness to pay for information and a marginal cost for obtaining it
- Types of products
* Search goods: can be determined by quality of searching price, color and smell
* Experience goods: quality discerned by experience of taste, durability or preparation
* Credence goods:  (Darby and Karni 1973)
_ imperfect market because it is not practical for consumers to assess quality of product
_ asymmetry of information between producer and consumer, ex. safety-approved or recyclable
_ labeling can transform credence attributes to search attributes that allow consumer to judge quality of good before purchase
- Quality Attributes of Food Products (Hooker and Caswell, 1996): 
* Food Safety: Foodborne pathogens, Heavy metals, Chemical residues, Food additives, Naturally occurring toxins, Veterinary residues
* Nutrition: Fat, Calories, Fiber, Sodium, Vitamins, Minerals
* Value: Purity, Compositional integrity, Size, Appearance, Taste, Convenience of Preparation
* Package: Package materials, Labeling, Other information such as recipes
* Process: Animal welfare, Biotechnology, Environmental impact, Chemical use, Worker safety
- "Recognizing that attributes like these have value to consumers, Lancaster (1971) characterized consumer demand for products instead as consumer demand for a bundle of attributes, where each product has one or more attributes. The essence of Lancaster’s framework is that a good by itself does not yield utility, but it possesses characteristics (attributes) that create it. Therefore, there is a demand for attributes in the marketplace. There is also a supply of attributes." (p. 156-157)
- An ecolabel such as the Marine Stewardship Council's (MSC) program creates a vertical supply curve
*only demand determines price
* the cost of certifying a fishery is fixed
* one level of environmental friendliness (fishery is well managed by MSC's standards and criteria)
* no gradation of management
* if size affects cost of supplying a well-managed fishery then supply can shift left or right
* food safety is a classic example of producers choosing not to supply product attributes because private marginal cost may outweigh private marginal benefits even though social benefit of information about attributes outweigh social cost
- freshness, fat content, color and shape have largest influence on price for bluefin tuna sold in Tsukiji Central Wholesale Market in Tokyo
* fat content is experience and search attribute
* determined in Carroll, Anderson , and Martinez-Garmendia (2001) using hedonic analysis, limited to success of data set in ability to differentiate various pieces of information pertinent to determination of product price
- Conjoint analysis vs. contingent choice
* used when market does not exist, product with certain characteristic has not reached the market
* Conjoint analysis
_ Holland and Wessells (1998): 9 hypothetical fresh salmon products presented, respondents asked to rank products, attributes that changed were price and production information such as farmed vs wild vs. no information, seafood safety inspection (USDA, FDA or no information)
^ used rank-ordered logit model (Beggs, Cardell and Hausman 1981)
^ credence goods: inspection, wild vs. farmed
^ search good: price
* Contingent Choice
_ Johnston et al. (2001): consumer tradeoff between products that are and are not ecolabelled in U.S. and Norway
^ using logit model, choice of certified is regressed against price, demographic attributes, preference attributes and environmental purchasing behavior
^ directly asked which agency would be trusted to provide sustainability claim (WWF, MSC or National Marine Fisheries Service (NMFS) in U.S. and WWF, MSC or Norwegian Fisheries Directorate (NFD))
+ NMFS had highest U.S. rating
+ 23% unsure
+ certifying agency is only important to Norwegians
* primary reason for use of conjoint choice is no market data exist from which to compare consumer demand
- seafood consumers place relatively little value on guarantees offered solely by industry groups (Wessells and Anderson 1995)

Further Research
- "There have not been any studies determining the cost structure of supplying seafood safety, environmental friendliness, or other attributes, nor have there been studies of the costs of supplying this information to the consumers." (p. 160)

Useful references
Homans, F.R. and J.E. Wilen. 1997. A model of Regulated Open Access Resource Use. Journal of Environmental Economics and Management 32:1-21.

Mariojouls and Wessels, 2002. ???. Journal of Environmental Economics and Management 32.

Hooker, N.H., and J.A. Caswell. 1996. Regulatory Targets and Regimes for Food Safety: A Comparison of North American and European Approaches. The Economics of Reducing Health Risk from Food, J.A. Caswell, ed., pp. 1–17. Storrs, CT: Food Marketing Policy Center.


Holland, D., and C.R. Wessells. 1998. Predicting Consumer Preferences for Fresh Salmon: The Influence of Safety Inspection and Production Method Attributes. Agricultural and Resource Economics Review 27:1–14.


Caswell, J.A. 1998. Valuing the Benefits and Costs of Improved Food and Safety and Nutrition. The Australian Journal of Agricultural and Resource Economics 42: 409-24.

Darby, M.R. and E. Karni. 1973. Free Competition and the Optimal Amount of Fraud. Journal of Law and Economics 16:67-88.

Lancaster, K.J. 1971. A New Approach to Consumer Theory. The Journal of Political Economy 74: 132-57.

Stigler, G. 1961. The Economics of Information. Journal of Political Economy 69:213-25.

Aldrich, L. 1999. Consumer Use of Information: Implications for Food Policy, Food and Rural Economics Division, Economic Research Service, U.S. Department of Agriculture. Agricultural Handbook No. 715.

Nelson, P. 1970. Information and Consumer Behavior. Journal of Political Economy 78: 311-29.

Nelson, P. 1974. Advertising as Information. Journal of Political Economy 81: 729-54.

Beggs, S., S. Cardell, and J. Hausman. 1981. Assessing the Potential Demand for Electric Cars. Journal of Econometrics 17:19–20.
Akerlof, G.A. 1970. The Market for ‘Lemons’: Quality Uncertainty and the Market Mechanism. Quarterly Journal of Economics 84:488–500.

Wessells, C.R., and J.G. Anderson. 1995. Consumer Willingness to Pay for Seafood Safety Assurances. Journal of Consumer Affairs 29:85–107.

Wessells, C.R., J. Kline, and J.G. Anderson. 1996. Seafood Safety Perceptions and their Effects on Consumption Choices under Varying Information Treatments. Agricultural and Resource Economics Review 25:12–21.

Johnston, R., C.R. Wessells, H. Donath, and F. Asche. 2001. A Contingent Choice Analysis of Ecolabelled Seafood: Comparing Consumer Preferences in the United States and Norway. Journal of Agricultural and Resource Economics 26:20–39.

Holland, D., and C.R. Wessells. 1998. Predicting Consumer Preferences for Fresh Salmon: The Influence of Safety Inspection and Production Method Attributes. Agricultural and Resource Economics Review 27:1–14.

Carroll, M., J.L. Anderson, and J. Martinez-Garmendia. 2001. Pricing U.S. North Atlantic Bluefin Tuna and Implications for Management. Agribusiness 17:243– 54.
Anderson, J.L. and S. Bettencourt. 1993. A Conjoint Approach to Model Product Preferences: The New England Market for Fresh and Frozen Salmon. Marine Resource Economics 8(1):31–49.

Wirth, F.F., C.K. Halbrendt, and G.F. Vaughn. 1991. Conjoint Analysis of the Mid- Atlantic Food-Fish Market for Farm-Raised Hybrid Striped Bass. Southern Journal of Agricultural Economics 23:155–64.

Thursday, July 22, 2010

Review: (Halbrendt et al, 1991) Conjoint Analysis of the Mid-Atlantic Food-Fish Market for Farm-Raised Hybrid Striped Bass



C.K. Halbrendt, F.F. Wirth, G.F. Vaughn (Delaware Agricultural Experiment Station, Department of Food and Resource Economics, College of Agricultural Sciences, University of Delaware, Newark)

Southern Journal of Agricultural Economics, July 1991

Objectives
- Examine buyer preferences toward farm-raised hybrid striped bass at wholesale, retail and restaurant levels
- Analyze purchase preference of mid-Atlantic seafood buyers when purchasing farm-raised hybrid striped bass
* utility values for different levels of four hybrid striped bass product attributes for wholesale, retail and restaurant
* relative importance of various hybrid striped bass product attribute (calculated from estimated attribute utility values
* measure buyer preferences for different product configurations to match preference of particular market levels

Methods/Approach
- conjoint measurement
* multivariate market research technique that finds relative importance of product's multidimensional attributes (Green and Wind)
* any decompositional method that estimates structure of buyers' preferences given buyers' overall evaluations of a set of alternative prespecified products in terms of levels of different attributes
* this study uses full profile approach: respondents rate a set of "total" hypothetical products
* to cut down on number of product profiles fractional factorial design can be used without losing information to effectively test effects of attributes on buyers
_ most common is orthogonal arrays
^ product profiles select so that independent contributions of all main effects are balanced, assuming negligible interactions (Green and Wind)
^ Desirable properties: 1) allow one to gather data on large number of product profiles using relatively small number of product profiles
^ most efficient statistically
* conjoint design
_ Step 1: Find Attributes and Levels
^ Attributes reflect key product characteristics or dimensions, should include most relevant to potential buyers (Cattin and Wittink)
^ Attribute levels should cover entire range of representative levels
^ (for this study) based on a priori knowledge of seafood marketing, review of past fish marketing studies and discussions with several large-volume fish buyers in mid-Atlantic
_ Step 2: Combine attributes and levels into hypothetical farm-raised hybrid striped bass products so preference ratings can be assigned
* conjoint administration
_ respondents rated each profile on scale from 0 (least preferred) to 10 (most preferred)
_ respondents were informed that a single 10-oz portion fillet can be obtained from a 1.5 lb hyrbid striped bass fish and 2 8-oz fillets can be obtained from a 2.5 lb fish (given for ease of respondent conversion)
_ list of 2,485 wholesales, retailers and restaurants from mid-Atlantic region was obtained from Dun's Marketing
_ 12% response rate of 296 firms (91 wholesalers, 84 retailers and 121 restaurants), original was 24% but firms that only sold shellfish were eliminated
_ not all surveys were complete, only 1,790 usable observations (preference ratings) out of 2,664
* Model specifications
_ functional form for each attribute is specified
^ a priori  notions can be used to determine shape of attribute's functional form, verified by scatter plots of buyer's utility rating with various levels of quantitative attributes and significance levels of estimated coefficient when subjected either to linear or quadratic forms
^ fish size: quadratic, purchase price: linear, product form and seasonal availability: 'part-worth' or dummy variable specification (uses "mean deviation coding" = coefficient for the base level is easily calculated as negative sum of (k-1) level coefficients, intercept becomes mean preference rating, and dummy variable coefficients measure deviation from the mean rating)
_ functional forms for each attribute are combined into conjoint preference model for estimation
^ Rating = f(Size, Form, Season, Price)
* market-levels are aggregated to provide information on statistical significance of inter-market levels using dummy variables for each market level
^ reveal how base preference level (represented by intercept term) varies between market levels by indicating how slopes change between market levels for different attribute variables
* conjoint preference model estimated using Ordinary Least Squares (OLS)

Results and Conclusions
- Low price and round form were found to be important in product preference rating for wholesale and retail markets and fillet form in restaurants
- All markets preferred larger size fish
* highest utility for wholesalers and retailers is between 2.0 and 2.5 lbs
* for restaurants the highest utility occurs between 2.5 and 3.0
- Fish-attribute variables
* Fish size: there is an ideal-point for fish size and fish size preference decreases as fish size changes from the ideal-point (positive sign for linear and curvilinear quadratic functional form)
* Purchase price: buyer utility decreases as price increases (negative coefficient)
* Form: fillet form had greatest effect on preference rating, then gutted, then round
* Availability: Year-round preferred over availability from April-October
- Market-Level Variables
* restaurant has greatest effect on intercept indicating a higher preference than wholesale or retail markets
* preference by market not significantly different from overall mean preference level (all dummy variables for markets were NOT significant)
- Attribute-Market Interaction
* market-level difference in fish size and purchase price between retail and restaurant (price-market interaction differed significantly from mean level)
* coefficients for interaction between filleted product form and restaurants and wholesale significantly different from mean preference level (fillet more important to restaurants than wholesalers or retailers)
- Attribute Utility Values
* intercept coefficient added to each market-level coefficient to compute separate market intercept parameter, then attribute coefficient added to different attribute market interaction to arrive at attribute parameter for each attribute
* for quantitative values (price and size) parameters are marginal utility values, for qualitative variables (form and season availability) can be interpreted as strictly utility values
- Calculation of Utility Values
* fillet has the highest utility for all markets
* wholesale and retail prefer round over gutted while restaurants prefer gutted over round
* year-round availability has higher buyer utility than Apr-Oct
* buyer utility for purchase price decreases as purchase price increases, highest purchase price utility at $2/lb
^ retail markets have lowest utility at all price levels, suggesting more importance on price than other markets
- Relative Importance of Attributes
* 1) for each attribute, determine highest and lowest utility values for the attribute, difference is attribute utility range; 2) take sum of ranges over all attributes: RI = (100 x range)/sum of ranges
* wholesale and retail markets very similar, relative importance weights almost identical
^ purchase price most important attribute in preference rating, contributing over 50 percent to the rating
^ product form and fish size were similar for both markets accounting for 18 to 21 percent of preference rating
* product form most important attribute for restaurants (42.8% of preference rating) then purchase price (38.1%)
- Market Simulations
* Utility = Base Level Utility + Sum of Attribute Level Utilities
* Analyst chooses product configurations (for this study: based on current mid-Atlantic market conditions on sea trout, a close substitute to hybrid striped bass)
 Errors in experimental design, statistical analyses or analytical approaches?
- "mean deviation coding" which is equivalent to traditional dummy variable coding from a mathematical point of view no citation
- Made attribute-market interaction conclusions even though market variables were not significant

Assumptions
- Wholesale, Retail and Restaurant differ in requirements; supplier must know which attributes influence purchasing decision in order to penetrate market
- products are multidimensional, the buyer must make overall judgment about relative value of attributes (Green and Wind)

Opinion
According to Google Scholar this article has been cited 72 times. Many of the citations are seafood related or food. In addition to the Cattin and Wittink article and updates since 1982 I have found many helpful resources. This article in particular is useful because of the use of rank of profiles. I would like to investigate the difference between full-profile approach and fractional factorial design using orthogonal arrays.

Overall this paper met its objectives. Purchase preferences at the wholesale, retail and restaurant level are known for the hybrid striped bass. For our purposes we need to know more about purchase preference for local vs. foreign or mainland import and aquaculture vs. wild which can differ per species. Appropriate species will have to be chosen and assumptions and information will have to be given to the respondents. The importance of other purchase factors such as taste and appearance may or may not need to be included. The role of price also needs to be determined.

Useful References
Cattin, P. and D.R. Wittink. "Commercial Use of Conjoint Analysis." J. Marketing, 46(1982): 44-53.

Green, P.E. and V. Rao. "Conjoint Analysis in Consumer Research: Issues and Outlook.: J. Consumer Res., 5(1978):103-123.

Monday, July 19, 2010

Review: (Kalish and Nelson, 1991) A comparison of Ranking, Rating and Reservation Price Measurement in Conjoint Analysis



Shlomo Kalish (Leon Recanati Graduate School of Business Administration, Tel Aviv University, Tel Aviv, Israel 69978) and Paul Nelson (William E. Simon Graduate School of Business Administration, University of Rochester, Rochester, New York 14627)

Marketing Letters 2:4, (1991): 327-335

Objectives
- Compare RP to traditional MA from predictive validity perspective
- Empirically compares traditional ranking and rating preference in CA to reservation prices (direct monetary measure of product value)
Key Methods/Approach
- used consumer survey results to compare predictive ability and goodness of fit of RB model using directly measured reservation prices with those of MA model using common rank and rating preference measures
- full profile representations of 12 hypothetical products
- each respondent state preferences using ranking, rating or reservation price
- product: round-trip airline tickets from New York City to Hawaii
- Attributes and Levels
* service Level: minimum, regular or premium
* seating room: regular or spacious
* non-stop or not
- included price when rating and ranking, but no price with reservation price
- price determined by part worth costing of features where cost of each feature reflected approximate market price at time
- $100 was added or subtracted from full cost to avoid correlation problems?
- estimation done on individual level
- ranks and ratings fit as a linear function
- stated reservation prices fit as a linear function of products' attribute levels only
respondents
- administer to undergraduate and first year graduate students in 6 different business classes
- no students had previous class exposure to conjoint analysis
- 225 useable questionnaires with 1/3 using each measurement method
* rank were measured from most preferred to least preferred
* rating measured on scale of 0 to 100 points in 5-point intervals
* reservation price respondent was asked to indicate what sum of money would make them indifferent between product and money
- preference order predicted for both original twelve and four holdout products
- estimation was done using OLS regression analysis, justified for RP and only partly justiied for ratings and less so for ranks
- fit measured using average across individuals of the Spearman rank correlation between ranks of actual measurements and fitted values (traditional measures (R^2) cannot be used since dependent variable are measured on different scales

Results and Conclusions
- reservation prices do very well in terms of fit, but are inferior in terms of predicting choice on a holdout sample
- little difference is found in performance of ranks and ratings
- Goodness of Fit
* Fit is high for all cases (rank: .91, rate: .88, reservation price: .96), but highest with reservation price perhaps because it allows a much larger range of values for dependent variable than ratings or rankings
* ranking and rating may have some favorable bias because there are 5 rather than 4 parameters, but mitigated since RP incorporates price before preferences thus accounting for price effects
* ratings scored lowest possibly because of response style bias (Kalwani and Silk, 1982)
- Predictive Validity
rank and rate measurement consistently  outperform reservation prices (Percent #1 correct- rank and rating: 62%, reservation: 46%), median (rank: .80, rate: .80 and RP: .60) and average also consistently larger (rank: .57, rate: .49 and RP: .43)
* reported correlations are averages and medians of rank correlations between choice ranking of four holdout products (stated choice) and predicted ranks (estimated choice)
hypothesis that average correlations are equal for ranking and rating or ratings and reservation prices cannot be rejected
- hypothesis of no difference between 3 measurement scales cannot be rejected
- regarding prediction of first choices
* no difference between 3 scales can be rejected at 5% level
* ratings outperform RP at 3% level
* rankings outperform RP at 5% level
* no statistically significant difference between rankings and ratings
- study was broadened to other products
* rankings averaged 56% first choice predictions and RP 40% (difference larger for low ticket items and high involvement items)
* when respondents are well informed all three measure perform roughly equal
* RP not as robust to respondent involvement as ranks and ratings
- future improvements in RP measurement may make it acceptable alternative to ranks and ratings in conjoint analysis

Errors in experimental design, statistical analyses or analytical approaches

- Sample size was not exactly split 3 ways (rank: 68, rate: 97, reservation price: 89)

Opinion
This study has been cited 62 times according to Google Scholar. The authors do point out that the result refer to a single study of a single product class and may or may not be generalizable. The paper was straight-forward and concisely written meeting its state objective of comparing rank, rating and reservation price via the single study. They also briefly discusses other examples which helped support their conclusions.

This information will be helpful when setting up the conjoint experiment for restaurant chefs. Given this information we would not use reservation price. According to PingSun rating is easier than ranking for participants. I think that rating could be used to sum up part-worths. Choosing ranking or rating may not be necessary if choice experiment is used, however we used questions in the value chain survey that could be useful as part-worth (Please rank the below factors according to importance when purchasing fish and seafood). Ranking or rating could be used to sum part-worths, but I think rating would be more telling of the participant’s attitude.

Useful Information
- Advantages of Reservation Price (RP) over Multiattribute (MA) models:
* directly connected with traditional economic demand theory (function plots number of consumers with reservation price greater than a said price vs. price)
* provides simplified algorithmic framework to evaluate price and positioning alternatives through choice simulation
- Reservation Price
* hypothesizes each customer has a maximum price they are willing to pay for a given product which equals the product's value to the consumer
* compares reservation price for each product with purchase price and chooses based on utility maximization (consumer surplus or net value according to Hauser and Urban (1986))
* because consumer input and response difficulty are larger for reservation prices than rank or rating, it is predicted to have lower fit and predictive ability
- Possible advantages of RP over MA:
* product description do not always need to include price, reducing attributes by one Icould be significant in some cases)
* in some applications a monetary valued utility function is easier and more intuitive
* directly measured RP is ration scaled allowing for use in regression and other metric estimation techniques
* scale is comparable across individuals eliminating difficulties of interpersonal utility that exist with rating and ranking
- RP vs. MA
* RP: monetary (ratio) scaled measure, does not take price into account, chooses product with highest utility
* MA: treats price as an additional attribute taking into account
* equivalent if prices enters MA model linearly (Ratchford, 1979 and Srinivasan, 1982), rescaled by dividing price coefficient is measured in monetary units
 - 3 ways to compare measurement scales
* evaluate consistency of preferences across 3 scales (not used here because respondent workload is high and learning effects significant)
* fit estimated model to original preferences
* predictive validity: after doing tasks respondent asked to rank choice among 4 hypothetical products
- 2 types of models: 1) main effects part worths model vs 2) linear model (used in this paper), could affect fit or predictive validity 
- little difference between OLS and monotonic regression results (Green and Srinivasan, 1978)

Useful References
Cameron, T.A., and M.D. James. (1987). "Estimating Willingness to Pay from Survey Data: An Alternative Pre-Test-Market Evaluation Procedure," Journal of Marketing Research 24, 389-395.
Ratchford, B.T. (1979). "Operationalizing Economic Models of Demand for Product Characteristics," Journal of Consumer Research 6, 76-85.
Green, P.E., and A.M. Krieger (1985). "Models and Heuristics for Product Line Selection," Marketing Science  4, 1-19."
Cattin, P. and D.R. Wittink (1982). "Commercial Use of Conjoint Analysis: A Survey," Journal of Marketin 46, 44-53.
Kalwani, M. and A.J. Silk. (1982). "On the Reliability and Predictive Validity or Purchase Intention Measures," Marketing Science 1, 243-286.