An Experiment Comparing Risk-Tolerance Questionnaires

John E. Grable, Ph.D., CFP® (1)
Amy Hubble, Ph.D., CFP®, CFA®
Michelle Kruger

University of Georgia
Athens, Georgia USA

Over the past several years, our research team has been testing different methods used by financial advisors to evaluate client risk attitudes. If you are currently in the practice of providing financial advice to consumers, you already know that assessing a client’s risk tolerance is a mandated activity in nearly all countries. You also know that the assessment marketplace has become intensively competitive over the past five years as more and more firms have entered the market with evaluation tools. Essentially, two approaches to assessing risk tolerance are being presented and debated by test developers: techniques based on economic modelling and assessments founded in psychometric theory. This brief report highlights findings from a study our team developed to determine which approach offers financial advisors the more valid measurement.

The purpose of the study was to compare and contrast the predictive validity of risk-tolerance questionnaires. Here is what we did. First, we asked approximately 160 people to complete an online survey. The survey asked each participant to answer a series of demographic and socioeconomic questions. Participant were asked to provide estimates of the amount they held in risky assets, like stocks, their likelihood of gambling, their investment experience, and their investing knowledge. We also asked participants to answer different risk-assessment questions.

The first set of risk-tolerance questions was based on the notion of economic risk aversion. These questions required participants to answer a series of income lottery questions. In the economics field, results from income gambles and lottery questionnaires are thought to provide insights into an investor’s revealed preference. If an advisor asks a client enough of these questions it may be possible to estimate a person’s expected utility function and derive a risk aversion coefficient. For those advisors who use a mean-variance optimization framework when making portfolio recommendations, the economic approach appears to be very efficient. Here is an example of the economic questions that were asked (2):

Question 1: Suppose that you are the only income earner in the family, and you have a good job guaranteed to give you your current (family) income every year for life. You are given the opportunity to take a new and equally good job, with a 50-50 chance it will double your (family) income and a 50-50 chance it will cut your (family) income by a third. Would you take the new job?

If the answer to this question was ‘yes,’ the participant was then asked:

Question 2: Suppose the chances were 50-50 that it would double your (family) income, and 50-50 that it would cut it in half. Would you still take the new job?

If the answer to the first question was ‘no,’ the participant was then asked:

Question 3: Suppose the chances were 50-50 that it would double your (family) income and 50-50 that it would cut it by 20 percent. Would you then take the new job?

Participants who answered ‘no’ to the first and third questions were classified as having high risk aversion (i.e., low risk tolerance). A participant who answered ‘no’ to the first question and ‘yes’ to the third question was classified as having above-average risk aversion. A participant who answered ‘yes’ to the first question and ‘no’ to the second question was classified as having below-average risk aversion. Those who answered ‘yes’ to the first and second questions were classified as having low risk aversion (i.e., high risk tolerance). Theoretically, one should expect those who score low in risk aversion to take more financial risks when faced with a choice in which the outcome of a decision is both unknown and potentially negative.

We also asked a series of traditional psychometrically-based risk-assessment items. The questionnaire used in the study was developed using classical test theory as a guide for question selection. The questionnaire has traditionally shown excellent validity and reliability. The types of questions asked were similar to ones that are included in most robust psychological measures of risk tolerance.

Several weeks later, we randomly invited 40 participants to visit our lab at the University of Georgia. Each person who agreed to visit the lab received $US10. Once in the lab, the participant was asked if she or he would be interested in an opportunity to win an addition $10 or $20 by playing a simple game of chance (i.e., a monetary risk-taking activity). The question was asked as the person stood next to a Las Vegas style craps table, as shown in Figure 1. Those who opted out of the game were compensated for their time. Those who opted in were asked a series of follow-up questions.


Figure 1. Gaming Table Used in Study.

The advantage to using a game of chance is that the odds of winning and losing a bet are predetermined. Also, games of chance add a sense of realism to risk-taking activities. The game was a simple one. Those who indicated a willingness to gamble were asked to choose between a wager that would return an additional $10 or one that would return $20. Here is the exact question:

Here is how the game works. You will be given a pair of dice to roll. You must wager your $10 gift card. In order to win $10, you must roll a 5, 6, 8, or 9.  If you roll any other number you will lose $10. In order to win $20, you must roll a 2, 3, 4, 11, or 12 to win; if you roll any other number you will lose $10. Which game would you like to play?

Each person’s choice was recorded. Participants were then allowed to take a practice roll of the dice. This was followed by the interviewer providing details about the true odds associated with the selected wager. Participants were allowed to change their bet, however, few moved off of their original choice.

At that point, participants played the game and either won or lost. It is important to note that the game was rigged, although participants did not know this at the time. Everyone left the game with US$30, even those who lost the wager and those who won only US$10. In other words, participants did not know prior to the game that they were guaranteed to leave with $US30, which gave the game a sense of reality. Participant behaviors related to the game were used as a way to validate the scores obtained from each participant on the economic and psychometric risk assessments.

Our team’s findings are somewhat controversial. So, it is important to say, at the outset, that this study was exploratory. If time and funding permits, we would like to use a larger sample and increase the incentives for participation. Nonetheless, financial advisors may want to consider the findings from the study in the context of the tools they are using to measure client risk tolerance.

As a reminder, those who scored highly in terms of risk tolerance—regardless of which assessment approach was used—should have exhibited a willingness to play the risky game. Risk tolerance scores should also have been positively correlated with other measures of risk taking, including the likelihood of gambling, investment experience, and investment knowledge. It turns out that only psychometric theory-based risk-tolerance scores were predictive of actual behavior. Equally interesting was the finding that only psychometric test scores were positively correlated with the other indicators of risk tolerance. Specifically, scores from the psychometric scale were correlated with knowledge of casino games, the likelihood of gambling, financial decision making experience, and investing knowledge, as well as participant holdings of cash and equities. Scores from the economic test were not predictive of behavior or correlated with the other measures of risk taking.

These findings should give pause to financial advisors who are considering adopting traditional relative risk aversion approaches as a means of assessing a client’s risk tolerance. A key reason to use a risk-tolerance tool is to anticipate who, among a group of clients, will be the most or least willing to take financial risks. The economic-revealed preference approach provided very little predictive power. The psychometric questionnaire was the only measure to predict who was more likely to participate in the risk-taking game where the outcomes of the game were potentially negative—a situation similar to what investors face in the markets.

Results from this exploratory study suggest that a questionnaire developed using psychometric theory is likely superior in terms of predicting financial risk taking behavior, at least when compared across the measurement techniques examined in the study. One takeaway from the study is that a financial advisor should ask for and receive information from a test developer about the validity and repeatability of the tool being presented and sold. If a test developer can prove these elements, then the selection of one assessment approach over the other should come down to advisor preference and theoretical training.

Posted: 29/04/2019 12:08:46 PM by John E. Grable, Ph.D.; Amy Hubble, Ph.D.; Michelle Kruger.