Is more information better information?
Eight experienced bookmakers were shown a list of 88 variables found on a typical past performance chart on a horse, e.g. the weight to be carried, the number of races won, the performance in different conditions, etc. Each bookmaker was then asked to rank the pieces of information by importance.
Having done this, the bookmakers were then given data for 45 past races and asked to rank the top five horses in each race. Each bookmaker was given the past data in increments of the 5,10, 20 and 40 variables he had selected as most important. Hence each bookmaker predicted the outcome of each race four times – once for each of the information sets. For each prediction the bookmakers were asked to give a degree of confidence ranking in their forecast.
The chart below shows how both accuracy and confidence change as the information set grows over time.
Accuracy is pretty much a flat line regardless of the amount of information the bookmakers had at their disposal. However, confidence soared as the information set increased. With five pieces of information, accuracy and confidence were quite closely related. However, by the time 40 pieces of information were being used, accuracy was still exactly the same, but confidence has soared to over 30%.
So more information isn’t better information, it is what you do with it rather than how much you can get that truly matters.
Source: Behavioral problems adhering to a decision policy (pdf)