The Human Factor in Predictions and the Value of Unstructured Information
With the U.S. presidential election looming, it’s hard to avoid the talk of who’s ahead—everywhere you turn, there’s an article with the latest results of a new poll. Over the last 24 hours, I read two articles about predicting human behavior. David Brooks, poking fun at his ‘poll addiction’, supports the thesis that, while you can reach a certain level of predictability, essentially, human behavior is impossible to predict.
On the other end of the spectrum, I clicked over to an article that makes the case that what has been missing in building predictive models is the data. Now, the data is available in the form of social media content and will progressively more available in the future. Problem solved!
When we talk about models for predicting human behavior, I think we have to avoid the radical approach. As the political system demonstrates, we have made huge progress in predicting behaviors and reactions of the electorate, where elections are often won by a small margin, or even hanging chads in some cases.
But the objective cannot be perfection. We do not expect this from most of other models—we accept a margin of error. I believe that when we start including new data based on unstructured information, the margin of error in human behavior predictive models will not be eliminated completely, but it will shrink.
We will still have to wait for election night to know who the next president will be, but we will probably send out the party invitation the night before.
Author: Luca Scagliarini