Quantifying Qualitative Factors: The Increasingly Empirical Nature of Legal Scholarship

During my first official week as a PhD candidate at M-EPLI, I had the opportunity to spend the week in Rotterdam to attend Professor Jonathan Klick’s lecture regarding the increasingly empirical nature of legal scholarship. While the more traditional approach to legal research – relying on descriptive and observational analysis – will likely never go out of fashion, Professor Klick (a Professor of Law at the University of Pennsylvania Law School and the Erasmus Chair of Empirical Legal Studies at Erasmus School of Law) presented series of arguments about the value of applying regression models into legal analysis, particularly as it relates to the issue of causality (i.e. “how would promulgating a particular law impact the behaviour of the citizens?”). Majority of the lecture was spent on how empirical research can overcome limitations such as the unquantifiable variable problem and coping with bias concerns, but Professor Klick made strong arguments for empirical analysis by praising the use of highly computational analysis and the application of bias controlling measures such as propensity score matching.

Along with my colleague Catalina Goanta, who also attended this lecture, I had with me, my admitted aversion, if not fear, of trusting empirical research in legal analysis. My rebuttable assumption about the utility of empirical research in legal scholarship – especially in a field so vast and diverse as private law – has always been that one particular study about a certain group cannot be broadly applied to the general legal community as a whole. In other words, in my humble opinion, it is almost impossible for legal empirical study to rid itself of the external validity problem, unless the sample size of the research is the general legal community as a whole, which admittedly would be severely cost prohibitive.

Given my particular phobia, I found the discussion regarding the subject of Omitted Variable Bias (OVB) to be one of the most interesting subjects discussed during this four day lecture series, if only because I felt that the very existence of this bias weakened Professor Klick’s argument for relying on empirical research to predict behavioural patterns of people to an attributable law. In essence, OVB occurs when a model or a regression omits an important causal factor that would have otherwise impacted the outcome of the model. For example, it is widely accepted that there is at least a positive correlation between a person’s education level and their income level. However, it is rather difficult to make a causal inference between education and income because other factors such as one’s IQ or their socio-economic status could be omitted variables that would make it difficult for one to argue with absolute certainty that higher education – not higher IQ – “causes” higher income. Other limiting factors such as unobservable characteristics or subjectivity contribute to OVB and given the assumption that intangible factors will continue to persist in the human decision-making process, the task of truly isolating a person’s behavioural change to an attributable change in law will continue to be a rather arduous and daunting task.

In the end, as “scientific” as these empirical research and regression analysis can seem, Professor Klick admitted that using data to validate an assumption must rely on using personal intuition (or in layman’s terms: “gut feeling”), and so long as people continue to possess different intuitions, empirical research will remain an imperfect science regardless of the multitude of error and deviation minimizing mechanisms available. In the words of the great professor himself, “econometrics is much about art, as it is about science”. Although I learned a great deal about regression analysis and how it can be applied to legal research from this lecture, I still could not quash away my lingering doubts about the true utility of relying on regression analysis given how embedded certain biases tend to be and the overall lack of external validity with such an analysis.

I must note here in concluding that it is my sincere hope and wish that some time in the next four years here at M-EPLI, I will eventually see the light and learn to use – if not to love – empirical research in legal scholarship. Hopefully, this epiphany or revelation will happen prior to the writing of my thesis, but in the event that it does not, I hope my new colleagues will be kind enough to hit me over the head with a clog.