Overfitting in Corporate Recruiting

I read an interesting post from Maxwell Tabarrok about overfitting in academia. Much of what he discusses in that post is also relevant to corporate recruiting.

First, let's define the term:

Overfitting is a term used in statistics that refers to a modeling error that occurs when a function corresponds too closely to a particular set of data. As a result, overfitting may fail to fit additional data, and this may affect the accuracy of predicting future observations.

Going back to Maxwell's post, he notes:

Academia has an overfitting problem. Candidates are rigorously tested for decades on training tasks like GPA, Ivy League admission, reproduction of existing research, internships with field leaders, and citations for extending current paradigms.

But our selection criteria do not match with what we want academics to do. The out of sample prediction ability of academics is worse than random and laymen. They know lots of research they've already been tested on, but are rarely better than random at guessing the results of experiments in their field that they haven't seen before. Citations and academic rank do not increase the ability of economists to predict the outcome of economic experiments or political events.

Maxwell essentially argues that the academy says it wants one thing (progress) but it makes hiring selections based on a different thing (data which don't indicate a propensity to progress).

This sounds very much like what I've witnessed in corporate recruiting. Companies typically create a job description, and then recruiters sort through resumes looking for evidence of conforming to the job description. If a job description specifies, say, five years of financial modeling experience, then the resume had better demonstrate five years of financial modeling experience.

A recruiter will tell you that this is done in the name of efficiency: companies frequently get dozens, if not hundreds, of applicants for each job opening. And a recruiter has to quickly winnow that pile of possibilities into a few people to call in for interviews. What other heuristic is available to the recruiter, than to pattern match given the description provided?

And, from the recruiter's perspective, that's a reasonable argument to make. The recruiter's incentive is to provide the hiring manager with a pipeline of plausible candidates. The recruiter's job is definitely not to say "Wait a minute. We say we want dynamic and adaptable employees who are fast learners, but I'm selecting potential hires on the basis of what they've already proved, not what they can learn!"

Is there a solution here? In the context of academia, Maxwell notes that:

[T]he signs of overfitting are clear so we'd probably be better off if we required fewer credentials and less experience to do academic research.

This is part of the reason why Fast Grants, and Emergent Ventures work so well. There are obvious direct benefits from short applications and quick funding for urgent research.

I don't think that an equivalent to Fast Grants or Emergent Ventures makes sense in the corporate world. I do think, however, that companies that successfully figure out how to hire on the basis of potential will fare better. The past is prologue, to quote Shakespeare, but solely relying on past data to make hiring decisions means companies run the risk of overfitting their data.

The smartest companies realize this, of course, and recruiting at these companies is increasingly moving away from standard resumes to a more holistic view of applicants. Take, for example Jim O'Shaughnessy's comments in one of his podcasts:

• We are moving from a physical world towards a more digital one and we have to adapt our skillsets to maintain a competitive edge. Some tips on how to prepare:
  • Be endlessly curious. Consume as much knowledge as you can from various sources.
  • Your "proof of work" is always on display online and is more important than your Ivy League degree
  • Don't become prematurely certain. Keep in mind, "We are deterministic thinkers living in a probabilistic world."

Companies that can source applicants who demonstrate these behaviors will be sourcing applicants who are positioned for the future. These companies seem less likely to risk overfitting the data they use to make hiring decisions.

The world is changing rapidly, and this provides innovative companies with an incentive to avoid overfitting. Hopefully more of them figure this out.