Thread by @OsmSiddiqi, Story: Several applicants from #Nairobi for a job I'm hiring for knew [...]

Story: Several applicants from #Nairobi for a job I'm hiring for knew more in econometrics than me (not that I'm a maestro), a former TA at HKS, our textbook (judging you Wooldridge), 3 other grads from HKS, 2 in a PhD track for econ at Berkeley & McGill, and a Fletcher grad.

It's really fascinating from a behavioral perspective, inspiring, and quite worrying all at once.

We created a test to assess applicants' strengths in critical thinking, rhetoric, and yes econometrics. One question was a seemingly simple one on interpreting a ln regression.

In this case, the dependent variable was in natural log form (or as economists call it, log form...) - classic ln (income). The independent was binary in nature.

When creating the test, I flippantly asked a team member to choose any coefficient & ask applicants to interpret it.

She chose the coefficient of 0.54. Am I glad she did.

I want you to pause here for a second and think about how you would interpret this basic regression here:

ln (income) = B0 + 0.54*B1 + e....

This might be a good place to add a poll, before I continue the story...my first!

Ass. == associated
Inc. == increased by
B1 == increase of B1 by 1

All right fine, I'll get on with it.

...
...
...
...

Now....promise you didn't read ahead?

The first applicant I reviewed marked this as 71.6% associated increase with a 1 unit increase in B1. I was like...whaaat? How? Marked it as wrong.

Some applicants followed. Here and there answers.

One said the same as the above but 54%. Aha! Correct!

Or so I thought...

...

Then another applicant had 71.6% marked...and another...and then yet another...

This made me question what's up. What are these kids being taught? You know? So I opened up my econometrics textbook from grad school just to verify *in case*.

So sure I was right because, damn, I've known this for 7 years. Aha! Wooldridge! Validated. Here's the excerpt. No squiggly equals, nothing. Equals is equals. It is truth/fact/absolute.

Except it's wrong. Not by a lot in this example, but wrong.

Turns out, with simple searches online, this form of interpretation is simply a rule of thumb interpretation for 'small coefficients'.

Well, one website had a coefficient of 0.198 - which is actually a 22% associated change. That's different enough.

https://data.library.virginia.edu/interpreting-log-transformations-in-a-linear-model/

So what's a small coefficient? I did what you would probably do at this stage. Scratch my head/beard & write the equation myself / break it down. The formula on Virginia's website made sense. But I have an econometrics textbook saying -- not much about the formula at all.

So I reached out to fellow grads from my program, my girlfriend who's (way more) technical than I am (from Fletcher), & two students in PhD programs in Berkeley and McGill.

My gf to her credit had faced this conundrum before *after graduating* so she got it (mostly).

Another fellow grad had once made this question a part of training new employees into her company -- only to realize that the training she was offering was wrong and the new employees *corrected* it.

Other grads, including PhD track ones, were momentarily flabbergasted.

Eventually, we all realized - our training didn't quite emphasize this nuance (if it was mentioned at all) and 'small' coefficient might be highly subjective to the question on hand for a policy/program/AB test. For the latter, one might underestimate the difference in results.

I've marked everyone who said 54% and 71.6% correct because - if I was taught the former & it was wrong - how can I judge this response? It's wrong yes, but the candidate learned/retained the info well and communicated it well as well. The ones who got 71.6% though were bang on.

So, where did my girlfriend face this challenge? She once took an analytics test as part of an interview for a firm started by....fellow alum of my program.

She applied the *actual formula* to get the interpretation and...none of the multiple choice answers had her answer...

If you remember what this felt like in tests -- this is a moment of complete freak-out. What is going wrong? Did I do the math wrong/read the question wrong? It's a moment of extreme doubt - you assume the question is correct (authority). She chose the closest number and contd.

So we have trainees correcting trainers, applicants correcting reviewers, and applicants getting frustrated by test questions that were, well, wrong. How crazy is this? This is crazy to me. Absolutely crazy.

Btw, a former TA for the class - also did not know.

But everyone believed this to be the answer. Our technical peers did. A book validated it. So we believed the wrong interpretation to be truth. And we hold degrees/brands/jobs that make us authorities. This is scary stuff. This is herd behavior - being clear on a false 'truth'.

Luckily for me (and these applicants!!) this is the first time I'm using this question as a possible exclusionary criteria for selection & I *happened* to catch it in time bcoz enough applicants had the correct 71.6% ans. If it was only 1...maybe they wouldn't have gone through.

Have a lot of us who may have used this specific question caused harm to possible applicants who just knew better than us? Or students?

My gut says...yes. Probably. On the margins a mark here or there in large applicant pools can be make or break. And is always heartbreaking.

This is why I'm putting this out here *in case* this course corrects other folk who may have made this mistake or aren't aware that there are different interpretations being taught that need to be considered for an otherwise really simple math formula.

Some smart kids could have missed out on a chance because of my ignorance. Worth noting my ignorance on this stems from a possibly general issue in the way econometrics/econ can sometimes be taught. Rules of thumb. Not facts. Even with all the math insistence in our progs.

Keen to hear if others have faced the same problem. Clearly enough econ grads, PhD students, a textbook, a TA, different unis, and employers have for me to bet this is maybe a bit wider than just this anonymized list. /end

I should have mentioned - the job I'm hiring for is completely entry level and not "lead econometrician" though maybe I should change the role title now...

A friend just now put it best, perhaps

"The best kind of interview question is one that teaches the interviewer a thing or two." Hopefully this teaches many of us something. Many other teachable moments have come up while I've been hiring over the past many years. One day...

Latest Threads Unrolled: