Evans and Wright on the CFPA: Round 2

A couple of weeks ago I wrote a short critique of one piece of a long study written by David Evans and Joshua Wright about the Consumer Financial Protection Agency and funded by the American Bankers Association.  The related blog post is here.  Evans and Wright have responded.  

There's a lot that I thought was objectionable or questionable in Evans and Wrights study, but most of it was well within the bounds of reasonable argument.  I have no problem intellectually with arguments that any particular regulation could impose costs that outweigh its benefits.  Instead, I was was moved to write because Evans and Wright were making precise numerical claims about the cost impact of the CFPA, and that these claims were based on either (1) a highly questionable comparison to dissimilar regulation or (2) pure conjecture.  

In their reply, Evans and Wright spend a good deal of time arguing about things that are really beside the point to my critique.  For example, Evans and Wright emphasize that I have not proved the affirmative case for the CFPA's positive impact (a passing point I made to show that the economic impact of regulation is susceptible to multiple predictions) and that I have "disputed virtually none of [their] findings that the CFPA Act would impose high costs on lenders and ultimately result in denying borrowers choice."  Let's be clear.  My critique was about three spurious numbers.  I didn't set out to prove a positive case in the critique and don't need to do so to make my central point.  And to imply a concession from silence about other issues is ridiculous in this context.  This sort of logical move is, however, consistent with the problems with Evans and Wright's statistical claims.  

But let's get to the heart of the matter.  My issue with Evans and Wright is about the numbers, not about their priors regarding regulation.  There are three numerical claims in Evans and Wright's piece with which I took issue. First, Evans and Wright claim that a CFPA would result in a 160 basis point increase in the cost of credit and a derivative 2.1% decrease in credit demand.  These assertions were based on a comparison with a study of non-analogous regulations that have been found to have an 80 basis point impact.  Evans and Wright argue that even though the regulations are different, they are less invasive, so therefore at least twice the impact would be the lower bound. Why twice?  Just because.  Evans and Wright still have no justifiable basis for doubling, as opposed to tripling the number, etc.  It is not as if 160 basis points is within some statistical confidence interval or the like.  While a 160 basis point number appears to have the imprimatur of social science, it is just conjecture, or, to be charitable, a very rough guesstimate.  

In a cost-benefit analysis, however, precision matters.  A CFPA might be worthwhile at 120 basis points, but not at 160 basis points, for example.  The problem with Evans and Wright's methodology is that they can no better defend a 160 basis point number than a 120 basis point number or a 700 basis point number.  Evans and Wright emphasize that there were merely setting a lower bound, but that hardly makes their number more defensible.  Evans and Wright simply do not and cannot know the impact, including what the lower bound would be.  Of course, precision is beside the point if the goal is to produce a scare statistic, rather than a rigorous cost-benefit analysis.  

The third spurious statistic in Evans and Wright is a claim that a CFPA would result in 4.3% slower job creation.  They achieved this number by taking a statistic about the role of small startups in job creation and then "supposing" that a CFPA would inhibit five percent of this.  I noted there were problems with their job creation statistic (namely that it failed to account for the spectacular failure rate of small startups after their first year, when they result in net job loss, not creation).  But that was a side point.  The critical problem was that they "supposed" a impact number without any basis whatsoever for their supposition. 

Evans and Wright take issue with my statement that "The key point here, however, is the impact of the legislation is speculative and certainly not susceptible to precise statistical predictions.”   They state, "That is a nihilistic approach."  Actually, it is an intellectually honest approach.  A debate that is poisoned by spurious statistical claims, rather than their debunking, are what will engender nihilism.  It'd be great to have an empirically informed policy debate.  But that's not license to make up numbers.  Policy debates have to function within our epistemological limitations.  There's a constructive debate to be had about the CFPA.  But constructive doesn't mean making things up.  

Comments

[Chabot]

Good grief, man, you're arguing about numbers. And you say, in no uncertain terms, that you are. Isn't this downright duplicitous? Or at least, incredibly ironic?

[JD]

Chabot,

What's your point? Did you even read this post? He's arguing that they are fabricating numbers, pulling them out of thin air. They came up with these very specific numbers to support their theory and cannot in any way back them up.

[Chabot]

JD,

Longtime fans of this site know what I'm talking about. Basically, it's a question of whether Prof. Levitin is being hypocritical by calling out others about the legitimacy of their numbers. Making up numbers isn't an offense that is unique to Dr. Evans. In fact, one could argue that Prof. Levitin has, on occasion, been guilty of it. That's all I was saying. (If you still more guidance, search this site for "Home Depot.")

[Adam Levitin]

Chabot: Don't be coy. You've had a bee in your bonnet about me for a while. But if you're going to accuse me of making up numbers, you'd better be specific. There's a lot of numbers in that post about Home Depot. What's your beef? Here are the numbers in the post. Tell me what you think is bs and why.

(1) Home Depot's interchange costs have risen 16% in recent years, while purchase volume has increased 10%. The source is The Home Depot itself, at a Federal Reserve conference.

(2) $48 bn in interchange last year. That's sourced to a merchant group. Visa tells me they think it was more like $42 bn. But what's a few billion between friends? We don't know interchange amounts with precision, but we know it's in this range.

(3) Potential drop in interchange to 50 bp. That's based off of the rates post-reform in Australia and the EU.

(4) Bureau of Labor Statistics average total cost per employee--that seems pretty legit.

I'm guessing what you don't like is that I did a 100% conversion of estimated interchange savings to employment. Let me be the first to say that is not how things would ever work. Some of the savings will be pocketed by merchants, some will be passed on in lower prices, some in better service, some in R&D and infrastructure development. The reason I presented it all in employment was because the Home Depot's comments brought home that interchange has an employment angle and because jobs are a common unit for measuring economic impact of reforms. It was to show a different sort of point and made no claim of exactitude. It was prefaced as a "back of the envelope" calculation.

If you can't see the difference between this sort of casual illustrative calculation in a blog post and a faux empirical finding in a long written paper in which numbers are made up wholesale, then there's nothing to discuss with you.

[Chabot]

I suppose that there are two types of "empirical" issues that one could debate. The first would be inputs. In your (Adam's) response, these would seem to be the four numbered points that you make. Conceivably, I suppose that one could quibble about those. I don't have any problem with, much less any info about, the numbers that you cite. And a challenge of that type would seem to be counterproductive, if not downright silly. After all, the entire point of this site, I assume, is to debate ideas, rather than to bicker over a billion dollars here and there, or to argue over a few bps.

The more substantive empirical issue would seem to be how one uses those inputs. How does one use the four facts and what does one conclude from them? As you surmise, that's the problem that I have with your "Home Depot" calculations. Based on my understanding of the discussion surrounding that post, my impression is that you used some empirical facts to generate a provocative conclusion. A conclusion that is completely supported by the "facts" but is so dependent on the underlying assumptions that it's largely indefensible.

My objection is that there's a certain hypocrisy involved in calling out Evans and Wright about their provocative conclusions regarding the CFPA while being willing to post conjectures about Home Depot that are similarly provocative. Perhaps Evans and Wright make a mistake of the first sort so that they use bad inputs. But my impression is that you mostly dislike their outputs and the way that they reached them. I don't understand how complaints about their outputs and methodology are applicable to Evans and Wright, but not to you.

What I also don't understand, and perhaps this is a reason not to have a discussion with me, is why a non-peer-reviewed paper posted on the web is that much different from a blog post on the web. Based on what I've seen on your site, my impression is that Evans and Wright are hired hacks. Ok, so what. They know how to produce a professional-looking pdf that they post on the web. In contrast, your Home Depot post is unprofessional? It involves no bias or point of view? Or are you just less rigorous in your blog posts, particularly relative to the work of the eminent Evans and Wright? Unless things are peer-reviewed, my understanding of the internet is that the line has been blurred between "real" and "fake" content. So I'm not sure that distinction matters. It probably doesn't for Evans and Wright.

At the end of the day, hypocrisy bothers me. And my view is that the basic premise of your criticism of Evans and Wright is hypocritical. Perhaps I'm missing some distinction between the objectivity of the calculations. Or I don't understand the data. Or I don't understand the internet. But in light of your willingness to post your own provocative calculations, your criticism of Evans and Wright still bothers me. That's my main point.

[Adam Levitin]

Chabot: My Home Depot employment numbers are not "facts." It's not an empirical study. It's the result of reasoning by analogy. That's the methodology. The same is true of Evans and Wright. Reasoning by analogy is a basic mode of logic. But not all extrapolations are logical.

There are two fundamental distinctions between my interchange extrapolation and Evans and Wright's CFPA extrapolation. First, Evans and Wright invent some of their inputs. That makes the output inherently suspect. That's a pretty major distinction. Second, my calculation is an extrapolation from international interchange regulation to the US, while theirs is from a study of interstate branch bank restrictions to the CFPA. I'm going from apples to apples across country lines, they are going from applies to raspberries. One might argue that interchange wouldn't have a 1:1 transaction internationally. I don't see any immediate reason why it wouldn't, however. Should it be inherently higher in the US than in Australia or cross-border in the EU?

The only thing my argument on interchange has in common with Evans and Wright is that they both involve arguments by extrapolative analogy. I don't think there's anything hypocritical for me to criticize their extrapolative analogy because it is done illogically.

I'm still not sure what you don't like about my Home Depot post. You say it is a conclusion that is completely supported by "'facts' but is so dependent on the underlying assumptions that it 's largely indefensible." What's the problematic underlying assumption? The 1:1 extrapolation of interchange regulation impact? That's not what other commentators didn't like. They objected to my putting the impact all in a constant unit (employment) or to the gestalt of interchange regulation period. That's not a problem with my reasoning. That's a problem with my presentation of results.