Full Disclosure

In a Washington Times editorial today, Professors Todd Zywicki and Gail Heriot wrote an op-ed entitled "Junk Social Science Index" in which they criticized the testimony of Professor Elizabeth Warren and Dr. David Himmelstein before the U.S. House of Representatives' Subcommittee on Commercial and Administrative Law. Zywicki and Heriot attack Warren and Himmelstein's work on medical bankruptcies. Warren has posted a reply here on the Talking Points Memo blog where she thoroughly addresses the spurious claims Zywicki and Heriot make about her data. In the academic journal where their article first appeared, Warren, Himmelstein and their co-authors wrote a reply to their critics. Zywicki and Heriot do not discuss this reply, which begins with a critical point--all of Warren and Himmelstein's data are reported in the article and open for researchers to do their own interpretation, which is all Zywicki and Heriot did. They have no new data of their own.

What bothers me most is the tone that Zywicki and Heriot adopt. Rather than acknowledge that they are offering their own interpretation of Warren and Himmelstein's data, Zywicki and Heriot descend into a fit of innuendo and name-calling. They condescendingly refer to Warren, the Leo Gottlieb Professor of Law at Harvard Law School, as "Miss Warren" and to Himmelstein, an associate professor of medicine at Harvard Medical School and a practicing physician, as "Mr. Himmelstein." Zywicki and Heriot characterize Warren and Himmelstein's work as "junk" and "one of the most misleading pieces of research ever placed before Congress." They dismiss the entire committee hearing, and Warren and Himmelstein by association, as "just another arm of the publicity leviathan behind Michael Moore's new 'documentary' titled 'Sicko.'" Zywicki and Heriot apparently cannot even acknowledge Sicko as a documentary without some qualification.

Who is the publicity leviathan? It's only fair that people realize the source of these harsh characterizations. The Washington Times is well known for its radically conservative political views. Other commentary in the paper today included an op-ed suggesting that U.S. failure to stop Iran's acquisition of nuclear weapons was tantamount to France's failure to stop Hitler before World War II and an editorial not merely disagreeing with but actually accusing congressional Democrats of undermining the war effort. Zywicki will be known to many readers of this blog as the one legal academic who continually testifies to Congress in support of the consumer credit industry. For example, under the auspices of the pro-business Mercatus Center, he recently joined a report arguing against any restrictions on the subprime lending industry. Zywicki also served as head of the FTC's Office of Policy Planning in 2003-2004. Heriot is a member of a blog called "The Right Coast," where she recently blogged about her "conservative/libertarian instincts" (ironically enough in a post about when she was a volunteer for George McGovern). According to her web site, Heriot was counsel to the Senate Judiciary Committee under Republican Senator Orrin Hatch, a strong proponent of the 2005 bankruptcy law.

Views are not wrong merely because of the politics of the persons who hold them. But the politics of academics who hurl invective and call others' work "junk" while doing no data collection of their own should alert us when we have polemicists rather than dispassionate analysts.

FULL DISCLOSURE OF MY OWN: I should be so flattered to be part of the publicity leviathan for anything, but I'll make clear that Warren is a colleague, a co-author, a co-blogger with me here on Credit Slips, and a good friend. Both Warren and Himmelstein are collaborators of mine in the research project that led to the data collection involved in that article and an ongoing data collection effort.

Comments

[Josh Wright]

Bob, I get that you don't like the tone. Fair enough. But I find a few of your comments puzzling and have some questions so I don't misinterpret your post:

First, do you really believe that the claims Zywicki and Heriot make about the data and methodology of the study are "spurious?" In other words, do you believe that the study properly identifies, in the statistical sense, a causal relationship between medical debt and bankruptcy? Do you think it would survive peer review in an econ journal? Do any critiques of the methodology that folks other than Zywicki have raised strike you as significant?

Second, did you really mean that we should discount critics who merely interpret data rather than produce new data as biased? That strikes me as an odd thing for a researcher to say. Professors Warren et al.'s efforts to collect and report this data are certainly admirable. But we can all agree, I hope, that the issue is whether the data show the relationship asserted. Its just not enough to say "we recognize other people might have made other decisions with the data" and assert identification. Other researchers are supposed to interpret others data in social science and point out when it doesn't carry the burden. This happens quite frequently thank goodness. As it did in this instance where several parties have done so. Maybe you just mean that critics shouldn't be mean when they interpret data?

Third, what's up with the gratuitous shot against the "pro-business Mercatus Center?" Mercatus is home to a whole fleet of accomplished scholars including at least one Nobel winner (Vernon Smith) and another should-be winner (Gordon Tullock). It is disingenuous to imply that mere association with that organization suggests pro-business "bias."

***And in the spirit of the title of your post, my own disclosures: Professor Zywicki is a colleague at GMU and a friend. I don't take money from the credit card industry, I am not a member of Mercatus, or the Washington Times for that matter ...

[Full Disclosure]

The only thing better than your "certainly admirable" fluffing of Zywicki is your notion that "surviving peer review" is relevant at all...

I also think it's hilarious that Zywicki acts as if he is "above it all".

Let's see, we have him making a fool of himself over biology:

http://www.volokh.com/posts/1120775700.shtml

Making a fool of himself on medical issues:

http://timlambert.org/2004/09/zywicki/

Oh and I do love the Subsequent high road he took with PZ Meyers... calling him:

"infamous Lysenkoist"

And then subsequently editing it...

http://scienceblogs.com/dispatches/2005/11/email_from_zywicki.php#more

Or his foray into media criticism:

http://prawfsblawg.blogs.com/prawfsblawg/2005/06/an_unlikely_dia.html

[Katie Porter]

In response to Prof. Wright's comments, let me note that the journal that published the original piece by Himmelstein, Thorne, Warren & Woolhandler WAS peer-reviewed. No, it wasn't an "economics journal;" it was the nation's most prominent health policy journal (Health Affairs), which seems perfectly appropriate for the content. I doubt Prof. Wright meant to suggest that economics peer review is the only legitimate system.

I think that Zywicki and Heriot's op-ed used a needless agressive tone that weakens their credibility as scholarly critics. Rather than unfairly attacking Warren & Himmelstein's data as illegitimate and unfairly inflating the relevance the UST's own study--which even the UST admits does not capture some of the types of relationships between medical problems and bankruptcy that Congress ITSELF asked about (such as medical bills paid by credit card or debts to doctors or hospitals turned over to bill collectors)--Profs. Zywicki and Heriot could have noted the power of a term like "medical bankruptcy" to suggest that financial distress is usually caused by one factor (and they could have argued that it is not). Or they could have emphasized that studying bankruptcy debtors' bills doesn't full reflect how MOST Americans (who don't go bankrupt) deal with medical bills and that more research on that would be useful for policymaking.

I don't necessarily ascribe to either of the above critiques of Profs. Himmelstein/Warren but Prof. Wright seemed to be asking if Prof. Lawless thought one COULD legitimately criticize their empirical work and so I offer the above as examples. The reality is that Profs. Zywicki and Heriot's op-end didn't choose to make a careful critique of Profs. Himmelstein/Warren or appropriate praise of other other studies. And that's the thrust of Prof. Lawless's original post--they engage in innuendo and name-calling.

[Josh Wright]

Professor Porter:

Thank you for your thoughts in response to my comment.

You are right that I certainly did not mean to imply that economics peer review was the only peer review. I am certainly not arguing that the articles placement in a non-econ journal makes the analysis less reliable.

The point was that the economics standard provides a useful basis for comparison that Professor Lawless and I can use because I dont know the review standards for journals in fields outside economics, but I suspect that Professor Lawless has a feeling for the standards in econ journals. Other systems of peer review may, for example, be less concerned about issues of causation and proper identification than the economics system. This issue is critical to the policy debate and I am asking Professor Lawless how well specified he thinks the causal relationship is on terms that I can understand.

And no, I was not asking if one COULD legitimately criticize empirical work. Of course one can, and I will go so far as to say should. Both Professor Lawless and Professor Warren refer multiple times to the fact that the critics haven't collected their own data --- I was (I think pretty clearly) asking why this is relevant to the value of the critique? Critiques from folks who did not collect data are fairly common in the social sciences.

In addition to taking offense to the tone of the op-ed, Professor Lawless argued that the critiques were spurious. I am interested in why he reaches this conclusion, and whether he thinks the study has identified a causal relationship. I think these are not only fair, but interesting questions that are or should be at the heart of the policy debate.

[Katie Porter]

Prof. Wright: I appreciate your helpful clarification. I'll limit my response to one of your points. I think there is a difference between a critique from someone who has NEVER collected any data and someone who didn't collect data on the particular point of controversy. The former has experience in study design, methodology, and statistical interpretation. The latter *may* lack such experience, and I think that can hinder their ability to adequately review empirical work. For this reason, peer reviewed journals normally seek reviewers who have used a similar methodology because they believe those scholars are better able to evalute a study's merit.

Also, I do not believe that Profs. Zywicki and Heriot advanced in the op-ed, or have ever advanced, what they believe WOULD be a better way to examine the particular findings of Profs. Himmelstein and Warren, and I think this fact partially drives the "you have no data" response. Let me illustrate with an example. Credit card debts appear on bankruptcy court records as something like this: "Bank One Visa, $8,900." Relying on court records alone as the UST does means that you don't know that $8,000 of this $8,900 is from a hospital stay for open heart surgery. If Profs. Zywicki and Heriot are suggesting that putting the hospital stay on a credit card causes that debt to no longer be "medical," I think that position is indefensible. If they would agree the $8,000 hospital stay is a medical debt, then how exactly do they think one should identify that relationship? Profs. Himmelstein and Warren used a survey and it had a cut-off of $1,000 or more. If $1,000 is too little of an indicator of medical debt (although let me note that the op-ed's comparison to the general American population is probably inappropriate--the median bankruptcy filer had an income that was half of the median American so the amount of medical debt it would take to harm a near-bankrupt family would be significantly less), then what number is an appropropriate cut-off? By failing to suggest and justify improvements, Profs. Zywicki and Heriot do not advance the state of knowledge. It is not only that they have no data to refute Profs. Himmelstein & Warren; they seem unwilling to participate in the collective empirical project by building an appropriate critique that respects the potential of research to examine this issue.

[Full Disclosure]

Josh Wright:

"In other words, do you believe that the study properly identifies, in the statistical sense, a causal relationship between medical debt and bankruptcy? Do you think it would survive peer review in an econ journal?"

Katie Porter:

In response to Prof. Wright's comments, let me note that the journal that published the original piece by Himmelstein, Thorne, Warren & Woolhandler WAS peer-reviewed. No, it wasn't an "economics journal;" it was the nation's most prominent health policy journal (Health Affairs), which seems perfectly appropriate for the content.

Josh Wright:

"The point was that the economics standard provides a useful basis for comparison that Professor Lawless and I can use because I dont know the review standards for journals in fields outside economics, but I suspect that Professor Lawless has a feeling for the standards in econ journals."

*

Did you have a straight face when you clarified "the point"?

[Ronald Mann]

To respond to Josh specifically, I do think that the critiques are spurious, primarily because they ignore the nuanced nature of what appears in the peer-reviewed publication in question. The "critiques" amount to little more than regurgitation of the passages in the Himmelstein/Warren paper that discuss the limitations of their data and analysis. For what it's worth, I have little doubt, if there were an economics journal for which the subject matter was appropriate, that the discussion would pass peer review. Health Affairs is simply a more appropriate and prestigious journal for this type of work.

With respect to causation, you suggest that the issue of causation is critical to the policy debate, but it is important to distinguish the policy debate in which Todd is engaged from the scholarly enterprise reflected in the paper and the various related books that Professor Warren has published. Collectively, the work articulates the view that the causation question is a complex one, because bankruptcy arises from the interaction of numerous factors including not only lending practices, but also social welfare systems and exogenous shocks such as unexpected medical events. I assume that Professor Warren would be the last to claim that her Health Affairs paper "proves" that medical problems "cause" bankruptcy.

With respect to the distinction between collecting data and evaluating data that others have collected, the problem here is that there is a great deal of data, collected by various researchers in many different disciplines, tending to document a reality of bankruptcy as a complex socioeconomic phenomenon. Todd's approach, in contrast, and I generalize, is to reduce all bankruptcy filings to the single factor of moral failing on the part of those who file, a factor that is conveniently impossible to quantify. As the body of data grows, there comes a point where a reasonable observer of the debate might discount the unitary "bad actor" claim in the absence of substantial problems with the existing data or some new data.

To me, this current debate reflects a shift in political perspective from the BAPCPA hearings (which largely accepted the unitary contributor approach) to the more recent hearings (which are looking at many other contributing factors). In the end, it is one thing to say that Professor Warren's work does not prove as much about the relation between medical issues and bankruptcy as Professor Warren (or other researchers or policy activists) might say it does. It is quite another -- and not a statement that an academic could take seriously -- to suggest that this is the worst junk science ever presented to Congress. Of course, the former statement would not land an editorial in the Washington Times. But how much of a justification does that provide for someone with an academic career?

[John Pottow]

Josh, I think Warren et al. were appropriately circumspect in their causation claims. E.g., from the section on "Study Limitations" in the article: "Even when data are reliable, making causal inferences from a cross-sectional study such as ours is perilous. Many debtors described a complex web of problems involving illness, work, and family. Dissecting medical problems from other causes of bankruptcy is difficult. We cannot presume that eliminating the medical antecedents of bankruptcy would have prevented all the filings we classified as 'medical bankruptcies.'"

Do I think there are valid methodological critiques available to the Warren et al. study? Of course I do. One of them, for example, is whether $1,000 out-of-pocket medical expenses is too low (or too high) a "trigger." But I note that often critiques that start along this line conflate one metric of medical bankruptcy -- exceeding a trigger level of out-of-pocket bills -- with the ultimate tally. (Indeed, the Times Op Ed does just this when it implies, falsely, that Warren et al. claimed over half of bankruptcies were caused by "medical debts.") I should also point out the overlap in the metrics that led to the ultimate 54% number. If one just added up all the potential metrics used in the study, the total would be around 97%. The reduction of this to 54% means that many respondents qualified under multiple metrics: e.g., out-of-pocket expenses over $1,000 (one metric) AND, e.g., cited "illness or injury" in response to survey questionnaire (another metric).

I guess all that's left is for me to echo what I took to be Ronald's intuition: with the greatest of respect for your colleagues, scholarship and punditry rarely go well together.

[Josh Wright]

Ronald: Of course I would agree that the causation issue is complex. I would be highly suspect of anybody who argued that bankruptcy had a singular cause. And while I'm no expert in this field, I do not read Professor Zywicki to focus solely on the "moral factor" in his work despite your claim here. You both have characterized each others work in enough places and enough times that readers can decide that for themselves.

In any event, a properly identified relationship, as you know, need not rule out multiple causal factors. So I agree that the relationship is obviously highly complex. But certainly you agree that it is important to know the relative magnitude of these causal effects for policy purposes.

The real problem is in quantifying just how important these multiple causes are. The Warren et al results suggest they are a very important part of the story. Certainly the quality of these estimates depends on the quality of the data, methodology, and how well the relationship is identified. Thus my question concerning how much we should trust the estimates of the relationship. I read Professor Lawless, and now others here, as placing full faith in the estimates --- which I find interesting given the complexities of identifying this sort of relationship statistically that you point out.

John Pottow: As I noted in the previous post, the proper amount of circumspection is all well and good, but my question relates primarily to how well we should trust the estimates concerning the magnitude of the impact of medical debt on bankruptcy. Being circumspect about your data and methods is fine and appropriate --- but the question remains: what do the estimates tell us about the "true" relationship?

Directed to nobody in particular: surprisingly little of this discussion seems aimed at figuring out how much we should trust these estimates to guide policy decisions.

[John Pottow]

Josh, I guess my take is I would trust these data to a strong degree. I don't know what more you want than that. If it's a quantification, then my answer is 8. OK, more seriously, let's take the metric of the questionnaire response to reasons for filing bankruptcy. If a debtor checks "illness/injury" as a source for why she filed for bankruptcy, then I feel eminently comfortable saying that person's bankruptcy had a "medical cause" as a "true relationship." Various response bias considerations are addressed in the Warren et al. paper, and I find the discussions thereof persuasive (e.g., that debtors in the context of the instrument are unlikely to by lying given their candid answers to other personal questions). I don't have the Warren et al. paper in front of me anymore, but I'm pretty sure this one metric alone was about 25-30% of all respondents (i.e., about half of the so-called "medical bankruptcies"). Thus to answer your question about how reliable the methodology used in the paper is in gauging the "true" relationship, I would say that at least as regards to 25-30% of bankruptcies, very reliable indeed. I have little doubt that, as a lower bound, no fewer than 25-30% of bankruptcies have a "medical cause."

Also, I do not see these issues as unrelated. Circumspection is not just well and good, as you say (and you're right), it is in my view inextricably intertwined with the credibility of the researchers and hence the reliability of the data in capturing the "true" relationship. Authors who qualify their comments and engage in candid assessments of the limitations of their data accrue scholarly trust and increase the likelihood that they are selecting metrics that have high validity in capturing the underlying relationships.

I hope this helps. Also, one quick point: note that you too, despite the admonition in my very post to which you respond, are making the same conflation that I chastise the Op Ed for doing: this is not about the impact of medical DEBT on bankruptcy. This is about the incidence of medical CAUSES of bankruptcy (of which medical DEBT is a subset). The latter is a broader concern, and speaks more directly to the bad-agent debate well explored by other researchers.