We have performed further analyses of the experimental�control and pre�post data using the statistical methodology that (at least at this point) Rossi seems to prefer: Probit and logit with robust (Huber-sandwich) standard errors. Table 1 compares estimated treatment coefficients and standard errors for our experimental design models for two outcomes (own births and abortions)[1] using two estimation methods: Probit and Huber-adjusted probit. Table 2 provides analogous information from our pre�post analysis.

            Different estimation methods do not yield appreciably different coefficient standard errors or, therefore, warrant any change in our inferences regarding the impact of FDP on births or abortions. There is simply no empirical support for Rossi�s speculations about the possible impact of serial dependence of the observations.

Rather than withdraw his speculation, Rossi now resorts to even more speculation, asserting that the �other deficiencies� in our work �are sufficiently serious� that even estimates based on his own preferred statistical method, Huber-corrected probit, �are not credible.� This remarkable claim, which Rossi did not make in any of the previous drafts of his critique, raises an interesting question: If even the Huber-corrected estimates are �not credible,� why did Rossi argue so strenuously for computing them? 

Does the Pre�Post Analysis Suffer from Omitted-Variables Bias?
Rossi�s second major criticism is that our pre�post analysis �cannot take into account the effects of time or other events that might affect outcomes.� However, Rossi conspicuously neglects to identify any variables that we should have included but did not. He does note that the period from 1991 to 1996 saw increases in the proportion of Aid to Families with Dependent Children (AFDC) cases with either an ineligible adult caretaker or a never-married woman, and he speculates that �the effects estimated by the pre�post analysis are likely confounded with those changes.� However, he apparently does not realize that we included variables that explicitly identify the presence of ineligible adult caretakers and never-married women. Are there other variables that we should have included but did not? Rossi says nothing about what these other variables might be.

Rossi also says that the FDP evaluation should have done follow-up studies of births and abortions for women (in both the experimental and the control groups) after leaving AFDC. We endorse this idea, particularly because the evaluation contract did not provide for follow-up data collection of this nature and because, so far as we are aware, no such follow-up analysis has ever been funded. We would also note that, by its very nature, any such follow-up analysis of post-AFDC behavior in New Jersey would have to rely on respondents� self-reporting (which can be quite unreliable) instead of on administrative records and might well suffer from attrition. Thus, such an analysis might well be more �problematic� than Rossi seems to realize.

The evaluation of New Jersey�s FDP is the only completed evaluation of a family-cap policy that includes analysis of births, abortions, family-planning visits, and contraception use. Many of Rossi�s criticisms are based solely on speculation; the rest are resoundingly rejected by the empirical evidence. Perhaps most important, Rossi never explains why two distinctly different evaluation designs have led to similar results. To accept Rossi�s conjectures, one must be willing to believe that this confluence of findings is a mere artifact of design flaws and differences in estimation methods.

We can argue whether New Jersey�s family cap was a good policy. We can discuss whether one can generalize from New Jersey to other states. We cannot, however, avoid the conclusion that women on welfare in New Jersey responded to a family cap in an entirely predictable way by reducing pregnancies and births and (temporarily) increasing abortions.

References

Besharov, D. J.; Germanis, P.; and Rossi, P. H. 1997. Evaluating welfare reform: A guide for scholars and practitioners. College Park: University of Maryland School of Public Affairs.

Jagannathan, R. 1999. Who tells the truth when reporting abortions: A study of women on AFDC. Princeton, NJ: Princeton University, Office of Population Research, 1999.

                                ONGOING CASES                           NEW CASES

SE = Standard error

* p � .05 [??AU: correct here and in table 2?]

Table 2. Treatment-Effect Coefficients: Pre�Post Analysis

SE = Standard error

* p � .05