Relationship education: Scott Stanley responds

Stanley, one of the developers of the popular PREP program, offers a more optimistic view of the Building Strong Families study’s Oklahoma City data.                                                                                                                                                                                                                                                                                                            

People in Bad MergentheimI posted last week about the disappointing 3-year results of the Building Strong Families study, a major study of relationship education involving more than 5,000 families at nine sites around the country.

Scott Stanley, one of the developers of the PREP relationship education course and a (quite deservedly) well-respected name in relationship and marriage research, wrote me in response, offering a more optimistic look at the site-specific data from the study’s Oklahoma City location. He kindly granted permission for me to post his letter, in its entirety, here. The emphasis added is mine and I’ve plugged in some links, but otherwise left his writing unchanged:

Dear Ben,

Just read your post on BSF in OK. You raise important points like those raised by others.

I wanted to argue the other side a bit, particularly about the Oklahoma finding at 36 months in the BSF study. I won’t take the space to do it fully in this comment, but I would refer your readers to a blog post I did on the 36 month findings in light of a similar critique by Andrew Cherlin in the last week of 2012. The piece (my blog piece) links to Cherlin’s critique and several other interesting pieces that your readers may appreciate. Andrew Cherlin has read my piece on his critique, and I believe he has changed his mind about the importance of the stability finding in OK at 3 years out. (see links here and below)

The key point is simply this: As I explain in my piece, what you refer to a small difference in family stability is actually a 20% increase in family stability for these most vulnerable families 3 years later (49% in the program group vs. 41% in the control group, of families where the children born lived continuously with both parents over the three year follow-up period, which is a 20% improvement of the intervention group over the control group).

What you, as so many others, are likely unaware of is that the evidence for most government social programs (or other programs for that matter) is dismal (I also make this point in my piece). I give some suggestions for how people can look such things up in my piece as well.

This OK BSF finding is actually rather extraordinary, and it is unlike findings in most all evaluations of government programs, especially evaluations of initial efforts–it is significant, relatively large (this is not a small difference on such an outcome in a policy study), and it is on what is easily argued the single most important type indicator relevant to this study context and family policy. It is also not at all unusual to get earlier impacts on things that can change more immediately that may (as in this case) set up longer term, more important outcomes.

You may not be impressed with that finding, but there are those who know government evaluations quite well who find it remarkable. However, it is true that among people who have been critics of these fledgling efforts, the desire has clearly mostly been to focus on the pooled results across sites where the story is not positive.

In addition to the points I make in my blog in commenting on Andrew Cherlin’s piece, I would add these points here about the finding you dismiss: This impact on a hard indicator of an outcome (family stability) of prime interest three years after the intervention (1) happened in the only program to deliver a substantial dose of intervention to a high percentage of couples in the intervention group, (2) happened in the only program to have many significant impacts on relationship quality variables at the intermediate stage of evaluation, and (3) happened in the only site that also had a stability findings at the prior time point.

The link to my comment on Andrew Cherlin’s piece is here:
http://slidingvsdeciding.blogspot.com/2013/01/family-stability-and-relationship.html
[There is also a full disclosure comment at the end of that blog piece.]

The link to a resent presentation where you can see Andrew Cherlin’s comment on the OK finding is here:
http://www.brookings.edu/events/2013/03/20-knot-yet-marriage#ref-id=20130320_CCF_fullevent_pt1

# # #

My thanks to Dr. Stanley for his thoughtful email and the permission to repost it here.

Your questions and comments are similarly welcomed. You can post them in the comments below, by email to ben[at]bencaldwell[dot]com, or on my Twitter feed.

Major national study: Relationship education doesn’t help low-income parents

In a study of 5,000 couples, a structured program of relationship enhancement not only failed to help poor, unmarried parents. It appeared to make a few things worse.

People in Bad MergentheimAlmost three years ago, I posted on the Building Strong Families (BSF) project, a first-of-its-kind national study of relationship education (also sometimes called premarital education, marriage education, relationship enhancement, or simply RE). The study involved more than 5,000 couples at nine sites around the US, all of whom were low-income, unmarried parents. In theory, such couples could especially benefit from effective programs to improve couple and family functioning. Couples in BSF were offered relationship skills groups, individual support from assigned “family coordinators,” and referrals as needed to specialized services such as mental health counseling, housing assistance, legal services, and child care.

My initial post had focused on 15-month follow-up data, which was not promising, but gave reason to hold out for longer-term data. Late last year, the BSF program released its final report, which included a 3-year follow-up. It was a spectacular failure.

The executive summary was appropriately blunt (emphasis added):

After three years, BSF had no effect on the quality of couples’ relationships and did not make couples more likely to stay together or get married.

The first piece there, that the program did not impact the quality of couples’ relationships, should startle those therapists who use or recommend relationship enhancement programs. Even in the absence of long-term outcome data, these programs have often highlighted the studies that appear to demonstrate that they offer at least a short-term improvement in relationship satisfaction. (A good current summary of such studies can be found in this Journal of Marital and Family Therapy article, subscription or payment required.) In the BSF study, any such gains had evaporated even before the 15-month follow-up, unlikely to ever return. Couples who had gone through the program were no better than control-group couples on measures of relationship happiness, supportiveness and affection (treated as one variable), faithfulness, or conflict management.

The executive summary continued (again, emphasis mine):

BSF had no effect on couples’ co-parenting relationship; it had small negative effects on some aspects of father involvement. […] BSF had no effect on the family stability or economic well-being of children; however, the program led to modest reductions in children’s behavior problems.

Fathers who went through the BSF program were spending significantly less time with their children at the three-year follow-up than control-group fathers. At the same time, children of couples who participated in the program did experience significantly fewer behavior problems; that was the only variable, of 20 studied, to show a desired outcome from the program. Neither the executive summary nor the full BSF report clarifies whether these two findings are more directly connected. It is at least plausible that these families were making rational decisions about the level of father involvement, if such involvement was likely to be detrimental to the child. It is also certainly plausible that the two findings are unrelated. I hope the study makes its full dataset available to other researchers for additional analysis.

Perhaps most disappointing in the three-year dataset is the failure of the study’s Oklahoma site to maintain its strong standing over time. In Oklahoma City, the BSF program used a well-established curriculum based largely on the popular PREP program, engaged many more families in participating than other sites, and showed much better outcomes at 15 months. However, most of the positive impacts observed there at the 15-month follow-up had faded away by 36 months. The only positive impact unique to Oklahoma City that remained was a small improvement in family stability, an impact that had not been present in the earlier data. But this positive result was offset by another program site in Florida, where the 36-month data newly demonstrated negative impacts on relationship status and quality, father involvement, and family stability.

I suggested in my earlier post that perhaps the BSF program was not failing, but rather it was not drawing enough participation:

It would be more accurate to say that couples didn’t finish the programs than it would be to say that the programs didn’t work; with the exception of the project’s Oklahoma site (which performed much better than other sites in a variety of ways), only 9% of couples completed at least 80% of the relationship enhancement curriculum offered to them. That’s a big problem.

However, the program’s own executive summary suggests that the participation issue should not be reason to dismiss their overall conclusion that BSF largely failed (emphasis added):

Across the eight programs, only 55 percent of couples assigned to the treatment group attended a group relationship skills session. However, analysis of BSF’s impacts among couples who did attend found little evidence of effects on relationship outcomes. Thus, it does not appear that low participation rates explain BSF’s limited success in improving couples’ relationships.

Unfortunately, the field of relationship education is now left with not one but two major problems. 1, How can we design a program to teach family skills such that low-income couples with children would actually be able to make use of it? If the low participation rate was because of difficulty arranging transportation or child care, or other life stressors, programs must find ways to address those stressors first. (See also Maslow’s hierarchy of needs.) 2, Even if we could address all of the circumstances that keep couples from participating in these programs, what could we offer them that would actually make a difference in the long term? Three years and many millions of dollars after the launch of BSF — the average cost of treatment was about $11,000 per couple — we are no closer to an answer to that question.

# # #

The next wave of data on relationship education will come from a separate five-year study, the Parents and Children Together (PACT) program, which will examine programs that provide both relationship skills and employment services. The BSF report suggests that PACT may ultimately be more effective, given some research findings that fathers are more involved with their children when they see themselves as economically successful.

Your questions, comments, and predictions about future studies are welcomed. You can post them in the comments below, by email to ben[at]bencaldwell[dot]com, or on my Twitter feed.