NTSA and the Swimming Pool Dilemma

Opinion piece: NTSA Confuses Correlation and Causation


Recently, the National Tax-Deferred Savings Association (NTSA, an entity that exists to lobby on behalf of vendors and salespeople to government 403(b) & 457(b) plans) released a report titled “Improving Retirement Savings for America’s Public Educators” in which it aims to provide evidence that increasing or maintaining (many) choices for public educators increases participation. After reading the report, I found no evidence to support the conclusions reached. Further, I believe those conclusions were reached through clever misrepresentations of the data to justify a primary goal of the NTSA, which is to “Prevent elimination or limitation of retirement plan options for public employees.”

The one thing you must know about government 403(b) plans to understand this report is, unlike 401(k) plans, there are usually multiple vendors. Public school employers rarely offer less than five 403(b) vendors and sometimes over sixty. Each vendor might offer numerous products sold by a long list of (typically) insurance agents. It can be very confusing as there is not a single place to go to get unbiased information.

Insurance companies and their agents rely on the multiple vendor system to maintain their ability to sell retail products to public school employees. If a school district were to consolidate vendors and use the purchasing power of their employees to gain better pricing (think of this like a Costco using it’s buying power to buy wholesale) the sales agents and their product providers would either be out of business or would need to find a new market in which to sell their expensive products. There is a significant interest in maintaining the status quo of multiple vendors in public schools as the insurance companies and their agents are making billions of dollars from it. In recent years, a trend toward fewer, higher quality and lower cost vendors have threatened these insurance companies and this report is an effort to sway the public into believing that our nation’s teachers are better off buying at retail, rather than wholesale prices.

As I started reading the report, I found myself agreeing with some of it. The first paragraph of the Executive Summary was well written, and I agree that “…the importance of personal savings in 403(b) plans is more critical to their retirement security than ever.” The report then goes on to promote the association's members as the solution, but disguised as “research.” This access to data could have potentially been used to improve these plans; instead, it's used to justify the status quo.
The main conclusion was that “The data shows a decrease in the participation rates for 403(b) plans when the number of choices are reduced.” If true, this report would be evidence that Nobel winning research on behavioral finance is wrong. However, it’s not true. The data doesn’t confirm the claims, and it should concern everyone involved that it’s represented as something it’s not.

Have you ever heard the phrase “don’t confuse correlation with causation”? It’s a common phrase in the world of academia (and statistics) that means because two things happen to be correlated, one didn't necessarily cause the other.
Tyler Vigen maintains a website that demonstrates the absurdity that arises when we connect things which aren’t related. Let me give you a few examples to make the concept more relatable:

Did you know that between 1999 and 2009 the number of people who drowned by falling into a pool closely correlated with films in which Nicolas Cage appeared? The data is irrefutable, both went up or down together, but you can rest assured that the next time Cage appears in a film, there will not be an increased risk of falling into a pool and drowning. Sometimes two things that are entirely unrelated look like they are related. Statisticians have many ways to attempt to tease out what is, in fact, a true correlation and what is just a coincidence.

Did you know that every year people die by becoming tangled in their bedsheets? I had no idea. However, did you also know that per capita cheese consumption highly correlates with those deaths? If only we could get people to eat less cheese, we could save so many people from needless tangled bedsheet deaths.

It’s ALWAYS better to have a younger Miss America. No, I'm not misogynistic or ageist, I care about people murdered by steam, hot vapors and hot objects. The data clearly shows that the older Miss America is, the more murders by hot objects there are, the information seems to indicate we should stay away from anyone over 20 years of age.

I could go on and on. Tyler Vigen ended up writing a book showing all sorts of crazy things that correlate. My main point, just because two things relate, even if it might make sense that they are correlated, it doesn’t mean one is caused by the other, the work to prove causation is still ahead of you.

My problem with the NTSA report is that it doesn’t do the work to prove any of the claims made from the data. They merely cite correlations; they don’t prove causation (or even attempt it).

The central claim of the report (on page 4) is that “The research revealed that the number one factor driving participation and savings rates in school districts is participant choice.” and that,

“Simply stated, the data reveals a positive and significant correlation between the number of choices/advisors and participation.” The author might be correct that "the data reveals a positive and significant correlation,” but we can’t know that because so far the NTSA has refused to release the data. Notice that the claim is made real by the evidence of the data correlation? In other words, the NTSA is saying that because there is a positive correlation, therefore higher numbers of vendors CAUSE higher participation. However, didn’t we learn that correlation is NOT causation? Just because two variables are linked, it doesn’t mean one is caused by the other.

The NTSA has committed an academic foul here; they aren’t lying, the data DOES correlate. However, they spend no time attempting to find out if the correlation is just a coincidence or if it's caused by other factors (such as the age of the population, employer pay scales, the simplicity of enrollment process, employer-provided education, rural vs. urban area, teacher pay, the strength of the defined benefit plan, access to Social Security, etc.). All of the work is still ahead of the NTSA, they’ve not proved, as the report states, that more vendors lead to higher participation (or that reducing vendors causes participation to fall). They’ve only found a correlation. Interestingly, the correlation they found is the one they wanted to find, curious.

I’ve spent hours tearing apart this report claim by claim and wrote the framework for a six-page paper, but such a long piece would be boring. Why be boring when a short article on correlation and causation completely discredits the report.

The bottom line here is that this report is not worth reading or citing, it is industry propaganda to maintain the status quo. There are solutions to increasing participation, adding more vendors is not one of them.

It shouldn’t surprise you that the industry put out “research” that shows how much you need them, but this report should be taken with a grain of salt. If you want to understand better how 403(b) plans can be improved, I suggest the website run by my esteemed colleague, Dr. Dan Otter, www.403bwise.com. Dan’s site sometimes correlates learning with wittiness, but I’ve not yet been able to prove causation!

Author's note: I've been tough on the NTSA (formerly NTSAA) over the years and for good reason, they've consistently worked on behalf of salespeople and mostly low-quality vendors under the guise of representing public school educators. In recent years they've made a few changes that I feel are positive and there are some people involved in leadership whom I respect (even if I disagree with them). My hope with this piece is to convince the NTSA to let the data lead them, not the other way around.

Are you 403(b) Wise? 403bwise.org is the place on the internet to learn, advocate and build community.