Transparent stress tests?

This month, the Committee on Capital Market Regulation (CCMR) published a paper criticizing the procedures the Federal Reserve uses in conducting its stress tests. The claim is that, in its annual Comprehensive Capital Analysis and Review (CCAR), the Fed is violating the Administrative Procedures Act of 1946 (APA). The CCMR’s proposed solution is more transparency. As big fans of both stress tests and transparency in general, and of the CCAR in particular, we find this legal challenge very troubling.

We believe that making the stress tests more transparent in the ways that the CCMR suggests would make them much less effective. This would do serious damage to financial stability policy and (ultimately) increase the likelihood of another crisis.

Modern stress testing builds on the U.S. experience during the crisis. In late 2008, the solvency of the largest American intermediaries was in doubt. That uncertainty made their own managers cautious about taking risk and it made potential creditors, counterparties, and customers wary of doing business with them. Those doubts contributed to the extreme fragility in many financial markets, leading to a virtual collapse of interbank lending. A critical part of the remedy was a special disclosure procedure in which the Federal Reserve conducted an extraordinary set of “stress tests” of the 19 largest banks and, in May 2009, published the results.

The 2009 tests evaluated on a common basis the banks’ prospective capital needs in light of the deep recession that was under way. While observers questioned whether the tests were stringent enough—the “stress” scenario quickly turned into the central forecast—the results were sufficient to reassure the government, market participants, and the banks themselves that most of the institutions were in fact solvent. Partly as a consequence, conditions in financial markets rapidly improved. And, armed with the stress-test evidence of their wellbeing, most large banks were able to attract new private capital for the first time since the Lehman failure the previous September.

Stress tests are, in our view, one of the most powerful prudential tools available for safeguarding the resilience of the financial system. They take seriously the fact that when a large common shock hits, there is no one to sell assets to or raise capital from. Ensuring that each systemic intermediary can withstand significant stress raises the likelihood that the system can survive. And, importantly, by adjusting the scenarios, prudential authorities can maintain a chosen level of resilience. At least in principle, stress tests can both account for changes in the distribution of the shocks that can hit the system and contain the propagation mechanisms that amplify the impact of the shocks on the economy.

Now these tests have two essential parts: scenarios and models. The scenarios are the paths of economic and financial variables going forward. These typically include assumptions about real economic growth, unemployment, inflation, interest rates, risk spreads, key asset prices, exchange rates, and the like; both domestic and foreign. The models allow the supervisors to translate these macroeconomic assumptions into prices of other assets, like mortgages and loans. And finally, the model-driven shock is applied to the balance sheet (and off-balance-sheet exposures) of a bank as it stood on a particular day.

The CCMR’s legal argument is that the scenarios and the models are in effect rules that have future effect, as distinct from “adjudications” that generally resolve disputes on a case-by-case basis. Under the APA, rules promulgated by federal agencies are subject to a notice-and-comment procedure. The implication is that the law requires the Fed to publish both the scenarios and the models for public comment before they are put into practice.

Our view is that this type of transparency would render the stress tests nearly useless in preventing or mitigating a financial crisis.

To be clear, when it comes to many aspects of policy, we are strong advocates of transparency. Inflation targets are an excellent example. Having a clear, publicly stated objective for inflation promotes accountability and coordinates expectations well into the future. When everyone expects monetary policymakers to react in a way that will bring inflation back to a known target, it is stabilizing. Transparency helps to achieve the goal because it raises the cost of reneging on the commitment to price stability, making it time consistent (see, for example, here).

In the case of prudential policy, however, some forms of transparency do more harm than good. In an earlier post, we discussed the pitfalls of disclosing central bank discount lending details. To repeat the message: painful experience teaches us that transparency can feed a crisis. At the start of 1932, following thousands of bank failures, the new Reconstruction Finance Corporation (RFC) began lending to distressed banks, and failures temporarily slowed. But, the loans were controversial, so a July 1932 law required reporting of the borrowers by name—information that Congress quickly released. Shining a light on borrowers diminished banks’ willingness to use the discount window to avoid being seen viewed as fragile. Once this stigma undermined the ad hoc lender of last resort, an even bigger wave of bank runs and failures followed.

Now, in the case of stress tests, the challenge is for officials to obtain an accurate reading of the resilience of a bank’s balance sheet. Stress tests have three essential elements: timing, scenarios, and models. On the first, the Fed is already transparent, so banks know in advance the date of the balance sheet that will be tested. On the second, as it currently stands, the Fed publishes the scenarios, but only after the bank’s positions are fixed. And on the third, there is no disclosure.

The natural concern is gaming. That is, we worry that the banks will structure their balance sheet on the reporting day in a manner that minimizes their capital needs. They will do this through a combination of forecasting the stress scenarios and reverse engineering the models.

In their report, the CCMR admits the possibility of banks dressing up their balance sheets for the purposes of improving their stress-test outcomes. But the Committee puts the burden on the Fed to show that banks are gaming.

We are surprised by this, as we assume that the pressure on banks to maximize their return on equity creates irresistible incentives to game. This is presumably why they now employ so many people to help them minimize their capital needs while still passing the stress tests.

Again, painful experience suggests that such gaming is the norm. Consider, for example, the failure of the pre-crisis stress-testing regime for the government-sponsored enterprises (GSEs). Following the Federal Housing Enterprise Safety and Soundness Act of 1992, the U.S. Office of Federal Housing Enterprise Oversight (OFHEO) initiated stress testing of the GSEs in 2002. But OFHEO’s tests never exposed the massive capital shortfall of the GSEs. Why? Here’s what Frame, Gerardi and Willen suggest:

“One potential reason for the static approach was that OFHEO was required by law to fully disclose the stress test model and went so far as to publish all stress scenarios, empirical specifications, and parameter estimates in the Federal Register.”

They conclude:

“We believe OFHEO faced challenges emanating from statutory model disclosure requirements, as well as statutory limits to the specification of the house price and interest rate stress scenarios. These constraints, coupled with the political power of Fannie Mae and Freddie Mac, made it extremely difficult to introduce meaningful changes to the risk-based capital rule.”

In fact, with regard to stress test transparency, we would prefer that the Fed goes in the opposite direction from the CCMR recommendations, randomizing the date of the stress test to reduce the scope for balance sheet window dressing. Annual stress tests have little value unless banks’ risk exposures on the day of the test are representative of the risks they take throughout the year.

Financial regulation has become an arms race. Banks hire phalanxes of lawyers, risk managers, and financial engineers to help them figure out how they can maximizes their profits at the same time that they meet the letter of the regulations. On the rule-making level, banks employ lobbyists on a permanent basis, and are constantly seeking to influence the system in other ways. On the operational level, they aim to optimize subject to the constraints set by regulation and the market. That is why, in practice, every set of regulatory rules breeds innovation that gradually undermines their effectiveness. In the parlance of national defense, the bankers have substantial intelligence operations designed to influence and exploit the official sector’s rules and regulations, as well as try to beat their competitors.

Prudential authorities have a difficult time competing in this arms race. They have fewer resources, and more legal restrictions. The stress tests are their most important and most flexible tool; and, at least today, their only hope of competing effectively in this arms race. That effectiveness is dependent on the confidentiality of the scenarios and the models. Making these public would inevitably lead to gaming, leaving the financial system vulnerable.

Finally, we think the current level of stress test transparency is actually quite substantial and effective. Providing the scenarios after the fact, as the Fed does, is analogous to what we routinely do as professors: namely, publish class exams from past years to help students study, organize their thoughts, and prepare for the time constraint of an examination. It wouldn’t do, however, if we were to disclose this year’s test in advance in order to allow for open debate about its fairness and balance. If the test is to have any means of distinguishing among students, these concerns must be assessed after the exam, not before. And the public record of past exams makes it feasible to do so.

So, not being lawyers, we hope that the legal arguments made by the Committee on Capital Market Regulation regarding the Fed’s stress tests are wrong. If they are not, then we can only hope that Congress acts promptly to protect the stress tests from an application of the Administrative Procedures Act that would eviscerate them.