The Effects of Estimation Error on Measures of Portfolio Credit Risk - Research Paper Example

The effects of estimation error on measures of portfolio credit risk Introduction In the past few years, several new models for the measurement ofportfolio credit risk have been proposed (cf. Crouhy et al., 2000). They have the potential to effect major changes in the ways banks are managed and regulated. So far, however, little is known about the reliability of these models. Nickel et al. (2001) is the only paper which tests the predictive ability of portfolio models on an out-of-sample basis. The major reason for the scarcity of research is the lack of data suitable for back testing (cf. Jackson and Perraudin, 2000). In this paper, I therefore use Monte Carlo simulations to quantify the accuracy of credit risk models. More precisely, I analyse the impact of uncertainty about input parameters on the precision of measures of portfolio risk. I confine the analysis to losses from default, i.e., exclude the risk of credit quality changes, and model default correlations by means of correlated latent variables. The framework builds on CreditMetrics (JP Morgan, 1997), and closely resembles the one used by the Basel Committee on Banking Supervision (2001) to adjust capital requirements for concentration risks. The necessary inputs for assessing default risk are default rates, recovery rates, and default correlations. They are usually derived from historical data, which means that their precision can be inferred using standard statistical methodology. This is the first step of the analysis in this paper. In the second, I determine the accuracy of value at risk (VaR) measures in the presence of noisy input parameters. This is done separately for portfolios which differ in their average credit quality and in diversification across obligors. The aim of such an analysis is threefold. First, the results are useful for defining the role credit risk models should play in credit portfolio management and bank regulation. Second, modelling parameter uncertainty allows to compute risk measures which take estimation error into account. Since the loss distribution is a non-linear function of the input parameters, its estimate can be biased even if the parameter estimates are not. To correct such biases, I employ a Bayesian approach and analyse the predictive distribution, which averages the loss distributions pertaining to different but possibly true parameter values. 1 Finally, the analysis helps to identify inputs with a large marginal benefit of increasing input quality. The analysis shows that estimation error in input parameters leads to considerable noise in estimated portfolio risk. The confidence bounds for risk measures are so wide that losses which are judged to occur with a probability of 0.3% may actually occur with a probability of 1%. Several observations, however, suggest that available credit risk models can be useful for risk management purposes even though their application is plagued with data problems. The magnitude of estimation error is comparable to a setting in which VaR estimates can be based on a long time series of portfolio losses, and it differs little between perfectly diversified portfolios and small portfolios with 50 obligors. In addition, the bias in conventional VaR figures which results from estimation error is modest. The relative importance of the three input factors for the quality of VaR estimates depends on the portfolio structure and the extremeness of the events under analysis. The impact of correlation uncertainty, for instance, is larger for more extreme events and for riskier portfolios. Related papers are Jorion (1996) and Butler and Schachter (1998) who argue that market risk measures should be reported with confidence intervals and show how these can be estimated. The methods are not directly applicable to credit risk measurement because they are based on an analysis of changes in portfolio value. For credit portfolios, historical portfolio returns are typically not sufficient for assessing risk. Butler and Schachter (1998) suggest that precision estimates should be taken into account when credit risk models are evaluated for regulatory purposes. This point is supported by the econometric literature on the evaluation of predictions. As shown by West and McCracken (1998), standard statistical inference about predictive ability is misleading if the prediction is based on estimated parameters. As an illustration, I examine the effects of estimation error on a binomial backtest which is based on the number of cases the VaR was exceeded. If estimation error is neglected in the interpretation of the test, one will too often conclude that the structure of a credit risk model is flawed. Kealhofer et al. (1998) and Nickel et al. (2000) examine the problem of estimating probabilities of rating transitions. Gordy (2000a) gives examples on how changes in input parameters affect the VaR. The current study provides several extensions to these papers. It analyses the three inputs to default risk models in a comprehensive way, it distinguishes between systematic and unsystematic estimation risk, and it shows how to adjust the VaR for estimation risk. Lopez and Saidenberg (2000) suggest cross-sectional resampling techniques to make efficient use of available data and to produce measures of forecast accuracy. The Monte Carlo simulations employed here do not use actual data except for quantifying the quality of parameter estimates. The paper is organised as follows. Section 2 describes the methods used for computing default risk and the assumptions about the magnitude of estimation error. Section 3 presents the simulation results on the accuracy of VaR figures. Section 4' puts the results into perspective and shows how estimation error can bias risk measures and backtests. Section 5 concludes. 2. Methodology 2.1. Modelling portfolio losses I examine how estimates of the annual loss of loan portfolios are affected by noisy input parameters. Each portfolio consists of N loans of size 1=N. A loan is worth 1=N times the recovery rate in case of default and 1=N otherwise. The portfolios are thus homogeneous in terms of the loans face value; apart from the generalizations of Section 3.3, the estimated risk characteristics (default rates, recovery rates, asset correlations) will also be equal across obligors. The analysis is conducted for four portfolios. They contain either 50 or an infinite number of loans to different obligors, whose credit quality is similar to borrowers rated BBB or B by Standard & Poor s (S&P). The portfolios with an infinite number of obligors approximate the case of large bank portfolios, in which unsystematic risk is eliminated. They are called asymptotic portfolios in the following, while the portfolios with 50 obligors will be called small. The latter portfolios are indeed small relative to typical bank loan portfolios. The number is rather chosen to match portfolio structures of collateralized bond obligations or of mutual funds investing in corporate bonds. To assess portfolio credit risk, I employ a methodology similar to CreditMetrics. 2 By defining a default rate and a recovery rate for each obligor, the probability and severity of obligor-specific losses is specified. In a next step, default correlations are modelled based on the asset value model of Merton (1974). A firm is assumed to default if its value falls below a critical level defined by the value of liabilities. Correlations of asset values thus translate into default correlations. More precisely, a firm s logarithmic asset value Xi is modelled through a one-factor model: with Z denoting a common factor, and ei the idiosyncratic risk of obligor i. Depending on the realisation of the asset value Xi and the default probability p, a loan is mapped into one of the two possible states. The model is thus similar to a probit model in which events are driven by latent variables. In CreditMetrics, both Z and ei are assumed to follow a standard normal distribution. In this case, the asset correlation is equal to w2, and default occurs if Xi is below U1'p' with U denoting the cumulative normal distribution function. Gordy(2000a) notes that the assumption of a normally distributed factor is critical, which is why I employ a more general specification. The literature on stock market returns typically documents departures from normality, and proposes discrete mixtures of normal distributions as one alternative specification (e.g. Kon, 1984). I therefore model the distribution of the common factor as a mixture of normal distributions. Specifically, I assume that the common factor Z is drawn from two distributions which have both mean zero but can differ in their variances, where ' takes on the the value 1 with probability ', and 0 with probability (1- '). The variance of Z is normal distribution obtains by setting equal to . In modelling the idiosyncratic component 'i I follow CreditMetrics and assume it to be a normal variate with mean zero and variance. For a given default probability p and a realization of Z, the conditional probability that an obligor defaults is then given by where d, the default threshold, is the solution to With the assumption on z and . Equation-4 can be written as In a perfectly diversified homogeneous portfolio, the proportion of obligors that default will always be equal to the conditional probability . The proportion of defaulted loans is therefore monotonically related to Z. This makes it simple to characterise the unconditional distribution of portfolio losses. The conditional loss is, with r being the recovery rate. Let denote the ' quantile of Z. The VaR at the percentile level ' is given by With a probability of ', portfolio losses are larger than the 'VaR. For a homogeneous asymptotic portfolio, exact VaR figures can thus be obtained analytically. Any recovery rate risk, i.e. stochastic realisations of the recovery rate given that the mean recovery is r, is eliminated in the asymptotic case. The loss distribution of small portfolios can be determined through Monte Carlo simulations. First, random asset scenarios Xi are drawn according to the factor model (1). Depending on the realisation of the asset value Xi and the default probability pi, loans are mapped into one of the two possible states. (Default occurs if Xi is below di.) Applying mean recovery rates to all defaulted loans and summing over all loans yields the overall portfolio value. 4 The generation of random scenarios is repeated sufficiently often (20,000 times in this paper) to obtain a distribution of portfolio value. The 'VaR is calculated as one minus the ' quantile of the simulated distribution of portfolio value. 2.2. Modelling parameter uncertainty To model parameter uncertainty, I assume that risk managers estimate input parameters directly with historical data. Default probabilities are obtained from tabulated default rates, recovery rates from bond prices shortly after default, while the joint distribution of asset values is estimated from observed changes in asset values. Conditional on an assumption about data availability, the quantification of estimation error is straightforward in some cases. In those in which it is not I aim at being conservative, i.e., assume a relatively high level of estimation risk. As it seems impossible to find a specification generally accepted as representative, Section 3.3 will present several generalisations. Estimation risk has a systematic as well as an unsystematic component. The first arises because the estimate of the population average may differ from the true value, the latter is due to the fact that the obligors in a portfolio may differ from the population average. As an example, consider the problem of estimating the default rate of a borrower rated B. If one uses the historical default frequency of B-rated issuers as an estimate, there are two types of error. Due to sampling error or structural changes, the historical average default rate of B-rated issuers may not be equal to the expected default rate of this rating category. Such errors are systematic, they contaminate default rate estimates for all B-rated borrowers. Even if the historical default frequency and the expected average default rate are identical, an individual borrower may have an expected default probability which is higher or lower than the average. This type of error is unsystematic. Its origin may be misclassification of borrowers or loss of information due to the use of broad rating categories which obscure differences between obligors. 2.2.1. Default rates As a source for default rates, I use the 1999 ratings performance report of Standard & Poor s (1999), which covers the years 1981-98. In this 18-year period, the average default rate for BBB-rated issuers was 0.22% and 4.82% for B-rated issuers. These figures are only estimates of the true underlying default probabilities. When estimating the standard error of the mean default rate, i.e. the systematic estimation risk, one should allow for the existence of cyclical patterns in realised default rates. As is known from the econometrics literature, neglecting (positive) autocorrelations leads to an underestimation of standard errors (cf. Greene, 1993). To capture serial correlations, I first estimate the following autoregressive processes for the annual default rates RATE of issuers rated BBB and B, respectively (t-statistics in parentheses): Read More