Research Publication: Evaluating the Performance of Credit Ratings Using Default Data and Quantitative Measures


25 Nov 2020

    HONG KONG, 25 November 2020. Pengyuan International has released one research report, “Evaluating the Performance of Credit Ratings Using Default Data and Quantitative Measures”. 

    A credit rating is a forward-looking analytical opinion on an obligor or obligation’s overall creditworthiness, based on the principles of independence, objectivity and impartiality. It mainly focuses on the obligor’s willingness and capacity to fulfill its financial commitments. As one of the infrastructures in the bond market, credit ratings play an important role in revealing credit risks, assisting market pricing and improving market efficiency. However, a credit rating will be effective only if its performance has been validated and accepted by market participants. To improve the quality and recognition of credit ratings, a set of validation techniques should be established to provide reasonable performance measures for credit ratings.

    Credit spread significance test is a commonly used validation tool of credit ratings performance in China, mainly due to an absence of historical default data in China's bond market. Data unavailability makes it infeasible to directly evaluate the performance of credit ratings using default data. Moreover, defects are observed in both hypothesis and accuracy of the test. In order to validate the credit ratings performance accurately, the market needs a set of reliable validation techniques and measures which can directly reflect the accuracy of credit risk measurement. Given the accumulation of default data in China’s bond market in recent years, the quantitative method using default data to evaluate the performance of credit ratings should be implemented more widely.

    This research introduces a systemic approach to quantitatively assess the performance of credit ratings, which is built upon the evaluation methods wildly applied in other major markets. This approach provides diversified and effective tools for credit rating agencies (CRAs), investors, originators and regulators. Our quantitative validation system consists of three dimensions—discriminatory power, predictive accuracy and rating stability.

    ·      The discriminatory power of a methodology relates to its ability to rank-order the rated entities in accordance with their future status (defaulted or not defaulted) at a predefined time horizon. Three indicators are introduced to demonstrate the discriminatory power, including Area Under the Receiver Operating Characteristic Curve (AUROC), Accuracy Ratio (AR) and Kolmogorov-Smirnov (K-S) Statistic.

    ·      The predictive accuracy measures whether the credit rating methodology can predict the default rate in respect to each rating category accurately, which can be demonstrated by comparing the expected default rates of the credit ratings to the observed results. Statistical hypothesis tests, especially binomial test, Chi-square test and normal test, are the commonly used tools to conduct the validation.

    ·      The rating stability assesses the stability of the credit ratings, the stability of the characteristics of the rated entities or instruments and the distribution of the assigned credit ratings. Transition (migration) matrices, the movement of the credit rating (e.g., frequency and magnitude of the upgrades and downgrades), and Population Stability Index (PSI) are used to demonstrate the rating stability.

    The constructed quantitative approach is then adopted to validate the corporate rating methodologies of seven Chinese CARs.

    In terms of the assessment of the discriminatory power, we measure the ability to rank-order the rated entities in accordance with their future status in the span of one year. The results of AUROC, AR and K-S statistic show that China Bond Rating, CSCI Pengyuan and China Chengxin International Credit Rating (CCXI) perform well.

    In the assessment of predictive accuracy, we observe whether the one-year average default rate with respect to each rating category is consistent with our expectation. The results show that the three credit rating agencies with good prediction accuracy are China Bond Rating, CCXI and CSCI Pengyuan.

    In the assessment of rating stability, we construct transition matrices and calculate the frequency and magnitude of the upgrades and downgrades. The results show that the three credit rating agencies with high rating stability are CSCI Pengyuan, CCXI and Golden Credit Rating.

    Note: This press release is a translation of the original in Chinese. If there is any inconsistency or ambiguity between these two versions, the Chinese original will prevail.

     
    ANALYSTS CONTACT

    Primary Analyst                                 

    Ke Chen, PhD                                     

    +852 3615 8316                                  

    ke.chen@pyrating.com  

    Secondary Analyst

    Shiyu Wang, PhD

    +86 755 8287 1237

    shiyu.wang@pyrating.com


    MEDIA ENQUIRIES                           

    Charley Lui                                          

    +852 3615 8296                                  

    charley.lui@pyrating.com


    RATING SERVICES ENQUIRIES

    Gloria Song

    +852 3615 8324

    gloria.song@pyrating.com



    Date of Relevant Research Committee:
    23 November 2020

    Additional information is available on www.pyrating.com


    DISCLAIMER

    Pengyuan Credit Rating (Hong Kong) Company Ltd (“Pengyuan International”, “Pengyuan”, “the Company”) prepares various credit research and credit research related commentary (collectively “research”) in compliance with the established internal process. The Company reserves the right to amend, change, remove, publish any information on its website without prior notice and at its sole discretion.

    The research is subject to disclaimers and certain limitations. RESEARCH AND CREDIT RATINGS ARE NOT FINANCIAL OR INVESTMENT ADVICE AND MUST NOT BE CONSIDERED AS A RECOMMENDATION TO BUY, SELL OR HOLD ANY SECURITIES AND DO NOT ADDRESS/REFLECT MARKET VALUE OF ANY SECURITIES. USERS OF RESEARCH AND CREDIT RATINGS ARE EXPECTED TO BE TRAINED FOR INDEPENDENT ASSESSMENT OF INVESTMENT AND BUSINESS DECISIONS.

    This research is based solely on the public data and information available to the authors at the time of publication of this research. For the purpose of this research, the Company obtains sufficient quality factual information from public sources believed by the Company to be reliable and accurate. The Company does not perform an audit and undertakes no duty of due diligence or third-party verification of any information it uses in the research. The Company is not responsible for any omissions, errors or inconsistencies of the public information used in the research.

    NO WARRANTY, EXPRESS OR IMPLIED, AS TO THE ACCURACY, TIMELINESS OR COMPLETENESS OF ANY INFORMATION GIVEN OR MADE BY THE COMPANY IN ANY FORM OR MANNER. In no event shall the Company, its directors, shareholders, employees, representatives be liable to any party for any damages, expenses, fees, or losses in connection with any use of the information published by the Company.

    This research focuses on observing trends from the credit markets. This research has not been made available to any issuer prior its distribution to the public. The Company does not receive compensation for its research.

    The Company reserves the right to disseminate its research through its website, the Company’s social media pages and authorised third parties. No content published by the Company may be modified, reproduced, transferred, distributed or reverse engineered in any form by any means without the prior written consent of the Company.

    The Company’s research is not indented for distribution to, or use by, any person in a jurisdiction where such usage would infringe the law. If in doubt, please consult the relevant regulatory body or professional advisor to ensure compliance with applicable laws and regulations.

    Copyright © 2020 by Pengyuan Credit Rating (Hong Kong) Company Ltd. All rights reserved.