The American Academy of Actuaries has released its “Model Governance Checklist” (dated August 2016). This checklist is the culmination of work that began one year ago with the formation of the Principle-Based Reserves Checklist Subgroup, which I am honored to chair. Our subgroup was charged with the creation of a model governance checklist in response to the need for good model governance as addressed in PBR regulation. However, while motivated by PBR, this checklist fulfills a more universal need as a resource applicable across most actuarial modeling endeavors. (Note that a working draft of this checklist had been previously shared and discussed at both of the American Academy of Actuaries PBR Boot Camps held earlier in 2016.)
When deciding upon the scope of the checklist, we cast a broad net across the actuarial modeling risk landscape. For example, risks associated with results consolidation and report generation have been addressed. Consequently, this checklist may elucidate areas of risk within an organization that may be overlooked in their existing model governance practices. And while recognizing that there will always be room for improvement, knowing where existing model governance gaps exist is often the necessary first step in risk mitigation.
As discussed in the checklist’s preface, it contains questions that are intended to foster awareness of potential model governance concerns. While these questions are not specifically directed towards any actuary or group, they provide practicing actuaries with food for thought when evaluating their model governance processes. Note also that while this checklist may be a beneficial tool, it does not constitute a list of requirements.
The Model Governance Checklist questions are grouped into the following ten categories which I further discuss below. (Note that there is no “Documentation” section, as documentation issues are addressed within each category.)
Questions under this category address an organization’s formal, documented model governance policies. Issues pertaining to organizational structure, including specific roles and responsibilities can be found in this section.
The modeling process section includes questions pertaining to model management processes and procedures. High-level model management “documentation” protocols are also included.
Questions in this section surround the development, management and documentation of modeling assumptions. Also touched upon in this section is the general use of experience studies and corresponding credibility.
After modeling assumptions have been determined and approved, they still need to accurately make their way into actuarial projection models. Questions in this section cover the input of asset and liability model assumptions, tables and model settings. This would
include: seriatim policy inforce files, rate tables (i.e., premiums, mortality, interest, asset default, etc.), model point policy mappings, etc.
The adage that “If it is not locked down, it is not under control.” holds true for the many modeling components that comprise an organization’s modeling infrastructure. Input data, models, modeling systems, modeling output repositories and reports should each have their respective access controls evaluated. The half-dozen questions which constitute this smaller category address these concerns but only at a high level. However, to expound deeper into these issues would invariably depend upon the specific modeling environments encountered within an organization.
Model changes and actuarial system modifications require special consideration in an organization’s model governance plan. The value of a robust formal system change control process becomes clearer upon the realization that model input validation and output testing alone could miss many system coding and modeling errors.
The nature and extent of actuarial system governance within an organization will depend upon the type of actuarial projection system in use. For example, open code systems may require specific code management processes, whereas closed systems may need to rely more on software version testing. The breadth of questions in this category cover these issues in more specific detail and provide plenty of material in which to consider incorporating into a model governance plan.
When evaluating the suitability of a potential model, it is important that the selected model and its underpinning theories/concepts align well with the desired purpose. Questions in this section touch upon this and other issues related to transparency of model inputs and versioning of modeling components.
Consolidation of Results
After actuarial models have completed their executions, their results are typically consolidated and stored in repositories for further analysis and reporting. The associated processes involved in these consolidations may be inadvertently overlooked or inadequately addressed in many model governance plans. Questions in this section cover various consolidation concerns, including issues surrounding late adjustments.
Controlling how modeled data makes its way into reports should be within the purview of model governance. In fact, reporting should be one of the main focal points of an organization’s approach to model governance. Similar to assuring that actuarial models use correct input, actuaries also take part in assuring that modeled results are correctly interpreted and appropriately integrated into the many downstream reports.
This section contains universal questions that are not restricted to any specific reporting or accounting basis. Questions cover issues such as interpretation of modeling results, report clarity and transparency.
Broadly traversing the spectrum of modeling activities, questions in this section cover the validation of: assumptions, model inputs, systems and models, results consolidation and reporting. Due to the inherent technical subject matter, many questions may require the reader to be familiar with common systems validation and testing methods.
Download Your Copy
The American Academy of Actuaries has recently announced the release of the Model Governance Checklist as a tool for practicing actuaries. But, if you missed this Academy Alert, the Model Governance Checklist can be directly downloaded from the American Academy of Actuaries website.