Beyond Sum Score Tests

The instrument a clinician employs to evaluate cognitive functioning in elderly patients has far reaching implications for diagnosis, treatment and recommendations for care. However, there is no consensus as to the role that geriatricians, gerontologists and neurologists should play in assessing cognition. Is screening with sum score tests adequate? Or do patients and their family deserve more from clinicians they view as specialists in and authorities on brain-behavior relationships? Patients with neurocognitive disorders, their family members and care providers are ill-served by cognitive assessment that is limited to sum score screening tests like the MoCA, MMSE and SLUMS. However, the first two of these (in hundreds of translations) remain the most widely used cognitive screening tools in the world. Despite often waiting for months to have a loved one evaluated by a physician, family members and concerned others are frequently kept in the dark for months, years or forever regarding the actual nature and extent of cognitive changes. This confusion persists unless or until testing is done with tools that employ adequately-extended measures or “metrics” in individual cognitive domains. Internists and general practitioners often refer patients with cognitive impairments to neurologists for “formal testing of cognition”. Yet the tests that the vast majority of neurologists administer are screening tools that fail to identify a pattern of impairments in individual domains. These screening tools do not allow physicians to alert family members as to which Instrumental Activities of Daily Living (IADLs) are at greatest risk. Several factors are likely to contribute to the maintenance of physicians’ reliance on abbreviated, sum-score tests.
a) A worldwide shortage of neurologists, geriatricians and gerontologists translates into limited time for cognitive testing.
b) Lack of training in cognitive testing during medical school and residency leads medical professional to view neuropsychologists as the traditional authorities on cognitive assessment. This allows neurologists and geriatricians to abdicate responsibility for cognitive testing beyond rapid sum-score screening. It also absolves them of responsibility for determining the nature and severity of cognitive impairments.
c) Physicians find it understandably unpleasant to inform patients that their neurocognitive disorder is likely progressive and they have no treatment for it. Lacking effective treatment to reverse cognitive decline associated with neurodegenerative disorders such as Alzheimer’s Disease, physicians may see little value in identifying specific cognitive impairments. Brief and insensitive tests seem to be all that is needed since nothing can be done to reverse cognitive deficits. The insensitivity of the MMSE in particular allows college-educated and professional individuals to deteriorate significantly before their score falls below 24/30 and suggests mild dementia. Even though many patients and their family members know that the patient’s functioning is far from his or her previous level, all too often inappropriate reassurance is given that their MoCA or MMSE score falls well within the average or normal range.
d) A reluctance to open Pandora’s box: Identification of impairments in one or more specific cognitive domains has the potential to make clinicians legally and/or ethically responsible for exploring the impact of these impairments on functioning in a range of Independent (aka “executive’) Activities of Daily Living (IADLs) such as driving, managing investments or modifying estate plans. A separate issue concerning cognitive testing performed by physicians is that cognitive screening tests are frequently given without a systematic review of shifting state factors that have the ability to impact a patient’s cooperation and performance. These include hearing and/ or vision impairment, physical pain, sleep deprivation, CNSactive medications, psychiatric conditions (anxiety, depression, psychosis). Additionally, trait factors such as ADHD, dyslexia, a history of auditory or visual learning difficulties are important to consider before interpreting scores on screening tests.

The suggestion we made in patient-specific cognitive profiles in the detection of dementia subtypes: A proposal (Alzheimers & Dementia. vol. 19, issue 10, pp. 4743-52) was that neurologists and other physicians consider using non-physician clinicians to assess cognition. Nurses, speech therapists and psychologists could administer brief domain-specific tests on a routine basis whenever cognitive impairment is a presenting symptom. These non-medical clinicians could also systematically explore instrumental activities of daily living using a variety of ADL measures currently available.

Three domain distinct, patient-specific cognitive tests are presented below that can be administered by non-MD’s. Such an approach is consistent with and parallels with the use of nonphysicians to measure height, weight and blood pressure, as well as review medications and explore other limited healthrelated areas prior to a patient meeting with a physician. BEFORE presenting 3 domain-specific cognitive screening tools that take less than 30 minutes to give, it is appropriate to provide context by mentioning several lengthier (30-45 minutes) tests used by neuropsychologists. The best known and most widely used of these abbreviated neuropsychological tests is the Alzheimer’s Disease Assessment Scale-Cognitive Subscale (ADAS-Cog). First developed in the 1984 [1], the ADAS-Cog has been used extensively in dementia research and is considered the gold standard in the evaluation of drugs designed to slow the progression of Alzheimer’s disease. The full ADAS consists of two components, The ADAS-Cog (11 cognitive tasks) and the ADAS-Noncom (10 tasks that address mood and behavior changes). Over the past three decades many modifications to the ADAS-Cog have been made to heighten the test’s ability to detect Mild Cognitive Impairment (MCI). There is no current consensus as to which of the 31 versions of the test is the best to use [2].

Several other tests, each significantly shorter than standard neuropsychological batteries, are popular among neuropsychologists. However, the time required (>30 minutes) to give any of these four tests makes them impractical for regular use by busy physicians. These tests include the Mattis Dementia Rating Scale (DRS-2) [3], the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [4] and the Neuropsychological Assessment Battery (NAB) Screening Module [5]. The Cambridge Neuropsychological Automated Test Battery (CANTAB) is a computer-based cognitive screening tool that is administered from a tablet [6]. Its 25 tests load onto specific cognitive domains and the examiner can customize administration. Depending on how many of its 25 tests are given, CANTAB can take up to 60 minutes to administer.

ACE-III

The Addenbrooke’s Cognitive Examination (ACE, ACE-R and ACE-III) [7,8] represents a hybrid test with features of both sum score and domain-specific testing. The ACE-III takes from 15 to 20 minutes to administer and score. Initially designed in the late 1990’s to address weaknesses of the MMSE, two successive versions of the test were created. The ACE and the ACE-R incorporated portions of the MMSE but the ACE-III removed the MMSE elements when the MMSE ceased being open access in 2001. The ACE-III utilizes two scoring approaches: A five-score profile and a 100-point sum score scale. Cut-off scores of two standard deviations below the mean in each domain are based on a study of 63 normal control subjects in in their 60’s. The ACE-III employs 24 brief texts to examine five cognitive domains with extended graded measures: attention/ orientation (18 points), memory (26 points), verbal fluency (14 points), language (26 points), visuospatial skills (16 points). This approach permits the comparison of performance in differing domains. The ACE-III is particularly helpful in distinguishing Alzheimer dementia (DAT) from frontotemporal dementia (FTD) and in detecting conversion from MCI to dementia. On the sum score scale suggested cut-scores are 88/89 for MCI and 75/76 for dementia.

The creation of an iPad-based version, ACE mobile, combined with a training video for the use of ACE mobile, was reported by Newman et al. [9] to increase the accuracy of measurement. Caution is warranted in use of the sum score scale on the ACE-III since impairments in basic attention or aphasic language difficulties may impact other domains and pull down the sum score. Fluency for letters and animals serves as frontal executive tests but the ACEIII does not examine reasoning or judgment. Shifting contextual factors such as pain, sleep disturbance and medications are not assessed [10,11].

CAMCOG

Designed in 1986 as the cognitive assessment portion of the Cambridge Assessment of Mental Disorders of the Elderly (CAMDEX) [12], the CAMCOG is a 107-point test created to address the limitations of the MMSE. Particular concerns were the MMSE’s insensitivity to early cognitive decline in individuals with higher levels of education or intelligence and its failure to test executive function. The CAMCOG examines eight domains: orientation, comprehension, expression, memory, attention and calculations, praxis, abstract thinking and perception. Like the ACE-III, the CAMCOG is a hybrid test that employs graded numerical measures in each domain but also uses a sum score with cut-points. Huppert et al. [13] validated the sensitivity and specificity of the CAMCOG. They determined that the cut-point of 80/81 had 93% sensitivity and 87% specificity in diagnosing mild dementia.

The CAMCOG has been used extensively in the assessment of patients with mild cognitive impairment (MCI), dementia, stroke and Parkinson’s Disease. The test takes 20-30 minutes to administer and represents an excellent balance between brevity and breadth of domains explored. The CAMCOG was revised in [14] to include measures of executive function. Leeds et al. [15] examined the executive function measures of the CAMCOG-R in a stroke population. They found the measures vulnerable to depression and suggested that the extra time required to administer the CAMCOG-R may not be justified. Huppert et al. [16] demonstrated that the socio-demographic variables of age, sex, education and social class each exert significant and independent effects on CAMCOG scores. The authors raised cautions about using a single predetermined cut-point when using the CAMCOG as a screening test for dementia. Clinicians administering the CAMCOG are alerted to important contextual factors such as pain, medical conditions and CNS-active medications by information gathered in the CAMDEX structural interview.

COGNISTAT

Cognistat takes 10-20 minutes to administer. It employs a screen and metric approach that saves the clinician considerable time. A patient who passes a demanding “screen” question in a cognitive domain receives a maximum score in that area. When the screen question is failed the examiner administers a “metric” series of questions of graded difficulty. Impairments in individual domains that reach the moderate or severe level of impairment serve as red flags alerting clinicians and family members that specific functional areas are at risk in terms of instrumental ADLs. Patients who pass all the screen questions can complete the test in 10 minutes. Known as the Neurobehavioral Cognitive Status (NCSE) until 1995, Cognistat was first described in the literature in [17]. It explores orientation, attention span and registration before examining five major cognitive domains: language (fluency, comprehension, repetition and naming), visuospatial skills, arithmetic, memory and executive functioning (abstraction and practical judgment). More than 250 peer-reviewed articles describe Cognistat’ s use with a wide range of patient populations including dementia, stroke, traumatic brain injury, neurosurgical, epilepsy, Parkinson’s Disease, psychiatric illness and substance abuse [18].

Schwamm et al. [19] compared Cognistat head-to-head with the Mini-Mental State Exam (MMSE) and the Cognitive Capacity Screening Exam (CCSE) in a neurosurgical population, demonstrating that Cognistat had greater sensitivity than either of the other two tests. In a population of 45 traumatic brain injury patients Nabors et al. [20] found that Cognistat results correlated significantly with NPT findings in the areas of attention, language, memory and spatial skills. Johannson et al. [21] reported that Cognistat outperformed the MMSE in detecting cognitive decline in a primary care setting and demonstrated comparable sensitivity and specificity to independent Neuro Psychological Testing (NPT). Cognistat results are displayed in a graphic profile designed to be easily understood by patients and family members. The pattern of cognitive functioning highlights areas of relative strength and weakness in the separate domains sampled. A checklist helps clinicians identify modifiable factors that can lower test performance [22]. This alerts clinicians as well as patients and family members to the presence of treatable medical [23,24] psychiatric [25] and pharmacological [26] factors that may lower test performance and lead to false positive diagnoses of dementia.

Several electronic versions of Cognistat exist. Cognistat Five is a five-minute, web-based test that screens for delirium, MCI and dementia. It examines orientation, memory and visuospatial skills. Cognistat Active Form is a portable, tablet-based, web-independent test. Cognistat Assessment System (CAS) is a web-based test that is designed for electronic pooling of test results and large-scale research projects. All of these tests provide automatic scoring. Test results are analyzed using an algorithm that utilizes the patient’s age, education and pattern of test results to place cognitive performance on a 7-point continuum (MCI Index) that extends from normal through MCI to dementia. The computer-generated report addresses potential side-effects of any CNS-active medications the patient is receiving. It also provides practical guidance in handling modifiable factors that have the potential to lower test performance.

CASP

Benaim et al. [27] developed the Cognitive Assessment scale for Stroke Patients (CASP) to address the difficulties associated with testing stroke patients who have language (aphasic) difficulties. Non-expert examiners can give the CASP in 10 minutes. The test’s use of visual items allows it to be given to patients with severe expressive (but not receptive) aphasia. Six domains are evaluated: language, praxis, short-term memory, orientation to time, spatial abilities (neglect and constructions) and executive functions. Each of the six functions is scored on a 6-point scale. Scores are presented either as a profile or as sum score. A 2014 study done by Barnay et al. [28] determined that 44 patients with expressive aphasia due a recent left hemispheric stroke were able to be examined with the CASP but none of the patients could be reliably administered either the MMSE or MoCA. A follow-up study of 50 non-aphasic patients [29] showed that neuro-visual impairments contributed equally to the total score on all three tests, suggesting that the CASP was a valid measure of cognitive impairments in both left and right hemisphere stroke patients. Reasoning and judgment are not assessed by the CASP and the psychometric properties of the CAPS remain to be explored.

OCS

Recognizing the importance of domain-specific testing in evaluating acute stroke patients, where aphasia, apraxia and/or neglect can impact test performance, Demeyere et al. [30] developed the Oxford Cognitive Screen (OCS). The OCS assesses five cognitive domains: language, praxis, number processing, orientation/ memory and attention/executive function. The test presents items visually and verbally and utilizes selection of items from a multiplechoice array. The OCS includes tests of praxis and neglect and takes 15-20 minutes to administer. Test results are displayed in a visual snapshot or cognitive profile that consists of a circle with sectors representing cognitive domains and subdomains.

In a 2016 study of the OCS by Demeyere et al. [31] the authors commented that “a standard of truth” does not exist for the assessment of cognition. They chose to compare the OCS with the MoCA as “a current standard of clinical practice.” Looking at a sample of 200 consecutive stroke patients, the authors found the OCS had 87% sensitivity to cognitive impairment compared with 78% on the MoCA [32]. Mancuso et al. [33] examined 325 consecutive stroke patients at 14 different Italian rehabilitation centers using the OCS and the MMSE. Using a cut-off score of 22 on the Italian version of the MMSE [34], they found that only 35.3% of the patients had cognitive impairments on the MMSE while 91.6% had impairments in one or more domains of the OCS. More than 80% of patients showed impairments in two or more of the OCS cognitive domains. The three OCS measures of executive function do not include an assessment of practical judgment. Routine administration of the OCS does not include a survey of CNS-active medications or other factors that can impact attention.

A wide variety of medical and psychiatric conditions impact cognition [34]. The prevalence of cognitive impairment continues to grow with an aging population. While primary care physicians are the frontline screeners for cognitive decline, Bernstein et al. [35] found that only 20% of PCPs report high confidence in interpreting the results of cognitive screening tests. Given the worldwide shortage of neurologists, psychiatrists and geriatricians, the cognitive screening done by PCPs and their staff has become increasingly important. This raises important questions concerning the training of non-medical individuals who perform cognitive screening [36]. Of particular importance is their awareness of shifting state factors that may reduce attention, impair performance on testing and lead to false positive diagnoses.

Reliance on simple sum score screening has come at a high cost to both the public and to health care providers. It has kept family members in the dark by depriving them of a chance to see the specific ways in which a patient’s cognition is intact or impaired. At the same time, it has perpetuated in the medical community a vague, confused notion of cognition as something that can be adequately captured with a global score. This impedes detection of cognitive decline in a range of medical disorders and lowers the quality of professional discourse on cognitive impairment. Cognitive screening tests do not diagnose or measure stroke, brain injury or dementia per se. Instead, they test cognitive domains affected by those conditions. Shifts in cognitive ability influence a patient’s core identity, relationships with others and his or her ability to exercise judgment. Cognitive impairments have medical, legal, financial and safety implications. There is no reason for a physician’s detailed knowledge of genetics, immunology and microbiology to keep company with a primitive and undifferentiated understanding of cognition.

In contrast to sum score tests, each of the mid-range multidomain tests presented above provides a clinician with a differentiated understanding of the pattern and severity of a patient’s cognitive decline. The use of such domain-distinct patient-specific tests enhances diagnostic skills, permits tracking of cognition over time and allows clinicians to play a more evolved role in educating family members and recommending treatment. By capturing a patient’s unique pattern of cognitive strengths and weaknesses, these tests help physicians distinguish treatable confusional states from dementia and assist in the subtyping of dementia. Future versions of screening tests, whether paperand- pen, computer-administered or clinician-administered but computer-assisted, will need to be cross-validated with NPT to guide the development of the next generation of screening tools [37]. In the meantime, physicians who continue to rely on general sum-score tests should be aware of the limitations of these tools. The risk of false positive diagnoses of dementia based on screening tests that ignore factors such as hearing loss, pain, polypharmacy and sleep deprivation needs to be emphasized.

1. Rosen WG, Mohs RC, Davis KL (1984) A new rating scale for Alzheimer’s disease. American Journal of Psychiatry 141(11): 1356-1364.
2. Kueper JK, Speechley M, Montero OM (2018) The ADAS-cognitive subscale (ADAS-cog): Modifications and responsiveness in pre-dementia populations. Journal of Alzheimer’s Disease 63(2): 423-444
3. Mattis S, Jurica P, Leitten C (2001) Dementia Rating Scale-2 (DRS-2). Psychological Assessment Resources 19(1): 145-147.
4. Randolph C, Tierney MC, Mohr E, Chase TN (1998) The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. Journal of Clinical and Experimental Neuropsychology 20(3): 310-319.
5. Stern RA, White T (2003) Neuropsychological assessment battery, Psychological Assessment Resources, Lutz, Florida, USA, pp. 1761-1765.
6. Smith PJ, Need AC, Cirulli ET, Chiba FO, Attix DK, et al. (2013) A comparison of the Cambridge Automated Neuropsychological Test Battery (CANTAB) with “traditional” neuropsychological testing instruments. Journal of Clinical and Experimental Neuropsychology 35(3): 319-328.
7. Bruno D, Vignaga SS (2019) Addenbrooke’s cognitive examination III in the diagnosis of dementia: A Critical Review. Neuropsychiatric Disease and Treatment 15: 4410-447.
8. Noone P (2015) Addenbrooke’s cognitive examination-III. Occupational Medicine 65(5): 418-420.
9. Newman CJ, Bevins AD, Zajicek JP, Hodges JR, Vuillermoz E, et al. (2017) Improving the quality of cognitive screening assessments: ACEmobile, an ipad-based version of the Addenbrooke’s cognitive examination-III. Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring 10: 182-187.
10. Hsieh S, McGrory S, Leslie F, Dawson K, Ahmed S, et al. (2015) The mini-Addenbrooke’s cognitive examination: A new assessment tool for dementia. Dementia and Geriatric Cognitive Disorders 39(1-2): 1-11.
11. Miranda DC, Brucki SMD, Yassuda MS (2018) The mini-Addenbrooke’s cognitive examination (M-ACE) as a brief cognitive screening instrument in mild cognitive impairment and mild Alzheimer’s disease. Dementia & Neuropsychol 12(4): 368-373.
12. Roth M, Tym E, Mountjoy CQ, Huppert FA, Hendrie H, et al. (1986) CAMDEX. A standardized instrument for the diagnosis of mental disorders in the elderly with special reference to the early detection of dementia. British Journal of Psychiatry 149: 698-709.
13. Huppert FA, Jorm A, Girling D, Barkley C, Beardsall L, et al. (1996) Psychometric properties of the CAMCOG and its efficacy in the diagnosis of dementia. Aging Neuropsychology and Cognition 3(3): 201-214.
14. Roth M, Huppert FA, Mountjoy CQ, Tym E (1998) CAMDEX-R: The Cambridge examination for mental disorders of the elderly- revised. Cambridge University Press, Cambridge, UK, pp.1-15.
15. Leeds L, Meara R, Woods B, Hobson P (2001) A comparison of the new executive function domains of the CAMCOG-R with existing tests of executive function in elderly stroke survivors. Age and Ageing 30(3): 251-254.
16. Huppert FA, Brayne D, Gill C, Paykel ES, Beardsall L (1995) CAMCOG-a concise neuropsychological test to assist dementia diagnosis: Socio-demographic determinants in an elderly population study. British Journal of Clinical Psychology 34(4): 529-541.
17. Kiernan RJ, Mueller J, Langston JW, Van DC (1987) The neurobehavioral cognitive status examination: A brief but quantitative approach to cognitive assessment. Annals of Internal Medicine 107(4): 481-485.
18. Mueller J, Kiernan JR, Langston JW, Flanagan R (2018) Cognistat Manual. Northern California Neurobehavioral Group, California, USA.
19. Schwamm LH, Van DC, Kiernan RJ, Merrin E, Mueller J (1987) The neurobehavioral cognitive status examination: comparison with CCSE and MMSE in a neurosurgical population. Annals of Internal Medicine 107(4): 486-491.
20. Nabors NA, Millis SR, Rosenthal M (1997) Use of the neurobehavioral cognitive status examination (cognistat) in traumatic brain injury. Journal of Head Trauma Rehabilitation 12(3): 79-84.
21. Johansson MM, Kvitting AS, Wressle E, Marcusson J (2014) Clinical utility of COGNISTAT in multi-professional team evaluations of patients with cognitive impairment in Swedish primary Care. International Journal of Family Medicine 2014: 649253.
22. Yaffe K (2018) Modifiable risk factors and prevention of dementia. JAMA Internal Medicine 178(2): 281-282.
23. Tingley K (2020) Can hearing aids help prevent dementia? New York Times, New York, USA.
24. Marquié M, Castilla MM, Valero S, Martinez J (2019) Visual impairment in aging and cognitive decline: Experience in a memory clinic. Scientific Reports 9(1): 8698.
25. Sahin S, Okluoglu OT, Cinar N, Bozdemir N, Cubul R, et al. (2017) Distinguishing depressive pseudodementia from Alzheimer disease: A comparative study of hippocampal volumetry and cognitive tests. Dementia and Geriatric Cognitive Disorders 7(2): 230-239.
26. Shmerling RH (2018) Could medication contribute to dementia? Harvard Health Blog.
27. Benaim C, Barnay JL, Wauquiez G (2012) Presentation of the cognitive assessment scale for stroke patients. Annals of Physical and Rehabilitation Medicine 55(S1): e188.
28. Barnay JL, Wauquiez G, Bonnin KHY, Anquetil C, Pérennou D, et al. (2014) Feasibility of the cognitive assessment scale for stroke patients. Annals of Physical and Rehabilitation Medicine 57(6-7): 422-435.
29. Barnay JL, Wauquiez G, Bonnin KHY, Anquetil C, Pérennou D, et al. (2015) The cognitive Assessment Scale for Stroke Patients (CASP) vs the MMSE and MoCA. Annals of Physical and Rehabiitation Medicine 58(2): 78-85.
30. Demeyere N, Riddoch MJ, Slakova ED, Bickderton WL, Humphrey GW (2015) The Oxford cognitive Screen (OCS): Validation of a stroke specific short cognitive screening tool. Psychological Assessment 27(3): 883-894.
31. Demeyere N, Riddoch MJ, Slakova, Jones K, Reckless I, et al. (2016) Domain-specific versus generalized cognitive screening in acute stroke. Journal of Neurology 263(2): 306-315.
32. Mauro M, Nele D, Laura A, Alessio D, Valentina V, et al. (2018) Using the Oxford Cognitive Screen (OCS) to detect cognitive impairment in stroke patients: A comparison with the mini-mental state examination. Frontiers in Neurology 28(9): 101.
33. Magni E, Binetti G, Bianchetti A, Rozzini R, Trabucchi M (1996) Mini-mental state examination: A normative study in Italian elderly population. European Neurology 3(3): 198-202.
34. Van DC, Mueller J, Kiernan RJ (1987) The case for psychiatrists as authorities on cognition. Psychosomatics 28(2): 87-89.
35. Alissa B, Kirsten MR, Katherine LP, Natasha ZRS, Christine SR, et al. (2019) Dementia assessment and management in primary care settings: A survey of current provider practices in the United States. BMC Health Services Research 19(1): 919.
36. Aleccia J (2019) Worries about accuracy of widely used test prompt training requirement. Kaiser Health News, California, USA.
37. (2015) FDA discussion paper: Neurodiagnostic devices intended to assess cognitive function. Silver Spring, Maryland, USA.