VIRTUAL MEETUPS - INSIGHTS

What’s the Buzz About?

Nov 5, 2019: Assessing Psychometric Assessments

Several organizations in India have begun exploring psychometric assessments and their importance and significance to business, especially in the last decade. Six industrial-organizational psychology professionals from different parts of the world (i.e., USA, India, and Canada) attended COSI’s second virtual conversation series on Nov 5th, 2019, to discuss the importance of psychometric assessments. This synopsis highlights the key topics discussed: (1) Importance of psychometric measurement, (2) Issues with psychometric measurement and their implications on business, (3) Possible solutions

Psychometric Measurement – Why is it important?

Measurement is an essential activity of science. Knowledge is acquired about people, objects, events, and processes by observing them. Making sense of these observations requires that they be quantified. The process of measurement and the broader scientific questions it serves interact with each other and the boundaries between them are often barely visible (Maul, Irribara, & Wilson, 2015). In the organizational context, decisions based on psychological measurement can impact decisions about several key activities which impact individuals and the organization itself – hiring, promotion, development, and succession planning and so on. Measuring the key human/individual attributes (e.g. personality, competencies, etc.) that might play a role in these decisions is what ‘psychometric assessment’ is designed to do. To the extent that this activity is poorly understood and/or executed, the human and organizational costs of poor decisions will be magnified.

Issues with psychometric measurement and their implications on business

In the workplace, it is important to use assessments that are reliable and valid. Reliability refers to how dependably or consistently a test/assessment measures a characteristic. Validity refers to how well or accurately it measures, what it claims to measure. Some of the issues discussed during the virtual event were: using off-the-shelf assessments which have psychometric issues related to validity (especially if they are not normed or validated locally in the Indian context in question), the issues around making accommodations for applicants with disabilities (including some hilarious misunderstanding of the term ‘accommodation’ itself in this context), and the availability of validation data (with reluctant and overeager clients making validation difficult), to name a few.

Additionally, in addressing the psychometric properties of an assessment, legal defensibility is also an important aspect to consider – however, in India, with employment laws not really evolved to the extent that requires strict adherence to testing standards, it is up to the assessment developer and user to ensure that high standards are maintained. This essentially means that each assessment needs to have a body of evidence behind it from its reliability and dependability over time, its consistency in how it is scored and its ability to benchmark and compare between people.

Another issue pertinent to the use of psychometric assessments in the Indian context is the use of assessments that were developed for a different population, that were not culturally adapted to be used for the population it was being used for. The implications of using assessments that are not reliable or validated can affect the decision making processes of the organizations using these tests. Organizations invest capital in selection, hiring and promoting employees, and if the assessments used do not meet the necessary criteria, it could lead to ineffectiveness of the assessment in predicting future success, having a generic assessment tool not customized to the particular job/organization, and worse, a waste of time, money and effort.

Possible solutions

One reason why more organizations do not use rigorous assessments to select employees is because many executives and HR professionals have misconceptions about the value of using them. Another reason why formal assessments are not used more in organizations is that there tends to be a lack of knowledge about the types of assessment methods that research has shown to be most effective for identifying who will perform best on a job. This, coupled with the fact that the area of selection testing is inherently technical and difficult to understand, leads many organizational decision makers and HR professionals to shy away from using formal assessments to guide their selection decisions. Thus, a central theme that emerged is the importance of further education and outreach required to familiarize assessment users and HR professionals in particular, on ‘what to look for’ before using psychometric assessments in India.

One more reason why the educated use of psychometric assessment is diluted has to do with the multitude of consulting firms selling different selection products and tools. It is important for organizational decision makers and HR practitioners to be educated consumers regarding these products to ensure they are bringing competently developed and effective assessment methods into their organizations.

Prior to using psychometric assessments in India, practitioners should ask questions around the following:

Origin and development (especially if it is not an in-house or custom developed assessment)
Credibility of test/assessment developers (are they trained industrial-organizational or educational psychologists, with sound foundations in developing reliable and valid assessments?)
Localization efforts (local validation, cultural adaptation and scoring/norming for use in the local setting)
Costs and benefits (some rigorous methods such as in-person assessment centers might be costly but some cheap solutions might prove useless or counter-productive if it impacts applicant experiences or yields poor quality candidate pools)

Suggested references and further readings

https://trainingindustry.com/articles/measurement-and-analytics/how-to-assess-assessments-making-informed-decisions-about-selection-and-development-assessments/
https://www.researchgate.net/publication/308720768_Selection_Assessment_Methods
https://www.siop.org/Business-Resources/Employment-Testing
Assessment, Measurement, and Prediction for Personnel Decisions, Robert M. Guion
Essentials of Personnel Assessment and Selection, Scott Highhouse, Dennis Doverspike, Robert M Guion
Ryan, A. M., & Tippins, N. T. (2004). Attracting and selecting: What psychological research tells us. Human Resource Management, 43, 305-318.
Rynes, S. L., Colbert, A. E., & Brown, K. G. (2002). HR professionals’ beliefs about effective human resources practices: Correspondence between research and practice. Human Resource Management, 41, 149-174.
Pulakos, Elaine D. (2005). Selection Assessment Methods. SHRM Foundations

What’s the Buzz About?

Sep 27, 2018: Emotional Intelligence

In the 1990s, Peter Salovey introduced a concept known as Emotional Intelligence (EI) to the world. Later, it was popularized by Daniel Goleman in his book: Emotional Intelligence. He contended that EI contributes to an individual’s successful navigation in a social environment by appraising and regulating emotions in the self as well as in others, and eventually using them adaptively to achieve one’s goals. Needless to say, there was a surge in EI research and practice thereafter.

The COSI virtual panel was interested in exploring this construct and its practical significance to business. Five industrial-organizational psychology professionals from different parts of the world (i.e., USA, India, and Canada) attended the first of COSI’s virtual conversation series to discuss EI. This is a synopsis of their discussion which can be thematically categorized into: (1) Definition, (2) Measurement, and (3) Practical use.

Definition: What Exactly Is EI?

EI lacks a universal understanding of its conceptualization; it tends to be used loosely and is sometimes conflated with other constructs. Tracing its history, researchers in the past have distinguished between EI as a trait vs. EI as an ability (Mayer, Salovey, & Caruso, 2000). The ability-based model states that EI is the emotional-reasoning ability, which is distinct from traits and can be assessed through measures such as situational judgment tests. In contrast, the trait model of EI states that it is an array of non-cognitive, dispositional characteristics that influence one’s ability to successfully cope with environmental stressors. This model of EI was heavily criticized for including everything not covered by cognitive ability and for being too redundant with personality traits to justify a distinct construct. It is therefore clear, that the definition of EI is unclear!! For what it’s worth, the Mayer-Salovey model of EI as an ability has had more research support than the Daniel Goleman (more popular but less scientifically robust) idea of EQ as ‘everything but the kitchen sink’!

Measurement: How Is It Assessed?

Closely related to the previous theme, the measurement of EI is also an important issue. The Emotional Quotient Inventory (EQ-I; Bar-on, 1997), and the Mayer, Salovey, Caruso Emotional Intelligence Test (MSCEIT; Mayer, Salovey, & Caruso, 2002) are commonly used measures.

Analogous to the ability-based and trait-based EI models discussed earlier, Law, Wong, Huang, and Li (2008) mentioned that there are two types of EI measures: task-performance and self‑report. The former is more theoretically grounded, predicts well for high emotional-labor jobs but shows substantial demographic differences. On the other hand, the latter shows robust criterion validity and smaller demographic differences but lacks adequate theoretical grounding (Joseph & Newman, 2010).

There is unanimous agreement that there is a need for more research and subsequently better understanding of EI in our field. Until then, it is wise to use EI measures in conjunction with other valid and reliable measures.

Practical use: How Are People Using it?

Soon after its inception, EI raised great interest among researchers in terms of its ability to predict important outcomes. Further discussion relating to practical utilization of EI showed that EI seems to be a good measure for evoking insight, self-awareness, and personal development and hence may be appropriate for use in executive coaching; however, using EI for selection is unanimously deemed challenging and legally risky.

I-O psychology has some robust predictors of job performance at its disposal (e.g., GMA, conscientiousness); they are inexpensive (for the most part), easy to explain and use, and (most importantly) have demonstrated strong empirical evidence supporting their use, so their emphasis is unlikely to change. As for EI, although it has demonstrated some criterion validity for some micro- and macro-level outcomes (Charniss, 2001; Zeidner, Matthews, Roberts, 2004) such as career success and adjustment, its use for selection may be difficult to defend, especially when the extensive local-validation efforts are lacking. That said, EI can perhaps help employees better adapt to their work environment, cope effectively with work pressures, perform well on the jobs involving interaction, and socialize well with co-workers.

In summary, EI is an intuitively appealing construct; however, the jury is still out on whether EI is unique (or just an example of construct proliferation); whether it is a trait or an ability; how to best measure it; and whether it is worth using it. Additionally, research has largely given up on EI whereas practice has not. For I-O practitioners, it is sometimes challenging to make their clients understand the issue of mixed evidence for EI. Their best bet, therefore, is understanding what EI means for their clients and using “that” conceptualization to find the most valid and reliable tool at their disposal, and expending it to solve the problems their clients/organizations face!

References

Bar-On, R. (1997). The emotional intelligence inventory (EQ-I): Technical manual. Toronto, Canada: Multi-Health Systems.

Cherniss, C. (2001). Emotional intelligence and organizational effectiveness. In C. Cherniss & D. Goleman (Eds.), The emotionally intelligent workplace (pp. 3–12). San Francisco: Jossey-Bass.

Joseph, D. L., & Newman, D. A. (2010). Emotional intelligence: An integrative meta-analysis and cascading model. Journal of Applied Psychology, 95(1), 54-78. doi:10.1037/a0017286

Law, K.S., Wong, C. S., Huang, G. H., & Li, X. (2008). The effects of emotional intelligence on job performance and life satisfaction for the research and development scientists in China. Asia Pacific Journal of Management, 25(1), 51-69. doi:10.1007/s10490-007-9062-3

Mayer, J. D., Salovey, P., & Caruso, D. R. (2000). Models of emotional intelligence. In R. J. Sternberg (Ed.). Handbook of Intelligence (pp. 396-420). Cambridge, England: Cambridge University Press.

Mayer, J. D., Salovey, P., & Caruso, D. R. (2002). Mayer– Salovey–Caruso Emotional Intelligence Test (MSCEIT) user’s manual. Toronto, Ontario, Canada: MHS Publishers.

Zeidner, M., Matthews, G., Roberts, R.D. (2004). Emotional Intelligence in the Workplace: A Critical Review. Applied psychology: An International Review, 53 (3), p. 371. doi:10.1111/j.1464-0597.2004.00176.x