Assessment Validity and Compliance

The information contained on this page is for general informational purposes and is not a substitute for professional legal advice.

This page outlines the process Vervoe takes to develop assessments that adhere to high standards of validity so that employers can understand how assessments fit into their compliance profile. Vervoe assessments are created according to this process to ensure all tests and AI Grading mitigate discrimination or unfair assessment practices. 

On this page:

• EEOC Specific Information
• Assessment Creation 
• Assessment Validity
• Minimizing Bias
• Client Responsibilities

 

EEOC Specific Information

In the United States, the Equal Employment Opportunity Commission (EEOC) has provided guidance to help companies mitigate discriminatory practices in tests and other selection procedures for pre-hire and selection of employees. These guidelines are designed to provide clarity on requirements under the law, including:

• Title VII of the Civil Rights Act of 1964 (as amended by the Equal Employment Opportunity Act of 1972)
• Title I of the Americans with Disabilities Act (ADA)
• The Age Discrimination in Employment Act (ADEA)

The information on this page outlines how Vervoe approaches assessment creation and use to ensure there is no disparate impact on a particular group, is job-related and consistent with business-necessity. 

 

 

Assessment Creation

Vervoe utilizes a standardized process for creating skills assessments across different industries and role types. Our process ensures Vervoe’s assessments are rigorously developed following the standardized procedures outlined above. All assessments have a focus on skills and are administered online via the Vervoe platform. This ensures every test taker has the same opportunity to take the assessment with the same set of instructions, procedures, and time limits (if applicable) to keep test-taking conditions consistent across roles, organizations, and assessments. 

Vervoe assessments take into account the following information when created:

• Identification of key skills required to perform the job. Relevant job materials, including job and position descriptions, are analyzed to identify the skills and attributes for success in the role. Where applicable, Vervoe consults with key personnel in the organization and further analyzes key performance indicators and skill matrices. This process ensures that each assessment includes the skills candidates require to perform.
• Consultation with experts. To create bespoke assessments, Vervoe involves subject matter experts from within your organization, or more widely in the marketplace. Through consultation, we develop content specific content aligned with skills and within the context of the role. For technical assessments, Vervoe partners with subject matter experts for content creation and verification.
• Business outcomes. All bespoke assessments created for our customers is directly tied to relevant and critical business outcomes. This is an iterative process to determine the predictive validity of an assessment used at the start of a recruitment process.
• Quality Assurance. Before publication, all assessments are reviewed by Vervoe’s in-house assessment and I/O Psychology team. If assessments have been created specifically for your organization, your relevant subject matter experts are heavily involved in the quality assurance and approval process before use.

Vervoe’s proprietary AI grades candidates based on their interaction with the platform on how they take the assessment and what their answers are to each question. Candidate’s background, education, location, gender, race, or any other personal factors are not known by the software and, therefore, not considered when grading responses. The only analyzed elements are how well an individual can answer the questions and, therefore, perform the role. 

Furthermore, Vervoe has quality assurance checks in place whereby, if the AI identifies a possibility of bias that has crept into manual grading, it is surfaced and investigated by the data science team. If there is found to be a breach, scoring will be adjusted. This is another measure Vervoe has implemented to reduce the likelihood of bias in the assessment and recruitment process. 

Our process ensures that every assessment appropriately tests the relevant skills (technical and soft) required to perform the role and can be replicated each time a new assessment is created. 

 

 

Assessment Validity

Summary:

• Any test that directly mimics what a person will do on the job can be considered “validated.”
• Tests of personality and soft skills are a riskier prospect even when they are “validated,” because they often lack the proper validation required to be EEOC compliant.
• Positive candidate experience and perceived fairness are two of the primary reasons why skill testing is an effective and expedient hiring practice.
  
  

Overview

The value of testing candidates prior to hiring them is having an expedient way of assessing their fitness for the job. The decisions made based on a test, however, cannot be any better than the information provided by the test, which makes it extremely important that the test is an accurate representation of the constructs that it represents. 

This is somewhat difficult, however, because the constructs being assessed, such as knowledge, skills, attitudes, and cognitive processes, cannot be known objectively and must be inferred. To that end, there are many statistical measurements that are used to assess the degree to which tests are reliable (consistent) and valid (accurate).

There are many types of psychometric validity, and it is a rare test (if such even exists) that hits every type. Looking specifically at tests for finding job fit, there are a few different types of validity that are particularly relevant, not just to ensure that the hire is a good one, but to ensure compliance with EEOC regulations.

 

 

Establishing Validity in Vervoe Assessments

There are multiple categories that make up an assessment’s validity, all of which our behavioral science team focuses on when creating all of Vervoe’s assessments:

 

Face Validity

Does it look like the test is assessing what it claims to measure?
Face validity is the most basic form of validity and requires general consensus that someone taking this test would need to exhibit the constructs that the test is assessing. For instance, a set of math problems being used to assess arithmetic ability has more face validity than a set of word problems, because the latter are assessing a combination of arithmetic skills and reading comprehension (which means the test is assessing arithmetic skills in a specific context, but not in general), etc. For some types of skill tests, this is the main type of validity available, and sometimes the only one that can be obtained when a test is first created.

Generally, these sorts of skill tests require job candidates to engage in tasks that could theoretically be completed by anyone who can do the job. As long as the tasks mirror activities that the individual will need to be able to complete in order to do the job properly, the questions have face validity and are EEOC compliant. 

That said, a company should always take care not to have a “disproportionate impact” upon a particular demographic group, as this can be an indicator that something is wrong with the test. For instance, if only 75% of members of Demographic A pass the test, while it’s 93% for other demographics, the test still may be EEOC compliant, but an investigation is a must.

Face validity in practice
An assessment for a call center agent might have questions about managing an angry customer on the phone—this would have high face validity. Comparatively, if call center agents are asked to complete a financial modeling task, candidates would be left wondering what relevance is to the position. This would have low face validity. 

All our assessments are designed for specific job roles, and will only include questions relevant to that position, within the context of your organization, ensuring all assessments have high face validity. 

 

Content Validity 

Does the tests cover the full range of the construct that it is supposed to measure?
Content validity refers to whether a given test covers a representative sample of the full range of the construct that it is measuring. For job tests, the matter of validity should be focused most on this point. In addition to ensuring that the questions appear to be about topics related to the position (face validity), the set of questions (as a group) needs to assess a sufficient range of tasks so that the evaluator can know that the set of answers paints a picture of whether the individual is capable of doing the collection of tasks required by the job.

As long as the set of questions provides a representative sample of the tasks the candidate will need to do on the job, and someone who can do the job properly can definitely provide solid answers to the questions, the questionnaire has content validity and is EEOC compliant. (Again, it is important to be aware of the possibility of disproportionate impact, and to be prepared to alter the task[s] if you are seeing a particular demographic scoring lower than another.)

Content validity in practice
Following on from our previous example, if call center agents only answer questions relating to dealing with an angry customer, this assessment would have low content validity, because there are many other facets of the role. However, if the assessment focuses on multiple skills required to successfully perform the role like dispute resolution, customer service, sales, and multitasking, this assessment would have high content validity as it covers the range of tasks required of the job. 

Our team works in consultation with the experts in your organization to understand the critical components of the role, and ensure high content validity in every assessment. Typically our team will work closely with people within your organization who have a deep understanding of the job. The process may involve 1:1 meetings with managers or staff in the role, understanding the role’s challenges, high performers’ attributes, and areas for development. 

We ask for access to position descriptions, training material, adverts, and performance metrics to fully understand what the job entails and, therefore, what should be included in the assessment. From this research and investigation, we strive to include a minimum of 3 – 4 key skills required to successfully perform the role to ensure the assessment encompasses the full construct that is being assessed and covers various elements of the job. 

From this, you can be confident that candidates who score highly on the assessment are better suited to the role and likely to perform better.

 

Construct Validity 

Does the test actually measure the theory-based construct that it claims to measure?
In hiring, construct validity has the greatest bearing on tests that do not directly assess knowledge and skills that will be used on the job. Rather, it is about determining whether, say, a test of attention to detail (an example of a “soft skill”) or extraversion (personality trait) actually demonstrates that the person has the indicated characteristic. When HR people ask if a test is “validated” or has “psychometric validity,” this is usually what they are talking about. 

But, when it comes to tests of skills to be used directly on the job, construct validity is far less relevant compared to face and content validity.

Tests of general cognitive ability and/or personality, however, absolutely need to have construct validity to be EEOC compliant, because they are indirectly related to successfully completing the actual tasks of the job. For a personality or “soft skill” test to be valid, it must assess a theoretical construct whose existence can be defended through a review of the scientific literature, and statistics must show that the test questions do indeed work together to describe a coherent construct.

 

Predictive/External Validity 

Does the information we learn from test performance predict/apply to performance in other situations?

Predictive/External validity is about whether the test predicts performance in areas that have not been directly tested. Even though a test may have the above types of validity, there remains a difference between the tasks that are tested in a context-free fashion, and doing the actual tasks in the context of the job. If a company is going to use a test, it is always helpful to prove that the test actually predicts high performance on the job and/or good fit with the company/team. For a test of skills that will definitively be used on the job, predictive validity is very helpful, but technically not required, for EEOC compliance. But, for any other type of test, this is crucial for a company to have, and they have to run the numbers within their own company.

One of the most important points to recognize is that predictive validity does not transfer between companies. If a study proves that Test A predicts high performance at a Big 10 consulting firm, that does not help a Big 4 accounting firm at all (especially if they are having to defend themselves against an EEOC-related suit). What good is a personality test (for example) if you cannot prove that it distinguishes between high and low performers in your company? 

Unfortunately, most tests cannot do such a thing, in part because most people who run tests of predictive validity do so only to prove that a particular combination works – what they fail to do, however, is prove that it is the best or only combination that works (this can be done, but it is very difficult and work-intensive to do). In turn, this means they haven’t actually proven that other scores should be causing people to be rejected, which can be risky and can cause companies to miss out on fantastic talent. 

Moreover, a given company can prove that a test has predictive/external validity only after using it for a while and measuring on-the-job performance, which means that the company cannot initially assume that these indirect measures of job success are valid.

Predictive/External validity in practice
Vervoe assessments have been proven to have predictive validity with our clients. For example, candidates hired using a Vervoe assessment in one organization outperformed sales targets by 4%. Another showed that Vervoe employees received higher customer satisfaction scores in their call center roles.

 

Skill assessment validity and compliance

One of the primary values of using skill-based testing in a hiring process is that there is a clear correspondence between the items on the tests and the tasks that will be performed on the job. As such, skill tests have both face validity and content validity, which makes them EEOC compliant. In addition, these tests expediently show which candidates are nominally capable of doing the job before employers spend time assessing a candidate’s application materials. By contrast, indirect tests of future job performance, like personality tests and assessments of general cognitive abilities and/or “soft skills” require extensive assessment to confirm construct and predictive/external validity, which makes them a riskier and less expedient move.

In all cases, however, it is imperative that employers keep an eye out for demographic disparities in performance, and to launch an immediate investigation if a significant one is found. At Vervoe, we are constantly monitoring our tests to ensure that candidates are taking a fair test that assesses the skills they will use to get the job done well. As an added benefit, many companies find that using fair tests also yields a more diverse slate of finalists that is more representative of the wide range of talent that exists in the world.

Though personality and cognitive assessments are very hard to validate for the hiring process, they can be helpful in onboarding and coaching new hires in a way that is tailored to their uniqueness, which can give them a smooth and powered-up start. But, what matters most of all is the candidate experience. An enjoyable candidate experience, combined with a perception that the process is fair (along with the test actually being fair!), is the best way to ensure that candidates will not only show their capabilities to the company but have a positive view of the company regardless of the outcome of the hiring process.

 

 

Minimizing Bias

Vervoe’s assessments are rigorously developed following the standardized procedures outlined above. All assessments have a focus on skills and are administered online via the Vervoe platform. This ensures every test taker has the same opportunity to take the assessment with the same set of instructions, procedures, and time limits (if applicable) to keep test-taking conditions consistent across roles, organizations, and assessments. 

Vervoe’s proprietary AI grades candidates based on their interaction with the platform on how they take the assessment and what their answers are to each question. Candidate’s background, education, location, gender, race, or any other personal factors are not known by the software and, therefore, not considered when grading responses. The only analyzed elements are how well an individual can answer the questions and, therefore, perform the role.

All questions within an assessment are written in ways that enable every candidate; regardless of background, race, gender, education or religion, can answer fairly if they have the relevant skills and there is no disparate treatment or impact. 

Bias and Disparate Impact

Vervoe takes rigorous steps to ensure the fairness and impartiality of our AI-driven assessments. To achieve this, we collaborate with Holistic AI, an independent auditor, to conduct thorough audits on our machine learning algorithms. These audits are designed to verify that our assessments and grading processes are free from bias and do not result in any disparate impact on candidates. Since our machine learning models evaluate candidates based solely on their performance in assessments, without access to any personal information, the integrity of our evaluation process is maintained.

The Audit Report can be reviewed here. 

To further support the validity of our platform, we’ve incorporated several key platform features: 

Closing the Loop: 

This functionality allows employers to mark candidates as ‘Hired’ within our platform, including the hire date and hiring manager’s details. A unique aspect of this feature is the automated survey sent to hiring managers 120 days post-hire, which assesses the employee’s performance on a scale of 1 to 10 and queries about their potential rehire if they were to hire for the role again. This direct feedback serves as a measure of predictive validity, offering concrete data on the hired candidates’ job performance. 

Assessment Insights: 

Our platform offers detailed insights into the overall performance distribution across assessments and breaks down scores by individual skills. This analytical view helps employers to verify whether an assessment is effectively distinguishing among candidates or if it might be too challenging or too simple, based on the distribution patterns. Adjustments to the assessment content can be made accordingly to ensure a balanced evaluation of candidate skills. 

Question-by-Question Analytics: 

To further refine the quality of our assessments, Vervoe provides data on how often each question is viewed and answered. This information helps identify if certain questions may be too difficult or irrelevant to the role, as indicated by a high view rate but low response rate whereby employers can make necessary changes to the assessment questions. 

Client Responsibilities for Use of Vervoe Assessments 

Clients who create their own questions or assessments, or alter questions or assessments, to test candidates are responsible for ensuring their assessments adhere to EEOC guidelines and minimize disparate impact or any potential discrimination in the screening process. 

Vervoe does not take responsibility for assessments created by individual employers and does not support assessments that intentionally screen candidates on factors other than their ability to do the job. 

It is the employer’s responsibility to offer alternative screening methods for candidates who request it, as required by the Americans with Disabilities Act or any equivalent laws. 

Employers who utilize Vervoe’s established assessments understand the limitations of using an assessment as a part of their screening process and agree to only use assessments that are appropriate for the role that is being hired for. 

Further, the employer must ensure that all assessments are used in accordance with the EEOC’s guidelines for fair and just hiring practices