Applicant Screening and Assessment

In " (PDF file) Reviewing Applicants: Research on Bias and Assumptions”, the authors highlight a number of studies demonstrating the implicit or unconscious assumptions that people hold, allowing such assumptions to influence their judgment and their subsequent decisions in important matters such as hiring. Your committee is strongly encouraged to review the recommendations offered in the article for mitigating bias and improving the quality of your decisions during the applicant screening and selection process. 

Furthermore, your committee is advised to review the Applicant Diversity Self-ID information available in the recruitment and determine if there is sufficient diversity present in the applicant pool or whether additional candidate sourcing (i.e. advertising and outreach) is necessary. 

Applicant screening should occur based on a clearly defined selection criteria. Committees often develop rubrics to aid in the screening and selection process. Rubrics provide a set of criteria and a rating scale that assist with measuring each candidate’s qualifications for the position. In their article, titled “ (PDF file) Do Rubrics Live up to Their Promise? Examining How Rubrics Mitigate Bias in Faculty Hiring”, Culpepper et al. identify the benefits and pitfalls of rubrics. 

While rubrics have the potential to enhance fairness and equity in the hiring process and committees should strive to develop and use them, committees should also be aware that poorly defined rubrics with broad criteria, lacking clear thresholds and examples, could in fact result in lack of consensus on committees and poor hiring decisions. The authors in the above noted article write: 

"Rubrics seemed to mitigate bias and enhance decision-making when committees took time to deliberate the criteria, ensuring that all committee members agreed about what the criteria meant and what evidence they would use to assess whether candidates met the criteria."

Therefore, committees are strongly encouraged to incorporate the following better practices into their process: 

• Establish a shared understanding of the standards and expectations with respect to the selection criteria. Identify the threshold for each criterion and the acceptable evidence from a candidate’s application. Consider the possible variations in interpretation of each statement; for example, in a seemingly clear statement like “recent grant-seeking activities”, there is the need to define “recency” and “grant activity”. It’s important to develop rubric subcomponents that define what constitutes excellence in SRC, teaching and service. 
• Ensure that your criteria are not intrinsically biased. For example, seemingly objective criteria such as number of grants, amount of research funding, and number of citations could unfairly advantage applicants with greater privilege or applicants whose research area involves significant grant funding, while blocking other high potential and highly accomplished applicants from advancing in the process simply because the criteria is not inclusive enough. 

• Embed EDI criteria into the core domains (i.e. teaching, SRC and service) and ensure that said criteria are meaningfully weighted. 
• Put the draft rubric to test by reviewing a few examples of applications against it and discuss as a committee how each member applied the rubric and what changes, if any, need to be made to the rubric to enhance consistency and fairness in the evaluation process. The rubric can be tested against a select number of applications from the existing applicant pool or the committee chair could use a few applications from a previous search, with names redacted, for this exercise. 
• Discuss and decide how the committee will use the rubric scores to make decisions. For example: 
  • Will the rubrics be used to implement a “cut-off” score resulting in a list of candidates who will be reviewed more closely during committee deliberations? 
  • Are there additional ways to consider candidates who may have been strong in a certain area but who had not scored very well overall? For example, if a candidate scored particularly high in one or more domains, could they be considered further and under what circumstances? (It might be tempting to include candidates that the committee is particularly “excited” about, but it’s important to identify what specifically would warrant that excitement, so that there is fairness, consistency and objectivity in the process). 
  • Will your committee re-review and discuss all candidates from equity-deserving groups who perhaps were within certain points of the cut-off score? 

• How far into your process will you use the rubrics - longlisting, shortlisting, selection of final candidate? Or perhaps, are different sets of rubrics needed for different stages (i.e. will your committee be measuring different things during the interview/campus visit vs. what it measured at the time of screening?). 

• Have each application reviewed by more than one committee member. Alternatively, Culpepper et al. recommend evaluating candidates from equity-deserving groups a second time with bias in mind. 
• Schedule enough time to review each application thoroughly, using an anti-deficit approach, looking for reasons to include vs. exclude the applicant in the long/shortlist. 

• When screening, committee members should ask themselves whether they are being influenced by any factors such as: 
  • An applicant they know, or whose advisors or mentors they know, or who holds a degree from their alma mater or have any other attributes in common with them. 
  • A background or qualifications of a candidate that may be unconventional and therefore prone to being held to a different standard. Consider whether the achievements are being fully recognized and valued and whether the applicant (vs. their mentors or supervisors) is the one being credited for the achievements. 
  • Assumptions about the applicant’s potential or interest in the position (e.g. an assumption that a female candidate will not be willing and able to take on a leadership position in a few years). 
  • Language in reference letters (if applicable), recognizing that letter writers can be prone to minimizing the contributions or achievements of equity-deserving groups (e.g. men’s research is often described as seminal whereas women researchers might be described as warm and collaborative). 
• Consider whether the long/shortlist reflects the diversity in the applicant pool (e.g. if 50% of the applicant pool identified as women, is there close to that representation in the long/shortlist?). If needed, consider:  
  • Returning to the applications of those who were just below the cutoff to determine whether it is possible to include any of them in the long/shortlist, so that more information about their qualifications can be collected at that stage. 
  • Whether it is possible to further diversify the long/shortlist. The greater the diversity on the long/shortlist, the more likelihood of the committee members being able to assess the candidates fairly. Consider intersectionality as well. 

A  (google doc) template rubric can be found here (external link)  and should be customized to reflect the position your committee is hiring for. Additionally,  (google doc) a template for evaluating candidates can be found here (external link)  and should also be customized to reflect the selection criteria in the rubrics established by your committee. Committees may consult with the OVPFA on draft rubrics.  

In the recruitment portal, applicants are asked whether they are legally entitled to work in Canada. Those who select “yes” may be permanent residents or citizens of Canada or they may hold a work permit. If they hold a work permit, a review of the terms and conditions on the work permit is necessary and can be undertaken by the OVPFA/HR and the committee can be advised further based on that review. 

If foreign applicants do not hold a valid work permit, they are also asked whether they may be eligible for a work permit under one of the free-trade agreements as a citizen of one or more of the following countries: United States, Mexico and/or Chile. Note: applicants are not asked to indicate the specific country of citizenship. 

Applicants’ responses to these questions are visible to the hiring/appointments committees in the recruitment portal. 

For all other foreign candidates, TMU will need to seek authorization from the government in the form of a Labour Market Impact Assessment (LMIA). This is obtained by demonstrating to the government that we complied with their advertising/recruitment requirements, considered all Canadian (permanent residents and citizens) applicants and found none to be qualified. Upon a successful LMIA application, the foreign hire will be able to apply for a work permit. 

Hiring/appointments committees are asked to contact vpfa@torontomu.ca or Zenab Pathan, Director, Faculty Recruitment and Development at zenab.pathan@torontomu.ca as soon as possible to discuss the possibility and options associated with considering foreign candidates. 

When considering applicants who are or have been internal Limited-Term Faculty (LTF) or contract lecturers, the hiring committee may review the Performance Conduct File or the Service Record Files as appropriate during the search process to access relevant information. For faculty positions, internal candidates cannot be given access to the CVs of those on the preferred candidate list, nor can they attend the public presentation and offer input on candidates.

Longlisted candidates are often interviewed virtually. Guidance on conducting virtual interviews can be found in this  (google doc) virtual interview checklist (external link) .

Shortlisted candidates are typically invited for in-person interviews. For faculty positions, the Collective Agreement refers to developing a list of preferred candidates (i.e. shortlist). Those on the preferred candidate must be invited for a campus visit which includes a formal interview by the Department Hiring Committee (DHC) and teaching and research presentations and the visit may also include meetings with other relevant individuals or groups on campus.

No questions should ever be asked that elicit information from candidates about any of the prohibited grounds (external link)  under the Ontario Human Rights Code. Unsolicited information should also not be considered. 

Interview questions should be developed based on the selection criteria that is created based on the qualifications in the job advertisement. For example:

See Recruiting and Hiring Diverse Faculty Guidelines for further guidance and sample questions. 

• Different types of questions serve different functions. For example, “Tell us about yourself”, “What attracted you to this position?” and “What do you believe makes you qualified?” are typically the first questions asked, intended to put the candidate at ease. 
• Questions such as “How does your current research program fit/work with the department and the university?” and “What pedagogical changes do you see on the horizon in your discipline?” allow the candidates to demonstrate their knowledge of the field and understanding of the TMU context. 
• Behavioural questions such as “Tell us about the biggest conflict you have ever been involved in at work, how you handled it and what you learned” or “What experience have you had in mentoring diverse students, faculty and/or staff?” look at the candidates’ past experience as a predictor of future success. 
• Situational questions such as “With whom would you collaborate” and “How would you involve students in your research at TMU?” or more complex scenarios involving difficult work situations are another way to assess the candidate's qualifications and suitability for the position. 

A few  (google doc) sample interview questions (external link)  can be found here.  (PDF file) Additional optional questions are also available.

Careful consideration should be given to the smaller but important aspects that will affect the candidate's experience and their ultimate decision to continue in the process. These include:

• Provide an itinerary in advance, including the date and time of the interview, presentation, etc. 
• Allocate prep time and a quiet space for them to prepare.
• Introduce them to members of your department/school.
• Provide a campus tour, focusing on the interesting and unique features of the campus. 
• Provide a physical layout including where to find the washrooms, food options, etc.  
• Provide a contact in case any support is required at any point. Determine in advance if there will be a designated person escorting the candidates around for meetings and meals. 
• Ask them about their preferred pronouns.  
• Ask them about any dietary restrictions/preferences or share menu options in advance.

• Provide names and titles of expected panel members who will be interviewing. 
• Give them a sense of the types of questions they will be asked or information they should be prepared to share. 
• Ask them if they have any accommodation needs to allow them to participate in the interview and other parts of the campus visit. 
• Inform faculty candidates about the public nature of their presentation and that their CV will be made available to members of the department/school. Provide detailed information on the format of the presentation including who will be a moderator, how questions will be addressed, the time allocation for the presentation and the Q&A, etc.
• Ask them if they wish to meet with anyone from the university in particular during their visit and/or receive information about anything specific such as benefits, immigration and/or relocation support.
• Ensure consistency in the information that is provided to all candidates and the opportunities given to showcase their qualifications and meet with various individuals during the visit. 

The Collective Agreement requires the DHC to make CVs of the preferred candidates available to faculty within the school/department (except to those that may themselves be candidates for the position) for feedback.

Additionally, all members of the school/department are to be invited to the research presentation delivered by the candidate and final input is to be collected prior to a hiring recommendation being made.

The following  (google doc) template can be used/customized to collect feedback on candidates (external link) .

It is important to consider obtaining a variety of references (e.g. from students, PhD supervisors, research collaborators, industry partners, etc.) so as to obtain a complete picture of the candidate's qualifications and achievements.

The referees should not be in a conflict of interest with the candidate and be in a position to provide a fair and objective assessment. They should be knowledgeable about the candidate's qualifications and the field at large so as to provide a critical analysis of the candidate’s achievements.

Often, faculty candidates offer reference letters and hiring committees find these to be sufficient. If the committee would find it helpful, it does have the option to follow up with the same references for more information. In such a case, the candidate should be notified beforehand. Bearing in mind that each hire is a significant long term investment, it makes sense to err on the side of caution and do a little more due diligence. 

The benefit of reference calls/meetings is that they allow committees to ask probing questions and for the referee to provide additional information that may not be available in the letters.

The template ads indicate that names and contact information for three references be provided as part of the application. Some committees prefer to indicate that they will be requesting references at a later stage. In either case, reference checks are typically left towards the later part of the assessment process once the interviews/campus visits have occurred.  

Prior to contacting a referee, it is advisable to first notify the candidate that you will be doing so.

Ideally, the initial contact with the referee should be made in writing with a request to schedule a time to speak about the candidate’s application. In the same request, some additional information such as the following should be shared: 

• A copy of the job advertisement
• A link to the department’s website and relevant material such as the academic plan, strategic priorities, important work underway, etc. 
• Additional information on the department including the size of the department, student population, the programs offered, etc. 
• A link to TMU’s strategic vision and academic plan, with an emphasis on the values. 
• A general sense of what you would like to know more about relative to the candidate’s qualifications. 

The reference call/meeting should be conversational and you should take the opportunity to probe on the response given, as appropriate.

Committees are advised to give careful consideration to the requirements of the position, what information has been collected that needs to be verified and what information either needs to be clarified or sought.

Remember to stay on track by collecting facts and concrete and relevant examples that can help in the assessment of the candidate and to take detailed notes, including the date of the conversation and names of those participating on the call.

A  (google doc) few sample questions are provided in this template (external link) , which can be used/customized to collect and document the additional information acquired from references. Keep in mind that the referees, despite best intentions, will also be prone to unconscious bias that can influence the references given, whether in writing or verbally. See  (PDF file) Avoiding Unintended Gender Bias in Letters of Recommendation for further insights.