Contents Introduction TOACS: Format Developing TOACS stations Organizing TOACS examination Scoring Procedure Areas of Clinical Competence

INTRODUCTION

An extensive review of the fellowship part II examination of the College of Physicians & Surgeons Pakistan (CPSP) was conducted by the Department of Medical Education of the CPSP in collaboration with international experts in medical education. It was recommended that changes be made in the existing methodology adopted by the CPSP in order to produce consultants who would be able to deliver relevant and affordable health care to the people of Pakistan.

 This has led to the development of Task-Orientated Assessment of Clinical Skills (TOACS) as a component of the FCPS II examination.  In TOACS, the candidate will be assessed on tasks designed to demonstrate the desired clinical skills. This test will sample a large domain of clinical skills, patients and clinical tasks and hence is hoped to be more reliable. TOACS is developed on the basis that consultants solve problems by applying both clinical reasoning skills and procedural skills to solve a problem or arrive at a conclusion, hence in this examination both these essential attributes will be tested.

 Clinical competencies that can be tested in TOACS are listed later in the booklet. Interpersonal skills, presentation skills, research skills, reflectiveness and self appraisal are highly important skills in postgraduate education but are not assessed and hence do not form an important part of the learning that takes place. It is hoped that TOACS will increase the emphasis on these and have a positive educational impact.

 In the subsequent paragraphs details regarding developing TOACS stations, conducting and scoring TOACS are given.  It is expected that examiners will be able to:

1.  understand the rationale for including TOACS in the Fellowship exam

2. develop TOACS stations for inclusion in FCPS II clinical exam

3. conduct TOACS during training as well as at the end of training

4. score the candidates and make defensible pass fail decisions

Top

TOACS Format

 Tasks-Orientated Assessment of Clinical Skills (TOACS) will be fully introduced in the November 2001 diet of FCPS II examinations and will replace the OSCE which is currently being practiced in four disciplines including medicine, surgery, obstetrics and gynecology and pediatrics. It will be introduced in other disciplines in 2002.

 TOACS will comprise of 12-15 stations of eight minutes each with a change time of one minute for the candidate to move from one station to the other.  The stations would either have an examiner, a patient or both. Structured clinical tasks will be placed on each station. The performance of each candidate will be assessed by the examiners on a global rating scale or the candidates will have to submit written responses to short answer questions in the response sheet. 

 There will be two types of stations: static and interactive. On static stations the candidate will be presented with a patient data, clinical problem or a research study and will be asked to give written responses to questions asked.  In the interactive stations the candidate will have to perform a procedure for example taking history, performing clinical examination, computer applications, clinical/surgical procedures, counseling, assembling an instrument etc. One examiner will be present at each such station who will either rate the performance of the candidate or ask questions testing the reasoning and problem-solving skills.

 Top

Developing Toacs Stations

 Each TOACS station has to be prepared on a standardized format with its description, instructions and procedures given in detail.  These details will enable each station to be clearly identified, to be set up and run in the manner intended, and to allow for scoring of candidates to be done in a systematic and standardized way.  The information that is required for each station is indicated in the sections below. Sample sheets are annexed.  

This sheet contains information about the test station and allows it to be clearly identifiable as part of a TOACS station bank. Its contents are:

• Station title

• Objectives to be assessed

• Area/ topic covered

• Type of station (interactive / static)

• Resource requirements (patient, equipment, clinical photographs, laboratory data, x-rays etc.)

• Names of examiners developing the station

• Date that station was developed

• Dates on which station was used in previous examinations

This section should give a very detailed description of the station, including the clinical task or skills that will be assessed. Of considerable importance are the key features of the task identifying the aspects of a candidate’s performance on the station, which should be demonstrated if the task is to be completed satisfactorily.  Its contents should include

• Station title

• Objectives to be assessed

• Area/ topic covered

• Skills/clinical task to be assessed

• Key features of clinical task

The procedure instructions are to be given, again in detail, so that the station can be implemented in a standardized manner on repeated occasions.  This sheet should include all the instructions to examiners on how to introduce the task to candidates and how to manage the time allocated to each component. These instructions must be written down in a way so as to ensure their presentation in a similar fashion to each candidate. The instruction must include:

• Station title

• Introduction of the station and task to the candidate

• Instructions to candidates on how to proceed with the clinical task

• Indicative time allocation within the station that is time allotted to the different components within that station

• Prompt questions to be used i.e. the questions, which will be asked by the examiners to assess the key features of that station

• Ways in which examiners will be best able to observe or interact with candidates

• Any other pertinent information necessary to ensure that the station runs smoothly

If standardized patients are being used the instructions to them should be written for training the person in giving the required answers and in responding to physical examination in a consistent and standardized manner.  The patient has to be advised on how to react to expected situations, questions and statements, but also know the bounds within which they may respond to unanticipated situations.

This is for the examiner to score each candidate in a standardized and systematic manner through interaction and/or observation.  The salient feature of this type of exam is that it elicits examinee’s response only in relation to the most relevant elements in the resolution of each problem, that is its “Key Features”. The operational definition of a key feature contains one or a number of the following characteristics:

• the critical or essential steps in the resolution of the problem,

• the elements most likely to lead to an error in the resolution of a problem by the candidate taking the examination, or

• the difficult aspects of the identification and management of the problem in practice.

The type of scoring of the candidate’s performance is known as global scoring, where the impressions of the performance in sections of the clinical task are assigned a score depending on their completeness and relevance according to the defined key features.  These scores are entered on the structured scoring scale developed in the station plan and are made available to the Chief Examiner at the end of the examination.

 The instruction must include:

• Station title

• Structured scoring scale including key features

• Weightings of station components

• Indication of acceptable score for candidates reaching a minimally acceptable standard of performance

• space for recording the assigned score

This is for the candidates and contains information on the procedure and the task that the candidate has to perform on the station.  The task should be clearly worded and without any ambiguity.  

This sheet would contain the questions of which written answers are needed from the candidate.  The questions should not be testing factual recall as these can be tested in a theory examination. Short answer questions are best suited for this type of examination.  A panel of examiners should agree upon the keys to the questions prior to its administration and the keys submitted to the examination department along with the other sheets for that specific station.

Top

Organizing Toacs Examination

The toacs examination physical setting

As a clinical examination based on a series of realistic clinical tasks, it would be appropriate to provide a setting for the TOACS examination that minimised interference from extraneous sources.  The room(s), in which TOACS is set up should only be accessible to persons involved in the examination.

Preparation before the toacs

• Form a group comprising of subject experts and a medical educationist.

• Keep the objectives of training and the examination and a table of specifications in front of you

• Develop TOACS stations according to the required framework

• Review the stations in the light of the objectives and the table of specifications to ensure their consistency and validity with the competencies required

• Make a list of staff needed for a smoothly running TOACS

• Prepare guidelines for examiners and candidates

• Ensure availability of examiners, patients and other required examination material

• Draw a plan showing the layout of the TOACS stations

• Ensure the availability of the venue for the day of the examination

• Ensure that the stations are all set a day before the TOACS starts (tables, chairs, couches, screens, bell, stopwatches etc.)

It will be the responsibility of the convenor to:

• Arrive well before time

• Re-check the stations

• Designate a timekeeper at a central place

• Distribute plans to invigilators and examiners

• Meet with the examiners and brief them about the plan of the day

• Explain the whole procedure in detail to the candidates

• Ensure that the examination runs smoothly and on time

• Attend to any irregularities that may arise

• Collect all candidate score sheets from all stations

• Note and record any situations that might have influenced candidate performance

• Ensure score sheets are conveyed to the Examinations Department with appropriate security

 Top

Scoring Procedure

fractions or decimal parts are to be given.  The aggregate mark for all stations will then be regarded as the total raw score achieved by each candidate with each station contributing equally to that total score.

As each station will be scored by only one examiner it is important that examiners on all stations are well trained on the construction, implementation and scoring of the station for which they are responsible.  Further, when a TOACS station is to be repeated at more than one time or in more than one site it is essential that all examiners responsible for the same station are trained and calibrated together to ensure consistency of standards of scoring.

A scoring schema or template is to be established for each TOACS station consistent with the objectives and skills being assessed.  The approach to be used for scoring will be a global approach for each significant component of the clinical task that each candidate is to perform.  As opposed to a very detailed checklist approach to scoring, global scoring allows for qualitative judgement of each candidate’s performance.  The scoring will take into account the systematic approach to the clinical task, its appropriateness for the clinical situation and the skill with which it is performed.  It will take into account such features as the ability to justify, synthesise and prioritise findings and the appropriateness of the communication and interpersonal skills for the patient, including any cultural or ethical considerations.

Examiners are to agree in advance on the key features necessary for a candidate to reach a level of performance that would be considered to be a borderline but minimally acceptable level of performance in each component and overall. The acceptable score for each component can be calculated considering the percentage of competence that is considered essential (for that component) by the examiners to work safely with patients.  This will then be converted to a score by dividing the percentage with the component score.

The sum of these minimally acceptable acores would indicate a level of performance that might be expected from a candidate who completes all segments of the TOACS station successfully.  This does not, however, in itself necessarily indicate a score that must be attained for a candidate to be regarded as having achieved an acceptable standard on that station.

  Top

Areas of Clinical Competence

The following clinical competencies are essential to be acquired in the process of becoming a physician. Research over the time has established that most of these competencies are highly case specific. That is, one’s competence is highly variable across cases due to variable experience, interest, ability, knowledge, etc.  

The application of clinical and basic science knowledge in order to diagnose and manage a patient’s medical problems.

The technical skills involved in interviewing a patient and conducting a satisfactory physical examination.  Includes knowing all the questions and physical examination maneuvers in the physician’s repertoire.

The ability to perform an appropriate focused history and physical examination giving information about a patient’s chief complaint and pertinent demographic information establishing facts.

Recognition of significant findings established through questions (history), inspection, palpation, Auscultation, and percussion.

The ability to integrate information about the patient and arrive at a reasonable differential diagnosis 

Systematic choice of laboratory tests and other diagnostic investigations.

Ability to perform diagnostic procedure.

Integrating the information in a most plausible problem.

The ability to perform a technical procedure with causing none or minimal discomfort to the patient

The ability to plan and carry out an appropriate approach to manage the patient’s Health related problem.

The ability to communicate clearly and effectively with patients and colleagues in both written and spoken language.

The ability to educate the patient effectively regarding the nature of the medical problem, the plan for dealing with the problem and the patient’s role in management.

The ability to interact and work effectively with patients and other health professionals. This includes key personal traits as organization, dependability, and ethical conduct.

  Top

The College | President's Message | Departments | Publications | Training Programs | Forms and Feedback | Contact Us | Links |   News | Events