Transcript Slide 1
WJEC Examination Review (Examples of forthcoming online feedback available from the autumn term) General overview At WJEC we are always looking for ways to improve provision to our customers. We provide the following information relating to examination sessions: 1 Item level data 2 Exam paper 3 Marking scheme 4 Examiner’s report However, we want to ensure that these materials are used in the most effective way, to recognise weaknesses and improve standards within the classroom. General overview Using the ‘Facility Factor’ from the Item Level Data, it is possible to focus on those questions which caused the most problems for candidates. Recognising weaknesses Exam paper Marking scheme Examiner’s report The online examination review provision is intended to lead teachers through the process of studying the data in detail, referring to the examination paper, marking scheme and examiner’s report to evaluate candidate responses and consider the way in which these elements are taught. At the same time teachers are encouraged to compare the information presented with their own centre specific data. General overview What do you think caused the problems for the candidates? Q2 Recognising weaknesses The question The online exam review first identifies questions which appear to have caused some candidates problems. Exam paper Marking scheme Examiner’s report It then asks users to consider what candidates might have found difficult when answering the question. General overview What do you think caused the problems for the candidates? Q2 Recognising weaknesses The question A chance to mark an example answer using the mark scheme Exemplar answers Exam paper Marking scheme Examiner’s report Then users are asked to consider some examples of answers given by the candidates, and mark them using the mark scheme. General overview What do you think caused the problems for the candidates? Q2 Recognising weaknesses The question A chance to mark an example answer using the mark scheme Compare this to the examiners’ comments Exam paper Marking scheme Examiner’s report Exemplar answers Then there is an opportunity for users to compare their thoughts with examiner comments. General overview What do you think caused the problems for the candidates? Q2 Recognising weaknesses The question A chance to mark an example answer using the mark scheme Compare this with the examiner’s comments Exam paper Marking scheme Examiner’s report Have we learnt something new from this? Exemplar answers Do we have to adapt the way in which we teach this aspect of the course? An example of online feedback from the January 2013 exams A feedback module will look like this when opened. The following slides will take you through this example module, providing instructions on how it should be used and explaining the tools available. A calculator is available to use from any page by clicking the spanner icon, and then the maths symbols icon. A timer is available to use from any page by clicking the spanner icon, and then the clock icon. This might be of use if the module is being used in the classroom. Click the X to close the calculator and the clock. The clock functions are:Start Reset Stop Click the question mark icon while using the unit for more information on the tools available. Click this icon to maximise the window. Press Esc to exit the full screen mode. Click this icon to get a list of pages available. This is a quick way of navigating to the desired page. Just click the page you need. Click one of these icons to move backwards or forwards through the pages. NOTES: 1. The item level marks used in the item level analysis and reported for each candidate are those submitted by the examiner at the time of marking. They will not include any changes made as a result of a remark, neither will they include changes made as a result of errors discovered during the internal script check that takes place prior to the issue of results. 2. There is a note of caution to be sounded. If the number of candidates is relatively small, the performance of one individual may have a disproportionate effect on the performance of the group as a whole. Therefore, care must be taken when making judgement about performance where the number of candidates is small. - Pointer tool - Pen tool - Highlighter tool - Rubber tool (selective delete) - Delete tool - Palette tool (pen and highlighter colour) The pen tool allows the user to make notes on any part of the screen. Choose the colour palette at any time to change the pen colour. This can be useful if there is need to distinguish between different types of comment, or if being used by students in the classroom. The highlighter tool allows the user to highlight any part of the screen. Choose the colour palette at any time to change the highlighter colour. Click this icon to delete parts of notes or highlights made. Click this icon to delete all notes and highlights. You will be asked to confirm that you want to clear all. This part of the screen shows the item level data. An item is a questions or part question within a script. The table shows the data for all WJEC candidates attempting this paper. It can be compared with a particular centre’s data in order to evaluate the achievements of the centre’s candidates. (Item level data for individual candidates and centres is available through the WJEC secure website. See the presentation on this facility for more information). 1. Question Title Usually the question number 2. N The number of candidates attempting that question 3. Mean (average) mark attained (Mean) The mean score is calculated by adding up the individual candidate scores and dividing by the total number of candidates. If all candidates perform well on a particular item, the mean score will be close to the maximum mark. Conversely, if candidates as a whole perform poorly on the item there will be a large difference between the mean score and the maximum mark. A simple comparison of the mean marks will identify those items that contribute significantly to the overall performance of the candidates. However, because the maximum mark may not be the same for each item, a comparison of the means provides only a partial indication of candidate performance. Equal means does not necessarily imply equal performance. For questions with different maximum marks, the facility factor should be used to compare performance. 4. Standard Deviation of the marks attained (S.D.) The standard deviation measures the spread of the data about the mean score. The larger the standard deviation is, the more dispersed (or less consistent) the candidate performances are for that item. An increase in the standard deviation points to increased diversity amongst candidates, or to a more discriminating paper, as the marks are more dispersed about the centre. By contrast a decrease in the standard deviation would suggest more homogeneity amongst the candidates, or a less discriminating paper, as candidate marks are more clustered about the centre. 5. The Max Mark This is the maximum mark for a particular question. 6. The facility factor (F.F.) The facility factor for an item expresses the mean mark as a percentage of the maximum mark (Max. Mark) and is a measure of the accessibility of the item. If the mean mark obtained by candidates is close to the maximum mark, the facility factor will be close to 100 per cent and the item would be considered to be very accessible. If on the other hand the mean mark is low when compared with the maximum score, the facility factor will be small and the item considered less accessible to candidates. 7. Percentage of candidates who attempted the item (Attempt %) For each item the table shows the number (N) and percentage of candidates who attempted the question. When comparing items on this measure it is important to consider the order in which the items appear on the paper. If the total time available for a paper is limited, there is the possibility of some candidates running out of time. This may result in those items towards the end of the paper having a deflated figure on this measure. If the time allocated to the paper is not considered to be a significant factor, a low percentage may indicate issues of accessibility. Where candidates have a choice of question the statistics evidence candidate preferences, but will also be influenced by the teaching policy within centres. The pointer can be moved to wherever needed on the graph. Click this icon to see the ‘Facility Factor’ information in graphical form. To use the information in the unit as printed sheets or for other purposes, click this icon to open a new window containing the documents in a zip file. Select the ‘Extract all files’ to save the information displayed to folder of your choice The ‘Compressed (zipped) Folders Extraction Wizard’ will open – click Next> You will be asked to choose where you would like the folder to be placed (use the Browse button to select your preferred location), then click Next>. You will now see the ‘Extraction Complete’ message. If the ‘Show extracted files’ box is ticked (as shown) when Finish is clicked, the folder containing all the files will open automatically. Clicking the question mark icon will display questions designed to encourage the user to think about the data, and compare it with his/her own centre based data which may be found on the WJEC secure website. (A presentation on this facility is available here.) Scroll through questions by clicking the arrows. Clicking the ‘Show suggestions’ dialogue will highlight the questions the feedback module focuses on. A similar facility is also available in graph view. Clicking on any one of the highlighted sections (in this case Q3, Q6 and Q8) will take the user to a new screen where the actual question can be looked at in detail. For example, if Q3 is chosen… Generic tool set. Navigation within the module. Download the materials. Pen tool for making notes, highlighting etc. There are some familiar tools here, and some extra ones to help with use of the module. Scroll bar to move through larger questions. The question may be annotated, using the pen or highlighter tool. You are able to scroll up or down the question if it is too long to fit on the screen. Click here to view the item level data again. There are two different candidate answers available to view for this question. Click one to view the candidate response. Click here to get a ‘floating’ mark scheme. Close by clicking the ‘x’ at the top right corner. Candidate 1 answer: Click here to go back to the question. Click here to view the item level data for this question. Click here to see the examiner’s comments. Click here to see the marks awarded. Click here to get a ‘floating’ mark scheme. Close by clicking the ‘x’ at the top right corner. Examiner comments are shown in speech bubbles. Responses for some subjects will have audio comments by the examiner. Examples GCE Electronics ET1 GCE Electronics ET4 GCSE English Literature GCSE Hospitality and Catering GCSE Media Studies Click any of the buttons to open examples of other feedback modules. (You must have online access to view these exemplars). NB: These modules relate to examination sessions previous to summer 2013. Advantages Teachers teaching more than one subject have access to the relevant subject materials at any time. These are materials for heads of department to use during departmental training. Leads practitioners through the process of data evaluation, recognition of problem areas and encourages the thought process of evaluating the way some aspects are taught. Over time, this bank of materials will grow and provide a comprehensive store of support files for teachers, eventually linking to resources and examples of good practice. In the hands of a good teacher, these materials could be used with students in the classroom as assessment for learning tools. The materials provide information for NQTs or others teaching subjects for the first time. Units can be built into a bank of information tackling issues on a similar aspect in a number of subjects. Low carbon footprint, as teachers can access the materials at anytime from anywhere. Materials are available totally free of charge. The units can provide a basis for discussion, and the sharing of ideas during regional meetings. All staff will have access to the materials. A dedicated website is being developed to host the materials that will be updated regularly as more units become available. Although the units are available as interactive web based files, all pages can be printed and used as paper-based activities. We hope you have found this presentation useful. Online review of the summer 2013 exams will be available in the autumn. If you have any questions, please e-mail us on [email protected].