Constructing valid classroom reading test tasks ( multiple choice questions ) for university students

The paper contains the materials of a workshop aimed to train teachers in constructing and validating reading test tasks in university classroom. Initially, it offers teachers to check their assessment literacy by doing a test. The author further presents brief theoretical input on types of validity relevant for classroom based assessment, alternates it with tasks for teachers to do independently/in pairs/teams, offers tips to construct valid multiple choice questions and takeaway materials to guide teachers in their classroom practice of assessing students’ reading skills. List of recommended reading and keys to tasks are provided.


O. G. Kvasova Taras Shevchenko National University of Kyiv
W ith the introduction of ECTS in tertiary schools in Ukraine, the teachers of foreign languages are required to write and administer tests as summative assessment more and more frequently. Two crucial questions arise in this connection: how well are Ukrainian university teachers of English prepared for constructing and administering such tests? Can tests written by practitioners with no proper training in the field of language testing and assessment serve a valid tool to measure students' achievements in learning?
The surveys of Ukrainian university teachers' assessment literacy carried out in 2010and 2013 [3] showed that 1) mainstream teachers predominantly write test tasks to assess language in use (lexis and grammar) and reading, 2) the most frequently employed testing format is multiple choice questions MCQ, and 3) teachers are concerned with the issues of constructing valid and reliable tests for classroom use. These conceptual areas laid the basis for the development of training module "Constructing and validating classroom reading test tasks".The goal of this publication is to provide university teachers of English with the brief theoretical input, major tasks, tips and takeaway materials for further use in the classroom. These constituents of module's Part 1("Constructing classroom reading test tasks") are presented below.It is desirable that readers do all tasks in consecutive order, check their performance with the keys provided at the end of the article, do additional reading of recommended literature. To begin with, check how testing wise you are. Level B1 Straightforward factual texts on subjects related to my field of interest. Everyday material, e.g. letters, brochures and short official documents. Straightforward newspaper articles on familiar subjects and descriptions of events. Clearly written argumentative texts.Personal letters expressing feelings and wishes.Clearly written, straightforward instructions for a piece of equipment. Straightforward factual language, clearly written general argumentation (but not details). Find general information I need in everyday material. Locate specific info by searching one long or several different texts. Ability to identify main conclusions and follow argument restricted to straightforward texts.

Level C1
Wide range of long, complex texts from social, professional or academic life.Complex instructions on a new unfamiliar machine or procedure outside my area. In real life situations people generally resort to two major types of reading expeditious reading and careful reading.These skills/subskills are targeted at while constructing reading tests.
Expeditious reading includes skills of A skimming to: obtain main ideas and discourse topic quickly and efficiently; establish quickly the structure of a text; decide the relevance of a text (or part of a text) to their needs B search reading to quickly find information on a predetermined topic C scanning to quickly find: specific words or phrases; specific names figures, percentages; specific items in an index

BRAINSTORMING:
What testing format appears "by far the commonest way of assessing reading"? Why? Do you agree that "it is inadequate to measure the uderstanding of text by only one method"? Why/why not? How far do you agree with the statement "The construction of MCQ is a very skilled and time consuming business"? Give your reasoning. What are advantages and disadvantages of writing/ administering MCQ for a) teacher testers, b) testee students?

THEORETICAL INPUT: MCQ structure A multiple choice question consists of a) a stem and b) responses or options.
A stem can be formulated as a question Where did John go? or a statement with a gap to fill John … to the store (see other variations of a stem in further activities).
There are normally four options of responses in reading MCQ: one is the correct answer and the rest three are distractors. All distractors should look plausible and, with the exception of the incorrect option, relate to the context.

TIPS:How to write good MCQ
In stems, questions/statements should be clearly formulated.The language of the stem and options should not exceed the difficulty of the input text. Use of metalanguage is undesirable unless it is the target of testing.
How to construct MCQ options All options including the incorrect answer should be plausible.
A correct. Unambiguous (just right) answer B correct but too broad, general C correct but too specific, focuses on one detail D incorrect. Looking plausible but the idea is not developed in the paragraph.
This pattern is only exemplary. It is a good rule to rarely put the correct answer first, however, it should not always go third or fourth.

Which of the people mentioned has recently died and is to be remembered in a concert? a) Christopher van Kampen b) Martin Brabbins c) Alexander Raskatov d) Oleg Kagan
The concert given by the Nash Ensemble on 5 October as part of the South Bank's Haydn Festival will be dedicated to the memory of Christopher van Kampen. On 13 October, with the viola player Paul Silverthorne and the London Sinfonietta under Martin Brabbins, he was due to give the world premiere of Alexander Raskatov's double concerto 'Misere in memoriam Oleg Kagan'. This concert will now also be dedicated to his memory.

5.
A spokeswoman for the RSPCA (the Royal Society for the Prevention of Cruelty to Animals), Jo Crozier, said: "If it hadn't been for DNA testing, we would never have been able to prove that it was these men who killed the animal, even though they were found only 600 yards from the body. All we would have had was circumstantial evidence and that wouldn't have been enough to convict them." Who or what helped the RSPCA to have the men convicted? a) Jo Crozier b) Results of DNA testing c) Where the men were found d) Circumstantial evidence

7.
Socialists defended a plan to cut France's work week to 35 hours after President Jacques Chirac criticised the proposal, and a poll said that less than half the French supported it. At the close of a job conference last week, Prime Minister Lionel Jospin said he intended to introduce the 35 hour week by 2000 as a way to create new jobs and reduce the country's high unemployment rate. Many French employers oppose the plan, which they say would increase costs and would not deliver on promises to reduce the 12.5 per cent jobless rate. Who supports the plan to reduce France's working week? a) the President b) most of the people polled c) the Prime Minister d) many French employers 8. Suzanne Somers is always going to struggle to lose the image of the superficial character she played in a recent TV drama. Try as she might, she is not particularly convincing here as an ace journalist who sets out to uncover the mystery behind the death of seven people in a night club. Ed Begley and Michael Nouri offer satisfactory support, but the direction is strictly routine, and the same goes for the script.
What is the writer's opinion of the film? a) Some of the acting is excellent. b) The film is well directed. c) The leading performance is weak. d) The script is well written.

9.
The new digital cameras are great fun and very easy to use. They let you review your pictures the moment you take them, so you can re shoot right away if you are not satisfied. But remember, a digital camera is just a computer XXXX. It's not a replacement for your ordinary camera.
What is the meaning of the missing word (XXXX) in the text?
a) "something that is of poor quality" b) "an item that is not essential, something extra" c) "something expensive but good value for money" d) "a fashion which always remains popular" 10.Which sentence best fills the gap in the text? "I do think this course should be fun," he said. "And people should get out of it what they want. Some come here wanting to be very serious and even a bit competitive about work. And that's fine. Others are very hesitant and find that a bit of humour can relax them.
I've taught high court judges and dustmen and the mix is part of the fun."

11.Read the text and decide how you would describe it.
With our travel agency, the holiday you book is the holiday you get. If you arrive and find we've failed to live up to our promises, let us know what the problem is within one day of your arrival. We'll spend 24 hours doing everything possible to sort the problem out. In the unlikely event that we can't resolve your problem and make you happy within 24 hours, we'll fly you home and give you your money back. It is… a) a commitment b) an appeal c) a warning d) a vow 12.Read the text and then predict the sentence that comes next. The original cottage gardens were there for only one purpose. They were essential to feed the working man's family. Until the plague in 1348, labourers worked for their masters in return for the rent of their cottages. Mostly they had to be self sufficient, growing what they could and keeping animals. To check understanding of explicitly stated information According to the paragraph (AtP), which of the following is true of N? The author's description of N mentions which of the following? AtP, X occurred because … . AtP, N did Y because … AcP, why did N did Y?
The author mentions all of the following except … . What does the author say about N? To check the skill to make inferences Which of the following can be inferred about N? The author implies that N … . Which of the following can be inferred from para X? It can be inferred from the text that … . The text supports which of the following conclusions? a) They tend to learn more. b) You get all sorts on these courses. c) I prefer the serious student. d) We only accept professional people. a) Therefore, many of them were able to negotiate wages to buy some essentials. b) Anything they "bought" was obtained by the age old system of bartering. c) The plague wiped out about a third of the population, so labour was hard to find. d) Still, it wasn't much and the cottage garden was always vital to their existence. To check the skill of reference The word "x" in the text refers to … . The word/phrase "x" in line 17 refers to … .

To check understanding of rhetorical purpose
The author discusses X in para 2 in order to … Why does the author mention X? The author uses X as an example of … . Why does the author use the word "X" when mentioning Y? To check understanding of essential/key vocabulary The word "x" in the text is closest to the meaning to … . In stating X, the author means that … . In line "X", the word "Y" could best be replaced by … .

To check understanding of main idea(s)
What is the author's main point?   2) There are grammar mistakes in the stem and in option c.
3) The stem is badly punctuated. 4) The wording of options is clumsy. CONCLUSIONS: the workshop presented in the paper was conducted by its author in several universities in Ukraine (Kyiv, Kharkiv, Cherkasy) and highly evaluated by its participants. In their feedback questionnaires the teachers noted that they had received a comprehensive overview of principles lying behind MCQ construction. The enthusiastic teachers received a post workshop task to construct MCQs of their own teaming up with those working in similar academic context (same syllabus, same course book, same year of study). The syndicates made up of 2 4 members developed one MCQ test task each, tried them out, modified as suggested and further administered the test tasks in their student groups. In many cases teachers administered testee questionnaire to collect feedback on clarity and efficiency of their self made tasks which helped them determine the tasks' strengths and weaknesses. The data analysed proved the overall effectiveness of the methods of constructing MCQs for classroom use. The PROSPECTS of further research lie in developing methods of constructing valid test tasks of other testing formats (gap fill, short answer questions, etc.) aimed at measuring other reading and/or listening skills.