Download Hierarchical Approach to Modifying MCQs in Computer-Based Instruction and more Study Guides, Projects, Research Construction in PDF only on Docsity!
DOCUMENT RESUME
ED 326 201 IR^014
AUTHOR Clariana,^ Roy^ B. TITLE Modifying^ Multiple-Choice^ Questions^ in^ Computer-Based Instruction. PUB DATE 16 Oct 90 NOTE 10p. PUB TYPE Viewpoints^ (120)^ --^ Reports Research/Technical (143) --
EDRS PRICE MF01/PC01^ Plus^ Postage. DESCRIPTORS *Comprehension;^ *Computer^ Assisted^ Testing; Difficulty Level; *Yeasurement Techniques; *Multiple Choice Tests; Secondary Education; *Test Construction; *Test Items IDENTIFIERS Paraphrase
ABSTRACT Research has shown that multiple-choice questions formed by transforming or paraphrasing a reading passage provide a measure of student comprehension. It is argued that similar transformation and paraphrasing of lesson questions is an appropriate way to form parallel multiple-choice items to be used as a posttest measure of student comprehension. Four parallel items may be derived from a lesson: (1) an item that is neither transformed nor paraphrased, thus testing simple rote memory; (2)^ an^ item^ that^ is transformed only; (3) an item that is paraphrased only; and (4) a compound item that is both transformed and paraphrased. These four items produce a hierarchy of difficulty from easy (rote) to difficult (compound), depending on the extent to which an item is altered from the original lesson question. In the three studies cited in this paper, posttest performance for the four types of items followed the hypothesized hierarchy of difficulty, and this is viewed as evidence that many of the cognitive skills underlying comprehension may also be hierarchically related. It is concluded that the formation of test questions from lesson items provides a practical way for designers to utilize multiple-choice items for instruction and also for posttest assessment, (DB)
,
- Reproductions^ supplied^ by^ EDRS^ are^ the^ best^ that^ can^ be^ made^ *
- from^ the^ original^ document.^ *
UIDEPARTMENTOFEDUCATION °IkeorEducetonalResearchendImprovement EDUCATIONALRESOURCESINFORMATION CENTERIERICI )111Th.sdocumenthasbeen.eiroducedes receumdfromthepotionOrOrganlietlon ondiphng rPAP,chineshavebeenmane 'productionduality
dcuntiotiiitpoognIonsstated mentdonotneCeSSerrlyrepresentbbocial.nlbs()PC6. OERIposdanorpohcy
ModifyingMultiple-ChoiceQuestionsinComputer-BasedInstruction
byRoyB.Clariana
October16,
Submittedto:
SyracuseUniversity/SchoolofEducation
ERICClearinghouseonInformationResources 030HuntingtonHall Syracuse,NewYork13244- (315)443-
EditorialContact:
Dr.RoyB.Clariana
Training/EvaluationSpecialist
EG&G,RockyFlatsPlant Building P.O.Box Golden,CO80402- (303)744-
RunningHead:Modifyingmultiple-choicequestions
'PERMISSIONTOREPRODUCETHIS MATERIALHASBEENGRANTEDBY
Roy B. Clariana
TOTHEEDUCATIONALRESOURCES INFORMATIONCENTER(ERIC)"
Modifyingmultiple-choicequestions 2
ModifyingMultiple-Chcrs,QuestionsinComputer-BasedInstruction
October
Multiple-choicequestions(i.e.items)areanunusuallyconvenientandpowerful
instructionalformatincomputer-basedlessonsthatprovideinformationtothelearner
throughthestem,correctresponseoption,andincorrectoptions.Theyallowfor
learner-lessoninteractionthroughlearnerovertresponseandprogram-provided
feedback.Importantly,thenumberofansweroptionsisfinite,andindividualization
viabranchingispossiblebaseduponthelearners'correctandincorrectresponses.
Additionally,multiple-choicetestscanbedeliveredandmarkedbycomputerthus
increasingtheefficiencyoftheinstructionalprocess.
However,ifmultiple-choiceitemsareusedduringinstruction,itwouldbe
inappropriatetousetheexactsameitemsforpost-lessonassessment.Suchatest
wouldonlymeasurerotelearning.Inordertotakeadvantageoftheconvenienceof
multiple-choiceitems,oneoften-usedprocedureistoutilizemultiple-choiceitems
duringtheiessonandthenuseparallelconstructed-responseitemsontheposttest.
Thismethodisstraightforwardandeffective.Oneminorproblemisthat
computer-markingofconstructed-responsetypeitems(i.e.fillintheblank)is
sometimesproblematic.Learnersmayknowthecorrectanswerbutmaymissanitem
duetosyntax,spelling,orotherunexpectedvariables.Oftenconstructed-response
posttestsmustberemarkedbyhand.
Thepurposeofthisstudyistodescribeamethodforadaptingmultiple-choice
itemssothatparallelformsofthesameitemmaybeusedduringthelessonfor
/'/WW1=INIVINMPlinf
Modifyingmultiple-choicequestions 3
instructionandduringtheposttestforassessmentDatafromthreestudiesareshown
tosupportthisitemmodificationmethod.
TransformationalAnalysis
Thetermtransformationalanalysis(Bormuth,1970)appliestotheprocessof
generatingtestitemsfrominstructionaltext(seeFigure1).Anderson(1972)
describedtwomethodsfordevelopingtestitemsfromEndingpassasesinorderto
assescomprehension.Thesemethodsincludeparaphrasingandtransforming.
EmitLasinginvolveschangingthewordsfromthereadingpassagebyreplacing
wordswithequivalentwords(i.e.semanticsubstitute),whilemaintainingthemeaning
ofthereadingpassage.Transforming,inthisinstance,involvesreversingtheoriginal
orderofthesentencesubjectandpredicateelementsThenounphraseinthesubject
ofthesentencebecomesthecorrectansweroptionoftheparallelposttestitem;andthe
nounphraseinthepredicatebecomesthestemoftheparallelposttestitem(i.e.passive
transform).
ReadingPassages PosttestItems
tra_mforn__L+.
processes
subjectdeletionitems
echoitems
tagitems
roteitems
transformitems
semanticsubstituteitems
...others
Figure1.Modifyingposttestitemsfromreadingpassages(Bormuth,1970).
5
Modifyingmultiple-choicequestions 5
(i.e.notparaphrasedandnottransformed)itemsobtainedthehighestscoreswiththe
transformedandparaphraseditemsnext.Thecombinationoftransformand
paraphrasemethodsproducedthemostdifficultitems.
Na
EAraphavad
Earaphriml
N2I
TransformedTransformed
Figure2.ScoresforthefourtypesofitemsusedbyBormuthetal.(1970).
Bomiuth(1970)observed,"Itisacommonlyobservedphenomenonintest
writingthatalteringtheformofanitemhassomeeffectonitsdifficulty"(p.15).
Applyingthesetwomethodsappearstoimpactitemdifficultyandthusposttest
performanceforeachtypeofiteminalogicalway(Davey,1988).Anderson(1970)
arguesthattheseitemsprovideameasureofcomprehensionoflessonmaterials.
Thispapersuggeststhatparaphrasingandtransformingcanbeappliedtolesson
iteanaLmallAtioicading.passaus(seeFigure3).
LessonOyestions PosttestItems
Verbatim
Paraphrased
Transformed
Paraphrased&Transformed
Figure3. Modifyingposttestitemsfromlessonitems.
7
Modifyingmultiple-choicequestions 6
Thiswouldallowforthedevelopmentofposttestitemsthatparallellessonitemsand
provideameasureofcomprehension.Ifthetransformandparaphrasemethods
describedaboveareemployed,againfourtypesofparallelposttestitemswillresult.
Whataretheeffectsonposttestperformanceofthesedifferentitemtypes?Theresults
ofthreestudiesaredescribedtosupportthismethodology.
Study
Asampleof32"atrisk"eleventh-gradestudentscompletedacomputer-based
socialstudieslessoncomposedof32four-optionmultiple-choiceitemswithfeedback
whichincludedfourassociated350-wordreadingpassages.Theposttestconsistedof
32parallelfour-optionmultiple-choiceitems,halfwereverbatimtothelessonitems
andhalfwereparaphrasedfromthelessonitems.Theresultsfortheseparaphrase
posttestitemsweresimilar(i.e.compareFig=4toFigure2)tothosereportedby
Bormuthetal.(1970).Transformedquestionswerenotused.
N.
Paraphrased
&whams'
NSA Lualimmrsi Transformed
78.5% n/a
59.8% n/a
Figure4.PosttestResultsforStudy1.
Study
Asampleof36"average"tenth-gradestudentscompletedaworkbook
containingfournatural-sciencereadingpassagesand40relatedmultiple-choiceitems
.
Modifyingmultiple-choicequestions 8
Discussion
Transformingandparaphrasinglessontexttoformmultiple-choiceposttest
itemsisnotanewidea.Applyingthissamemethodtolessonitemsratherthanlesson
textallowsdesignerstoutilizemultiple-choiceitemsbothforinstructionduringthe
lessonandforassessmentduringtheposttest.Thetestitemsproducedwillobviously
parallelthelessonitemsone-to-oneintermsofcontent(i.e.instructionalobjectives)
andappeartocreateasystematicrangeofitemdifficulties.Theseresultsareviewed
asevidencethatmanyofthecognitiveskillsunderlyingcomprehension"maybe
hierarchicallyrelated"(Bormuthetal.,1970).
References
Anderson,R.C.(1972).Howtoconstnictachievementteststoassess comprehension.ReviewofEducationalResearch,42(2),145-170. Bormuth,J.R.(1970).Onthetheoryofachievementtestitems.TheUniversityof ChicagoPress. Bormuth,J.R.,Manning,J.,Carr,J.,&Pearson,D.(1970).Children's comprehensionofbetween-andwithin-sentencesyntacticstructure.Journalof
EducationalPsychology,fd,349-357. Davey,B.(1988).Factorsaffectingthedifficultyofreadingcomprehensionitems forsuccessfulandunsuccessfulreaders.JournalofExperimentalEducation, IC67-76.
Note:
ThedatareportedinStudy1waspartofapreviouslyreportedstudytitled"Two
immediatefeedbackformswithtwoconditionsofcontextualization"inthe
JournalofComputer-BasedInstruction,inpress.ThedatafromStudies2and
werepartofadissertationcompletedatMemphisStateUniversitytitled"An
experimentalstudycomparingthreeformsoffeedbackacrosstwoconditionsof
acquisition".
I
Appendix 1
_ e
END
U.S. Dept. of^ Education
Office of Education Research and Improvement (OERI)
ERIC
Date Filmed
March 29, 1991
i
