Testing our Standards — Testing our Teaching: Illinois State University’s General Education vs. the Advanced Placement (AP) Examinations

By Elizabeth Doorn, Former Teaching Assistant, Department of HistoryDr. Patrick O’Sullivan, Director, Center for Teaching, Learning & Technology at California Polytechnic State University; and Dr. Stewart Winger, Associate Professor, Department of HistoryElizabeth Doorn graduated in 2013 and currently teaches social studies at El Quarto Año High School in Chicago.

Originally published in Progressive Measures, Spring 2014, Volume 9, Issue 2. 

The purpose of a liberal arts education is to provide students with the skills and knowledge to become active citizens and more fully realized human beings.  Having been exposed to richer worlds of thought, liberally educated students should better understand their place in the world and should be better empowered to shape that world.  The student of music will be familiar with the general theory of relativity, enhancing both the understanding of music and of life.  The student of chemistry will be able to communicate effectively in the spoken and written word, perhaps even in more than one tongue.  The student of agriculture will appreciate the Mona Lisa, perhaps inspiring a renaissance in the taste of food, as well as a tastier personal life.  How can students claim real expertise in any field (music, chemistry, agriculture, etc.) without an understanding of the connections and contributions to these other facets of humanity?  According to the University’s General Education Task Force (2012), “Illinois State University’s General Education Program prepares students to be globally engaged citizens who seek knowledge, appreciate diversity, think critically, communicate effectively, act responsibly and work collaboratively.”  In an attempt to fulfill this mission, 42 credit hours, or about one third of an undergraduate student’s total coursework, is dedicated to general education coursework at ISU.

But with the decline of the public sector in the United States since the mid-1980’s, almost all universities have become dependent on tuition revenue.  And generally speaking, no one department or college on campus has as its primary calling the duty to safeguard this vital and, dare we say sacred, mission.  Lacking powerful enough defenders in the give and take of bureaucratic life, general education has often become a source of revenue to support other priorities.  In addition, the desire for tuition revenue has led to a scramble for transfer students.  Credit is extended promiscuously to any and all community colleges without investigation because the cost to any one university of unilaterally rejecting potentially subpar transfer credit would be too great.

But does this system actually work?  Is there a downside to using general education courses primarily as revenue stream while, somewhat contradictorily, teaching our students to ‘get their gen ed credits out of the way’ as cheaply as possible even if that means taking a short summer course at a community college?  Are our students learning much from general education experiences in which intensively monitored reading and writing are often precluded by class sizes that bring in more revenue?  Several recent national studies suggest probably not2.

Turning to the specific context at Illinois State University, who sets the standards for our General Education program, and how are those standards measured?  Do certain approaches work better than others?  Do courses at ISU outperform community college courses which cost about a third as much per credit hour?  It is a vast and multifaceted subject matter, and so in this project we restricted our scope to two general education courses: POL 106 (one of five Individuals and Civic Life courses) and HIS 135 (one of seven United States Traditions courses).  In the fall of 2012, we compared them to their respective course-equivalent Advanced Placement (AP) examinations to see whether and what our students were learning against that widely accepted measure.

The Courses

The courses being evaluated in this study were HIS 135 and POL 106.  HIS 135 (History of the United States to 1865) is a general education course required for all history majors that is also available to other majors as an elective.  The Illinois State University (2014) Undergraduate Catalog defines HIS 135 as “Political, economic, social, and cultural developments from the colonial period to the Civil War” (p. 178).  Different faculty members teach this course each semester, and the course structure is largely determined by the instructor.  The fall 2012 HIS 135 course taught by Dr. Stewart Winger was designed to engage students in historical writing, analysis, and discussion, while also ‘covering’ the broad time period.  In no way was the course altered to ‘teach to the AP exam.’  Dr. Winger remained scrupulously aloof from the details of AP, leaving those aspects to researcher Elizabeth Doorn.

Following a model that Dr. Winger has used in previous classes, students wrote analyses of brief primary and secondary historical sources in response to a series of prompts before nearly every class.  The heavy grading load of twice-weekly assignments was handled by a graduate assistant and two undergraduate teaching assistants who had taken Dr. Winger’s courses before and who had internalized and enthusiastically accepted his grading rubrics.  Only occasionally was a quiz, in-class response, or in-class group writing assignment substituted.

These twice-weekly writing assignments helped prepare students for the three papers they wrote throughout the semester in which they demonstrated their understanding of the course content through thesis-centered historiographical writing using primary sources to evaluate the claims of other historians.  These papers of increasing length were posted to electronic peer editing groups.  Editors were required to root out basic grammatical errors under stringent penalty.  They were also required to apply the grading rubric to give and justify a provisional grade.  Students then rewrote their papers in response to the peer editing and handed in a hard copy.  These papers were graded by the graduate assistant and Dr. Winger and were  worth 60% of the course grade (10%, 20%, and 30%, respectively).

There were no formal tests or multiple choice assessments used in class.  Students did, however, engage in an online reading comprehension activity called LearnSmart which was provided by McGraw Hill along with the shortest available textbook.  Organized in multiple-choice, short answer, and true/false questions, each Sunday night students responded to online questions that coincided with each textbook chapter.  LearnSmart does not accept incorrect answers; incorrectly answered questions return in differing formats until each question has been answered correctly.  If students miss too many questions, the module times them out, instructing them to re-read the textbook section in question.  Students cannot simply accept their C- and go home.  Textbook content was almost never referenced directly by the instructor in class.  Course time was reserved for discussion of the reading assignments, written responses, and other interpretive issues.

In contrast, POL 106, defined as a course that “Examines the sources and effects of practices and institutions of participation, influence and cleavages in United States politics” by the Illinois State University (2014, p. 208) Undergraduate Catalog, is generally structured as more traditional lecture course.  Each student attending ISU must have taken one course in the Individuals and Civic Life category, or received equivalent credit from AP or from another institution, before graduation (AP credit is accepted only for POL 106 or POL 105).  Because of this, the student enrollment per semester is much higher, and the classes are larger.  The larger class size, as well as the significantly greater percentage of non-major students enrolled (as opposed to HIS 135), makes it more difficult to take a writing-intensive approach to the material.  It is because of these differences that the two courses work well together in this study.  How does class structure and student assessment affect a student’s ability to do well on the corresponding AP exam?

Why Advanced Placement?

Advanced Placement is a program of study created by the College Board that gives high school students the opportunity to earn college credit for advanced coursework in certain fields.  College Board assesses AP students with a test designed to determine their mastery of the material as well as their readiness for more advanced college courses in the subject.  Scores on the exam range from 0 to 5.  College Board defines a score of ‘3’ as ‘qualified,’ meaning the test-taker has essentially ‘passed’ the examination with College Board’s approval and confidence that he or she has earned credit for a introductory college course on the material.  For example, in 2012 a passing score of ‘3’ for the AP United States History exam earned an incoming ISU student course credit for U.S. History overview courses (HIS 135 and HIS 136; this has recently been changed to ‘4’ or ‘5’).  The Department of Politics and Government at ISU requires a score of at least ‘4’ on the AP United States Government and Politics exam in order for a student to receive credit for -and exemption from POL 106.  In contrast, the Department of Political Science at Southern Illinois University accepts a score of ‘3’ in place of its introductory course, while the Department of Political Science at Northwestern University requires a score of ‘5.’

Despite the varying expectations for AP scores throughout the state, the exam itself remains relatively constant.  Each year, thousands of students nationwide take an AP exam.  For this reason, we chose to use the exam in an effort to examine the effectiveness of HIS 135 and POL 106.  In both AP exams (United States History and United States Politics and Government), students would have the opportunity to demonstrate their learning through both multiple choice and essay questions.  The results would hopefully help determine which teaching and assessment strategy (information and memory-based versus critical thinking and skills-based) would help students do better on the overall test.

Many important objections to the AP exams and to testing itself could be raised.  Most importantly, teachers often feel forced by their administrators to teach to the test to the detriment of what they love about their subjects and what they feel is truly most important.  Also the method AP uses to grade written responses can be gamed by those who understand the rubrics; investigator Doorn expressed frustration at not being able to reward especially creative and informed written responses that did not satisfy the narrowly tailored points in the AP rubric.  But while deeply unfortunate, none of these weighty objections invalidate the exams for the purposes of this study.  Apart from the grading rubrics, the AP exams are very impressive tests and no student can achieve a ‘5’ without real mastery of the material and understanding of the discipline.  We are not here advocating high stakes testing or teaching to the AP standards.  At this point, we are merely using the AP exams as a measuring stick for program assessment.  For this purpose, the exams are, on the merits, superb, and their validity is already accepted by most departments at most universities, including ISU.

Method

Participants

The 37 participants in this study were all current students enrolled in Dr. Winger’s fall 2012 HIS 135 course.  Eighteen of these participants had no politics and government experience, meaning they had not completed POL 106 or POL 105 at ISU or earned equivalent credit through another higher education institution or an AP exam.

Procedures

Participants were asked to take three tests throughout the semester: one AP United States Government and Politics exam, and two AP United States History exams (a pre-test and a post-test).  Dr. Winger did not choose the versions of the tests that were used or know which version would be used when planning coursework and material for HIS 135, and no modifications were made to the course materials, expectations, or methods of teaching and evaluation based on any AP exams or teaching methods.  To insure student motivation, the pre-test for the AP history and the AP government and politics exams were taken for credit as part of the homework grade.  The post-test for the AP history exam counted as the final exam, but scores were curved for the purposes of course grading.

     Government and politics. One version (1999) of the AP politics and government exam was given to the participants.  Students answered all 60 multiple-choice questions.  The original exam requires test-takers to answer four free response prompts.  Participants answered one of the four questions, which was selected using a random drawing, and their scores were weighted to provide an estimated score for each student had he or she answered all four prompts.

History. The multiple choice (MC) portion of the AP history exam, worth 50% of the overall score, consists of 80 questions.  After questions pertaining to U.S. history after 1865 were removed from the test, the condensed version that participants took consisted of 28 of the original 80 questions.  The Free Response Question (FRQ) section, worth 27.5% of the overall score, requires test takers to respond to an essay prompt.  Participants chose one of the two FRQ prompts relevant to U.S. history before 1865.  The Document Based Question (DBQ), worth 22.5% of the overall score, also requires test takers to respond to an essay prompt but includes primary source documents, several of which should be used to support the claims in the written response.  These include data charts, personal testimonies, political cartoons, and official documents among other types of sources.  The DBQ measures the test taker’s historical writing skills, as well as his or her ability to read and interpret various primary source documents.

The pre-test consisted of 28 multiple-choice questions and 2 essay response prompts, while the post-test consisted of 36 multiple-choice questions and 2 essay responses.  Each question was weighted in order to provide a final point total that could be scored on the AP scale.  At the beginning of the fall 2012 semester, participants took the course-relevant MC portion of the 2006 AP history exam.  A week later, the FRQ portion of the pre-test (2012) was distributed, and the DBQ portion of the pre-test (2011, Form B) was distributed three weeks later.  The post-test was administered on the designated date of the final exam at the end of the semester.  Participants reported to the designated testing room and were given the 1996 AP history exam, excluding the portions of the test that referred to material not covered in the course’s designated time period.  The MC portion was graded using Scantron, while the FRQ and DBQ portions were graded by investigator Doorn using the AP scoring guidelines (see http://apcentral.collegeboard.com/apc/members/exam/exam_information/2089.html  for more information).  Ms. Doorn received training in grading the exams from faculty experienced in grading for AP.

Results

Government and Politics Scores

Students with no college-level coursework in government and politics earned an average overall score of ‘2’ on the AP exam.  Students with ISU course credit (from POL 105 or POL 106) also earned an average overall score of ‘2,’ as did students with transfer course credit (see Figure 1).  Community college and ISU courses had a negligible impact on student performance against a nationally-accepted standard accepted by ISU.

On the writing section, students with no experience earned an average score of 1.56 out of 9, while students with ISU credit earned an average of 1.53.  While the difference is seemingly trivial, it speaks volumes.  Not only do students who have earned college credit fail to distinguish themselves from those who have not, they actually underperform in comparison.  Students with POL 105 or POL 106 credit marginally outscored students with no credit on the multiple-choice section with an average of 25.85 (no credit average: 24.50).  This time, the experienced group did outperform their classmates who had yet to earn course credit, but again, the two groups are nearly indistinguishable.

History Scores

Participants’ pre-test scores provided a benchmark for evaluating the course’s ability to increase student knowledge and ability in U.S. history.  As the test is scored out of 180 total points (MC = 90; FRQ = 45; DBQ = 45), each AP exam score (ranges from 1 to 5) includes a range of point values.  The Department of History at Illinois State University required a score of ‘3’ on the exam for HIS 135/HIS 136 credit and now requires a score of ‘4.’  Pre-test scores indicated an average score of 29, which falls in the overall score ‘1’ range.  This is not alarming, as the participants taking the pre-test had yet to complete the HIS 135 coursework.  On the post-test, participants scored an average of 86.24, falling in the overall score ‘3’ range (see Figure 2).

Breaking these numbers down, we can better examine in which areas participants experienced the greatest improvement.  Pre-test scores for the MC section averaged 11.3 out of 28, with a range of 6 to 15.  The MC section as a whole accounts for 50% of the final test score.  If given a final score based on only this section, pre-test participants averaged a score of ‘2.’  Post-test MC results indicated an average of 19.6 out of 36, with a range of 11 to 30.  If given a final score equivalent based only on this section, the group averaged a score of ‘3.’  Similarly, scores on the writing portions of the exam increased from the pre-test to the post-test.  However, these sections’ scores increased more dramatically.  Participants’ exams were weighted and scored on the AP scale out of 180 points.  On average, the group increased this raw score by 35.5 points (19.7%) on the FRQ portion (see Figure 3) and by 29.7 points (16.5%) on the DBQ portion (see Figure 4).

Discussion

Based on these results, it appears as though a writing-intensive approach to course curriculum is the most effective way to increase both content knowledge and content skill.  While there was significant improvement in AP history exam scores at the semester’s end, this was not the case for the AP government and politics exam.  Exam-based teaching in large sections appears to create a ‘teach, learn, test, forget’ pattern.  A telling exception proved this rule: the one student who scored a ‘4’ on the AP government and politics exam had taken AP government in high school and scored a ‘4.’  The student retained the knowledge and ability.  As students engaged in a rigorous curriculum that required a use of the skills necessary to succeed on the written FRQ and DBQ portions of the AP history exam, they did not sacrifice the basic content knowledge they were promised in the course description.  In fact, rather than detracting from the amount of information students were able to retain and use, spending more time practicing critical thinking and writing skills appears to prepare students for the MC portion of the exam more than an information-based course curriculum does.

On the downside, University High School in Normal, IL, and Maine South High School in Park Ridge, IL, both expect an 80% pass rate (a score of ‘4’ or ‘5’) from AP history preparatory classes.  Only 29% of students in HIS 135 were able to achieve a ‘4’ on the AP history exam, and none earned a ‘5’ overall, even with a writing-intensive approach.  This might be attributed to any number of factors including quality of instruction, the fact that no AP-specific preparation was provided in Dr. Winger’s course, and the self-selection of students for high school AP courses, as well as intrinsic student motivation or ability.  Finally, it may be difficult to square a demanding ‘4’ on the AP exams for credit at ISU when only 29% of students in our own course were able to reach that mark.  Only 51% of students achieved a ‘3’ or higher on the post-test FRQ portion of the exam, easily the most difficult portion, so there is definitely room for improvement in this key category.

On the other hand, 70% of HIS 135 students were able to achieve a passing overall score of ‘3’ at the end of the semester.  Impressive gains were achieved in a writing-intensive environment, even on the multiple choice section of the exam, and because writing and document analysis were so central to the course, we especially take consolation in the post-test DBQ scores as 65% of students passed this section with an overall of ‘3’ or higher.

The AP exam is only one measure, and if not quite objective, it is an independent measure, one that is widely accepted, including by ISU.  The results here strongly suggest that smaller sections of writing-intensive general education courses would serve our students far better than mass sections supported by ‘clickers’ and other electronic gadgets intended to provide general education on the cheap.  If carefully trained, a graduate assistant and even undergraduate teaching assistants can provide essential support for slightly larger classes, but there are limits to what can be done by any one instructor.  Like a golf-swing, critical thinking and writing requires individual instruction.  They require practice and repetitive scrutiny.  This in turn requires writing, correction, and rewriting.  Without working through information in this way, it appears little is actually retained.  There can be no substitute for making students think and write again and again about challenging issues, supporting their arguments with evidence from difficult sources.

There are limits to the use of the AP exam in this way.  It is labor intensive, and the test is subject to critique on the merits.  But it does provide a useful benchmark.  The AP exam might be used to evaluate the program performance of our writing courses, ENG 101 and COM 110, for instance, and we call upon our colleagues in relevant disciplines to do this.  Any discipline that has a corresponding AP exam has a ready-made widely-accepted benchmark.  This information might also be used to check rampant grade inflation driven in part by the reliance on student course evaluations in the assessment of faculty.  Although 30% of students in HIS 135 did not achieve a passing score of ‘3’ on the AP history exam, they did not fail the course.  Perhaps they should have.  Had they been warned of that possibility from the beginning, perhaps they would have worked much harder and scored much higher.  More importantly, they might have learned something and grown intellectually in exchange for their tuition dollar.  Whatever else one says about the AP exams, they have had the effect of enforcement, reinforcing intellectual seriousness in the high school environment.  Using the AP exams or similar exams, such as the International Baccalaureate, in program analysis might do the same for ISU.

References

General Education Task Force (2012). Mission, Goals, and Outcomes. Normal, IL: Illinois State University. Retrieved from http://provost.illinoisstate.edu/downloads/gened/MissionGoalsOutcomeFinal.pdf

Illinois State University (2014). Undergraduate Catalog 2014-2016. Normal, IL: Office of Enrollment Management and Academic Services. Retrieved from http://illinoisstate.edu/catalog/pdf/undergrad.pdf

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s