AI FAQ
- Questions Raised at the April 13 Forum and March 23 FC Meeting
- Background on the Achievement Index
- Proposed Policy
- AI and Grading practices.
- Honors
- Grading in Different Disciplines
- Additional Questions
Transparency exists on many levels. The proposal charges the
implementation committee with providing basic explanations of
AI (e.g., it measures student academic performance
independently of course grading practices, it is based on a
student's performance relative to other students in the class
taking into account how those students' overall record of
academic achievement) and of justification for its use (e.g.,
it is a better indicator of relative student performance than
is GPA). In addition, the implementation committee will be
charged with developing materials that show students how their
AI was influenced by specific courses. While care will be
needed in crafting these explanations, there is little doubt
that it can be explained successfully.
It is not expected that most people have the background in
statistics to understand exactly how AI is calculated. Here,
AI has a different type of transparency. The algorithm has been
published in a respected peer-reviewed journal and is
available for scrutiny by disinterested experts. This type
of transparency is important in a complex society where
individuals and institutions frequently rely on methods
established by experts.
While students cannot calculate their own AI scores to check for
errors, we believe this to be a minor concern. First, the
source of errors is almost certain to be in the data
provided to the algorithm, not in the algorithm itself,
which is available for public scrutiny. Second, it is always
impossible for an individual to calculate her own position
in a ranked system. A student cannot now calculate her
class rank, for example.
Collegiality and cooperation among students is very important. The
committee does not believe that AI is likely to undermine
that spirit on campus because the proposed system does not
greatly change the mix of incentives for cooperation and
competition that exist currently. There are courses on
campus now that grade on relatively strict curves but
there have not been complaints of competitiveness about
those classes. In a hard curve, when one student moves
from A- to A another student must move down. With the AI
when one student moves from A- to A there is a very minor
dilution of the achievement associated with an A in that
class. If current classes with relatively hard curves are
not generating excessive competition between students
there is little reason to believe that AI would cause
it. At present students form study groups (which faculty
heartily endorse) because it helps all students in the
group learn the material and because it makes studying
more enjoyable. Those considerations will persist with
AI. Some students also spend time tutoring other students
in situations where the tutor does not need to study the
material further. There is no reason why AI should affect
such altruistic behavior. Furthermore, if synergies exist
that make successful cooperative learning in one class a
benefit to students in their other classes, cooperation
among students would be enhanced by the AI.
At a practical level, an item assessing competitiveness could be
added to course evaluations before use of AI begins. This
would provide a way of tracking any changes in
competitiveness due to use of AI. If increased
competitiveness proves problematic, then changes could be
made in administration of the policy, or in the policy
itself.
What is the problem being addressed?
The University uses Grade Point Average (GPA) as a summary measure of
student academic accomplishment but GPA is flawed because
substantial disparities in grading practices across
courses means that GPA reflects course selection as well
as student accomplishment. The inaccuracies of GPA lead
to inequities when GPA is used for purposes such as
awarding University Distinction, screening job candidates,
and maintaining scholarship eligibility. The existence
of such inequities creates grade-based incentives for
course selection. It also means faculty face undue upward
pressure on grades based on student demands for high GPAs.
Can the problem be
solved by better communication?
Past experience indicates that it is unlikely that problems with
grading can be changed by better communication among
faculty. There was substantial discussion about
grading at Carolina following the EPC report on grade
inflation in February 2000. Four years later, EPC
found found no change in disparities in grading
following the 2000 report. It is essential that there
be communication about grading, but communication will
always be limited in an academic setting where faculty
have very different academic training and where they
typically have contact with only a small number of
faculty outside their own department.
What are the basic policy
options for addressing grading disparities?
There are two basic policy options for addressing grading disparities:
mandatory target grade distributions (curves) for
departments or classes and statistical adjustment.
The 2000 EPC report on grading advocated consideration of common
target grade distributions for all departments,
but Faculty Council did not act on that
recommendation. Curves are problematic
because the meaning of grades varies across academic
disciplines and because administrative mandates on
grade distributions undermine both the faculty's
autonomy in and responsibility for grading.
Why statistical adjustment and why AI?
Statistical adjustment of grading disparities has the philosophical
and practical advantages of leaving
responsibility for grading in the hands of the
faculty. Many advantages of the AI over other
statistical adjustment methods are summarized in
other documents. The principal advantage is that
the AI does not penalize a strong student who
does well in a course where grades are typically
high. Thus, the AI does not replace the current
incentive to avoid classes where grades are
typically low with an incentive to avoid classes
where grades are high.
When would AI be implemented if
passed?
The proposal contains no specific timetable. However, given the
work that would go into design and
implementation of the system, it is very
unlikely that it could be implemented before
the entering class of Fall, 2009.
What is the Achievement Index?
The Achievement Index (or AI), like grade-point average (or GPA), is a
measure of students' performance that is calculated from the grades
that students receive in their classes. Calculation of AI differs from
calculation of GPA in that it takes into account differing grading
practices in different classes.
What is the goal of the Achievement Index?
The
goal of the AI is to measure the academic performance of students
independently of the grading practices employed in the particular
classes taken by different students.
Where did the AI come from?
The AI was
created by a statistician, Valen Johnson, who at the time was on the
faculty at Duke University. It builds on statistical methods that were
developed for use in standardized tests but it is appropriate for
college grading because it does not require standardization of course
content or of methods for evaluating student achievement.
No. Carolina would be the first university to use the
AI. However, the goal of providing information about grading practices
across courses has been addressed by other universities (notably
Dartmouth College and Indiana University) by listing the median grade
in a class on transcripts next to the student's grade.
What advantages are there in using the AI as
compared to listing median class grades?
The AI allows a more direct comparison with GPA than does median class
grade. For purposes of class rank and awarding distinction, where we
need to compare performance across different disciplines, we need to be
able to compare students' performance in numerous classes, not just in
any single class.
Brief descriptions
of the AI and how it is calculated can be found in the report on
Adopting the
Achievement Index and in the Primer on
the AI. Those
summaries provide references to the primary sources.
How will AI be used?
AI will be listed on students' transcripts. In addition, the
University will use it for those purposes that involve comparing the
academic performance of different students, such as providing information about students' class rank and awarding of Distinction and Highest Distinction upon graduation.
Why focus on University Distinction?
The University awards undergraduate degrees with Distinction or Highest Distinction to those students who have excelled in their coursework. At present, these awards are based solely on GPA; students with a GPA of at least 3.5 but under 3.8 receive Distinction and those with a GPA of 3.8 or higher receive Highest Distinction. As a University-wide form of recognition, Distinction requires comparing the grades of students who have pursued very different courses of study. AI provides a way of comparing the performance of students independently of the grading practices in the courses that they have taken. Therefore, it is a more accurate method than GPA for identifying those students who have excelled academically.
Will AI be used for
purposes other than awarding University Distinction?
The proposed legislation establishes an implementation committee
whose charge would include determining whether other uses of AI are
warranted. Possible other uses include: admission to undergraduate
professional programs, admission to first/second year Honors for
students who are already enrolled and admissions to Major Honors
programs. Decisions on whether or how it might be appropriate to use
AI for such purposes require discussion with the faculty, students and
administrators involved in those programs.
Will AI replace GPA?
No, students' GPAs and the grades earned in their classes will be
listed on their transcripts as in the past.
Will
AI be used in setting standards for continuing academic eligibility
and graduation?
No, the proposal recommends no change to standards for continuing
academic eligibility or graduation. Over the last two years EPC and
Faculty Council have made a number of important changes to standards
for continuing eligibility with the goal of increasing graduation
rates. No change in those new standards is proposed.
Why
does it make sense to use AI for University Distinction but not for
eligibility and graduation?
AI is useful in comparing the performance of different students,
not in setting absolute standards. Criteria for eligibility and
graduation should be based on absolute standards not comparative
ones. Carolina admits only those students who it believes have the
potential to master the standards for graduation; achieving a high
graduation rate is a legitimate and important goal for the University. In contrast,
awarding Distinction involves comparing the performance of different
students and recognizing those who have done the best in their
academic work at Carolina. As a comparative judgment, the award of Distinction should be based on the most accurate comparative measure available.
Will AI cause changes in grading that would indirectly affect eligibility or graduation?
The question of whether use of AI would have any effect on faculty
grading practices is addressed in Section 3 on AI and Grading
Practices. Even if the proposed use of AI prompted faculty to assign
somewhat lower grades, such a change would be very unlikely to affect
eligibility or graduation rates. There are 7 grades (C through A)
which fully satisfy eligibility and graduation standards. At present,
instructors in many classes distinguish student performance using only
a small portion of this range.
How
will people learn about AI?
The implementation committee will be charged with developing ways
of describing AI to students and faculty at the University as well as
to those outside of the University. In addition, the committee will
develop methods for providing information to students about their
academic progress in terms of AI and about how particular classes
might influence their AI.
Does
adoption of AI dictate how faculty should grade?
AI provides a statistical context for interpreting faculty grades
when comparing the academic performance of different students, it does
not provide any guidelines about how faculty should grade.
Does implementing the AI privilege courses
that grade "on a curve"?
No. The impact of a grade in a class on a student's AI does depend in
part on the extend to which the grades in a class differentiate levels
of student performance. Strictly speaking, grading on a curve means
that the proportions of students receiving certain grades is specified
in advance for the course, a method that ensures differentiation of
levels of student performance. However, differentiation of levels of
student performance can be achieved by specifying standards that must
be achieved for certain grades, as long as those standards lead to
differentiation of student performance.
How will AI affect professors' grading
practices?
There is no way to determine with certainty whether professors' grading
practices would change or how they might change. We assume that, currently, professors grade
students based on their performance in class and their mastery of the
course material. To the extent that professors are pressured by
students to award high grades for other reasons, the introduction of
the AI may tend to discourage that behavior.
How can a student strategically plan to raise his/her AI ?
Do well in classes in which other strong students are enrolled, and in which instructors issue a wide range of grades.
What is the difference between Distinction
and Honors at Carolina?
University Distinction is awarded at graduation based solely on
GPA; it does not involve any particular type of academic work or
program of study. At Carolina, students can pursue two types of
honors; the Honors Program centers on coursework while Honors in a
major field of study requires successful completion of a Senior Honors
Thesis. A student who successfully completes a Senior Honors Thesis
graduates with Honors or Highest Honors in a specific discipline. The
nature of the work in a Senior Honors Thesis is shaped by the academic
discipline in which the student is working and the thesis is evaluated
using the standards of that discipline by a faculty committee. In this
way, Honors in a major field of study is a much more focused form
of recognition than University Distinction.
Will AI be used in selecting students for
Honors programs?
Some students are accepted into the Honors Program upon admission to
Carolina; obviously AI could not be used in selecting those students.
Other students apply to the Program in the second semester of their
first year or the first semester of their second year. As discussed
above, the AI implementation committee would evaluate whether AI
should should play a role in decisions about admissions.
Currently, a
student must meet a minimum GPA standard to be eligible to undertake a
Senior Honors Thesis. Once again, consideration of whether an AI
standard should also be used would be left to the implementation
committee.
How will use of the AI affect Honors students?
The academic records of Honors students
over the last twelve years are slightly stronger
when evaluated by AI rather than raw GPA. For students in the first-
and second-year honors program, the average raw GPA is 3.59, while the
average achievement-adjusted GPA is 3.65; for 75% of these students,
achievement-adjusted GPA is higher than raw GPA. For students who earn
Departmental Honors, the average raw GPA is 3.70 while the average
achievement-adjusted GPA is 3.73; for 68.4% of these students,
achievement-adjusted GPA is higher than raw GPA. See AI and Honors.
How will use of the AI affect Honors
classes?
Carolina offers Honors sections of departmental course offerings. Students in the Honors Program (and some other students)
enroll in these sections to fulfill general education and major field
requirements. Because these sections enroll students who have chosen a
rigorous course of study and have met high selection standards, the
grades in honors sections tend to be higher than in non-honors
sections. For example, the average percentage of As awarded in Honors
sections was 42.3 while it was
24.6 in the comparable non-honors sections. The impact of grades in a section on AI depends on
two factors: the extent to which the instructor's grades differentiate
levels of student performance and how well the students in the class
have done in their other classes. For Honors sections on average, the
AI calculation for the grade of A places students above the 82.7th
percentile of the student body as a whole while for non-honors section
an A places students above the 75.8th percentile. Thus, the AI
calculation awards slightly more credit for an A in an Honors section
than for an A in a non-honors section, even though a higher percentage
of students receive As in Honors sections. See AI and Honors.
Do grades vary across academic disciplines?
Yes. Analyses of grading patterns at Carolina and elsewhere show
systematic differences in grading across disciplines (see the EPC
reports in
2001 and 2004.
Additional analysis by the grading subcommittee
suggests that over 15% of the variation in students' grades can be
explained by the departments they studied in and the instructors who
taught their classes.
How will AI affect different academic majors?
As discussed in the reports referenced above, there are substantial
differences in grading practices across disciplines. Those differences are incorporated into GPA as
measure of student performance but are not incorporated into AI. The
impact on different majors of using AI to award University Distinction
is shown in the tables that are linked here.
Do disciplinary differences account for
all variation in grading practices?
No. There is often substantial variation in
the grades assigned by different instructors within a department even when they teach the
same course.
Does implementing the AI imply that some
departments are better, or more rigorous, than others?
No. Professors are the experts in their fields, including in the
standards and expectations for grades and the pedagogical goals they
pursue. The AI does not interfere with that expertise. But the
university, in comparing students' performance across department and
instructor, should take into account these variations in grading
practices. The AI is a way of allowing the university to make those
comparisons in a fair way without interfering with instructors'
autonomy.
What is the intended consequence of adopting
AI?
The intended consequence is to make the comparison of student
performance across the university fair by reducing or eliminating the
effect of departments' and instructors' grading practices.
Will there be unintended consequences of
adopting AI?
Of course it is possible that adopting AI will have unintended
consequences. The proposal charges the implementation committee with
developing procedures for tracking the consequences of adopting AI. In that regard,
it is important to note that grades in classes and GPAs will still be
presented as in the past. The continued presence of these familiar
measures is likely to moderate any unintended consequences of adopting
AI. Some possible, though unintended, consequences of AI are discussed below.
How will AI affect the competitive
climate of the university?
There
is no way to determine with certainty how the competitive climate would
change following the introduction of the AI. In general, students who
tend to be competitive academically will likely continue to do so.
After implementation of the AI, students in any given section would
have incentives both to do well themselves and to have high-performing
classmates.
How will AI affect student selection of
courses and majors?
There
is no way to determine with certainty how students' selection of
courses and majors would change. We assume that, currently, students
select courses and majors either to fulfill requirements or to pursue
intellectual and career goals; the introduction of the AI would support those
reasons for course and major selection. If students are currently
selecting courses and majors with reputations for "easy" grading in the
hope of improving their GPAs, the introduction of the AI would tend to
discourage that behavior.
How will AI affect students in
graduate/professional school admissions and the job market?
There
is no way to determine with certainty how students' success in graduate
and professional school admissions or in the job market would change. We
believe that introducing the AI will help reinforce UNC's reputation
for academic excellence, which would in turn help UNC graduates in
their academic and professional careers. Since prospective schools and
employers will continue to have access to raw GPA as well AI, they may
choose to use either or both measures to evaluate prospective
candidates.