This report is merely an overview of a (nearly) five year evaluation process (2/89 - 8/93).
The following points are the focus of this overview report. A more detailed edition of the information included
in this executive summary follows the summary report. In addition, further indepth analysis of the data collected
for each segment of this evaluation will be conducted in the coming academic year (1993-94).
In February of 1989, planning began for a formal evaluation of Hutchins Library's bibliographic instruction program. The phases of the evaluation process were identified as follows.
By June, steps 1-3 had been completed and steps 4-5 were begun. The focus of the work during the summer of 1989 was the development of the pre-test/post-test (Hutchins Library Inventory) to be used with the incoming students in the fall (Class of 1993). The pre-test survey was administered at the same time, and as a part of the New Student Information Form. The Information Form provided additional data about incoming students; some of the additional data would be of interest to the library in light of the evaluation process, but would not be used as part of the actual pre-test/post-test.
In the fall of 1989, the pre-test/post-test was administered and data from it was collected. In spring of 1990, Hardesty's "A Scale to Measure the Attitudes of Classroom Instructors Toward the Role of the Academic Library in Undergraduate Education"* was used to survey the faculty. Scoring and summarizing the data from the pre-test/post-test was also begun. Scoring and entry of all but one section of the pre-test/post-test data was completed and collation of the faculty survey data was begun in the fall of 1990.
In the spring of 1991, the process of selecting a sample for focus group interviews was begun. Social Security numbers from the two test databases were matched, so that only those for whom both sets of data existed would enter the sample pool. Library workers were eliminated from the pool. The collation of the data and comments from the faculty survey had also been finished.
Compilation and analysis of the answers from the final section of the pre-test/post-test (three completion questions) were completed in late 1991 as part of the numerous segments involved in step 6. Development of questions and a script for focus group interviews also began in the fall of 1991. Institutional Research was consulted to help shape this section of the evaluation and to discuss other phases of the evaluation. A sample group was selected and student interviewers were trained.
Focus group interviews occurred during the short term of 1992 and the long process of transcription of the tapes began, continuing through the summer and into the fall of 1992. Institutional Research was consulted regarding the process to be used in the summary of the interview tapes and that process was begun. Senior Requirement papers were obtained in order to evaluate students' use of library sources.
During Short Term 1993 an assessment tool was developed to be used in evaluating Senior Requirement papers. The process of reading and evaluating the papers then took place during the spring term.
Because of the nature of such a project, the process stretched over several years and involved several staff members (Molly Pitts, Susan Henthorn, Tom Kirk). The following chart might be helpful in getting an overall picture of the individual pieces of the process. Dates indicate time span from drafting of documents to complete compilation of data only and do not include analysis and summary dates.
| SOURCE | TYPE OF DATA |
DATES
|
| Student Information Form | General Information |
8/89 -- 8/89
|
| Pre-test | Library skills assessment |
8/89 -- 8/90
|
| Post-test | Library skills assessment |
8/89 -- 8/90
|
| Faculty Survey | Attitudinal responses & comments |
4/90 -- 5/91
|
| Focus Group Interviews | General responses & comments |
9/91 -- 5/93
|
| Senior Requirement Papers | Use of sources assessment |
9/92 -- 7/93
|
Student Information Form.................General Information.............................8/89.....8/89
Pre-test.....................................Library skills assessment..........................8/89.....8/90
Post-test....................................Library skills assessment.........................8/89.....8/90
Faculty Survey......................Attitudinal responses & comments.....................4/90.....5/91
Focus Group Interviews.............General responses & comments......................9/91.....5/93
Senior Requirement Papers.............Use of sources assessment........................9/92.....7/93
Each of the six parts of this evaluation process provided data to be analyzed. The Student Information Form primarily furnished some preliminary demographic data on the group of students that would be studied during the four year process (students entering in the fall of 1989). Several of the responses from this form were compared with results for the pre/post test in order to determine whether any correlations existed. The correlation figures for these comparisons proved to be relatively low and not particularly significant.
The pre/post test inventory data showed that there were significant differences in student library skills between the date of entry and the end of the first semester. Some of the difference could be attributed to library instruction, but some of the change might simply be a result of growing familiarity with a new setting. One semester is not an adequate time-span in which to see drastic improvement; however, the results did show improvement in specific areas such as familiarity with the use and purpose of the serial record and an expanded awareness of more diverse and sophisticated periodical titles. Overall, students showed improvement in all sections of the Hutchins Library Inventory.
The Faculty Attitudinal Survey was conducted as a replication of a study done by Larry Hardesty and produced similar results. Overall, Berea's faculty are supportive of the library in one way or another. Some believe it is important for them to be actively involved in teaching their students to use the library, while others believe library skills are important but feel it is more the librarian's job to teach those skills. The major conclusion Hardesty reached as a result of his original study, and one that seems applicable to the Berea study as well, is that the attitudes of the faculty members seemed primarily shaped by local conditions. This gives librarians additional impetus to pursue strong and positive interactions with teaching faculty in the area of library assignment development and library support of classroom goals and objectives.
Student focus group interviews proved very revealing concerning student attitudes toward the library's bibliographic instruction program. A gap always exists between what a teacher perceives of as the purpose, content and results of a session and the perception of any given student. The exact nature of the difference between those two views remains to be discovered through some type of evaluation/response process, whether formal or informal. The results of this segment of the evaluation in particular provided insights and suggestions for the B.I. (Bibliographic Instruction) program which will be further enumerated in the following section of this report. In the Bibliography Evaluation portion of this process, the data collected was not such that correlation could be studied or measured. Instead, this section provided yet another way to look at students' library skills and usage. Although success in assembling a quality paper (research or otherwise) is dependent upon several skills, several criteria seemed appropriate for measure as an indication of library research skills. Among these were currency of sources, consistent form of citations and variety of types of sources used to list just three. Analysis of this data showed average results for most of the criteria, however, there was general disappointment in the quality of the papers and bibliographies overall.
Due to the nature of the evaluation process used, the need to make some specific changes in the program was evident prior to the formal conclusion of the process. Comments made during the focus group interviews were especially helpful in understanding the students' perception of the bibliographic instruction process and caused the bibliographic instruction staff to implement some changes immediately.
The major change implemented as a result of the interview comments was one of format. The students did not perceive any difference between the levels of the instruction sessions - freshman composition to upper level course instruction. The staff made an intentional effort to vary the location and format of the instruction sessions so that the "feel" of the various sessions would differ, as well as the content and methodology.
As for changes and challenges for the future, there are several that are immediately apparent:
The evaluation process did not indicate major flaws in the B.I. program, but there is always room for improvement and growth. The overall image students have of the library and librarians seems positive, indicating that students feel relatively comfortable in using the library. Many students mentioned librarians as being good sources for information and helpful throughout the research process. However the information and content communicated during the library sessions were viewed in turn as overwhelming, repetitive, dull or boring. Several students did not seem to grasp the concept of application of library research principles from one setting to another and failed to move in a completely successful manner to the point of synthesis of information, as evidenced by the senior paper/bibliography evaluation section.
While revealing several areas in need of improvement and/or growth, the challenges presented as a result of this evaluation are both possible and positive. As mentioned previously, the evaluation has already helped to infuse the instruction staff with new energy, ideas and methods for accomplishing the goals and objectives of the bibliographic instruction program. The timing of the implementation of the new general core curriculum has been an advantage in the entire process. The insights gained through this process will continue to influence the instruction librarians' planning for new general curriculum courses as they are each in turn developed and implemented. As a result of this process the instruction staff has both some specific and general changes to make in the instruction program. In the best sense of the phrase, to use a popular saying, "Let the game begin."
*Copyright 1982 by Larry Hardesty. Reprinted and used with permission of Larry Hardesty.
Hardesty, Larry. Faculty and the Library: The Undergraduate Experience. Norwood, NJ: Ablex Publishing, 1991.
Susan_Henthorn@berea.edu
mroyse@utk.edu