Hutchins Library

Blue Line

Bibliographic Instruction Program Evaluation


Blue Line


HUTCHINS LIBRARY
BIBLIOGRAPHIC INSTRUCTION PROGRAM
EVALUATION


PRE-TEST/POST-TEST

Student Information Form

Process

General information was gathered from an information form which new incoming students completed. All new students enrolling for the fall term of 1989 were included. Information on this form included such details as age, gender, average high school grades, estimated parental income, SAT and/or ACT scores, probable major, and sections dealing with 1) reasons for attending college, 2) hours per week spent in the last year on various activities and 3) goals and values. (See Appendix A.1 for a complete list of the questions from the form.)

In order to be able to match pre-test and post-test scores, means, etc., the information for this section was "cleaned-up" by matching surveys by social security number and including data only for those respondents who completed both surveys. After analyzing both the full pool of responses and the matched responses, it was determined that the second or partial set was close enough to the full pool to be representative. It is this second set of data that is reported in this document. This information was collected and analyzed in order to get an idea of the demographics of the class that the bibliographic instruction program evaluation would follow for four years. (See Appendix A.2 for detailed statistics.) An overall summary of that data follows.


Summary

The mean High School GPA as reported on the information form was between the grade of B and B+, with the most frequent response being an average grade of a B. Estimated parental income was between $15,000 and $25,000, with most students responding "from $10,000 to $15,000." Approximately 56% (108) of the respondents were female, 46% (85) male (total number = 193). Mean age was between 18 and 19, a fairly traditional sample with the most frequent answer being 18. Because there were many students who took only one of the standard entrance tests (SAT or ACT), there were many missing responses, which threw the statistics off considerably. Because of this, ACT and SAT scores were not analyzed. Probable college major also was not analyzed.

Two reasons for attending college elicited the highest responses, "to be able to get a better job" and "to learn more about things that interest me." These were followed closely by "to be able to make more money," "to gain a general education and appreciation of ideas" and "to prepare myself for graduate or professional school." Scoring lowest in this section were "there was nothing better to do," "I could not find a job" and "wanted to get away from home."

In the area of time spent on activities as a senior in high school, the response with the highest mean score was "socializing with friends," (6-10 hrs/week) followed by "studying/homework" (between 3-5 and 6-10 hrs/week). The least amount of time was spent on volunteer work (between 1 and 2 hrs/week) and partying (between 1 and 2 hrs/week).

Seven goals statements were most frequently responded to as being "very important," with no statement tipping the scale as being "essential." The highest mean was for the statement "helping others who are in difficulty." Some of the other high scores were for "being very well off financially," "raising a family," "helping to promote racial understanding," and "becoming an authority in my field."

While administered as a part of the student information form the statistics for the Hutchins Library Inventory (attitudinal and library skills questions) are discussed in the section of this document that covers the pre-test/post-test data.

HUTCHINS LIBRARY INVENTORY

Along with the standard data mentioned above, the Student Information Form included nine questions that reflected students' attitudes toward the library. At the same time that students completed the Student Information Form, a survey of basic library knowledge was administered. This survey (Hutchins Library Inventory), including the nine attitudinal questions was used as a pre-test (August 1989)/ post-test (November 1989). The post-test was administered in all sections of English 015, English 100 and Freshman Seminar, GST 105.

A summary of the responses and an analysis of the data follows. (See Appendix B.1 for copies of the pre-test/post-test.) Question numbers refer to the numbers assigned to the questions for coding purposes (added to the left of each question on the copies), not the numbers printed on the student copy of the inventory, since the printed numbers changed from pre-test to post-test.


STUDENT SURVEY COMPARISON

 
                                                      PRE-TEST     POST-TEST
Section I (1 = yes, 0 = no)                             Mean         Mean

Q.1 (instruction)                                       .861         .932

Q.2 (public)                                            .295         .233
 
Q.3 (high school)                                       .731         .855

Q.4 (Berea)                                             .171         .573

Q.5 (other college, etc.)                               .114         .125 

Section II	(1 = strongly agree...5 = strongly disagree) 
  (See App. B.2 - see note below)

Q.5 a (pre-test 41)  I feel comfortable...             1.746        2.109

Q.5 b (pre-test 42)  I feel confident...               2.119        2.422

Q.5 c (pre-test 43)  I expect to research...           1.306        1.477 

    d (pre-test 44)  I expect to study...              1.845        2.652 
		
    f (pre-test 46)  I expect ...recreational read.... 2.834        3.071 

Q.5 g (pre-test 47)  Librarians are good source...     1.518        1.844 

    h (pre-test 48)  Librarians are approachable...    1.979        2.203

Q.5 i (pre-test 49)  Information skills important...   1.275        1.622


LIBRARY KNOWLEDGE QUESTIONS
PRE-TEST AND POST-TEST
in order by question

                  PRE-TEST                    POST-TEST 
question    %       no.     total#       %       no.      total#     net %
         correct  correct  response   correct  correct   response   change 

Section III A

  6       29.3	     55       188       36.3      69        190        7.0
  7       10.6	     20       189       23.2      44        190       12.6
  8        4.7	      9       190       47.6      90        189       42.9
  9       16.8	     32       190       26.3      50        190        9.5
 10       70.4	    133       189       74.2     141        190        3.8
 11       88.4	    168       190       89.9     170        189        1.5
 12       46.0	     86       187       73.5     139        189       27.5
 13        8.6	     16       187       21.8      41        188       13.2
 14       30.7	     58       189       43.4      82        189       12.7
 15       29.6	     56       189       24.9      47        189       -4.7

Section III B

 16       90.6	    174       192       88.0     169        192       -2.6
 17       93.8	    180       192       90.1     173        192       -3.7
 18       43.2	     83       192       53.1     102        192        9.9
 19       78.5      150       191       82.2     157        191        3.7
 20       80.2	    154       192       88.0     168        191        7.8
 21       45.5	     86       189       61.3     117        191       15.8
 22       30.5	     58       190       37.7      72        191        7.2
 23       41.4	     79       191       44.0      84        191        2.6
 24       15.9	     30       189       23.2      44        190        7.3
 25       33.2	     63       190       43.7      83        190       10.5 

Section III C	(See Appendix B.3)

 26       84.7      149       176       88.8     167        188        4.1
 27       77.7      146       188       71.6     136        190       -6.1
 28       69.6      133       191       77.5     148        191        7.9


LIBRARY KNOWLEDGE QUESTIONS PRE-TEST AND POST-TEST (Sections IIIA, B and C) in order by net change in percent

 
	                        PRE-TEST	                   POST-TEST 
question    %       no.     total#       %       no.      total#     net %
         correct  correct  response   correct  correct   response   change 

  8        4.7       9       190       47.6      90        189       42.90
 12       46.0      86       187       73.5     139        189       27.50
 21       45.5      86       189       61.3     117        191       15.80
 13        8.6      16       187       21.8      41        188       13.20
 14       30.7      58       189       43.4      82        189       12.70
  7       10.6      20       189       23.2      44        190       12.60
 25       33.2      63       190       43.7      83        190       10.50
 18       43.2      83       192       53.1     102        192        9.90
  9       16.8      32       190       26.3      50        190        9.50
 28       69.6     133       191       77.5     148        191        7.90
 20       80.2     154       192       88.0     168        191        7.80
 24       15.9      30       189       23.2      44        190        7.30
 22       30.5      58       190       37.7      72        191        7.20
  6       29.3      55       188       36.3      69        190        7.00
 26       84.7     149       176       88.8     167        188        4.10
 10       70.4     133       189       74.2     141        190        3.80
 19       78.5     150       191       82.2     157        191        3.70
 23       41.4      79       191       44.0      84        191        2.60
 11       88.4     168       190       89.9     170        189        1.50
 16       90.6     174       192       88.0     169        192       -2.60 
 17       93.8     180       192       90.1     173        192       -3.70
 15       29.6      56       189       24.9      47        189       -4.70
 27       77.7     146       188       71.6     136        190       -6.10

SUMMARY VALUES
	                                                                                                                                                                                  
                                                         SURVEY 1   SURVEY 2
(1 = answer correct, 0 = answer incorrect)                Mean       Mean

Sum 1 - library knowledge, sect. I (Q6 : Q15)             3.332      4.595
Sum 2 - library knowledge, sect. II (Q16 : Q25)           5.505      6.089
Sum 3 - library knowledge, sect. III (Q26 : Q28)          2.241      2.361
Sum 4 - all three sections (Q6 : Q28)                    11.031     12.917

Note:	Perfect score = 23
(See Appendix B.4 for more detail. This Appendix is not available as a 
WWW document - please contact the authors for copies.)


ANALYSIS

Sections One and Two

The first section of the survey (Q1:Q5) was an informational question to determine how much prior formal library instruction students had received and in what setting. Differences in responses to Q1, Q2, Q3 and Q5 are not particularly significant and probably can be explained by a variance in the student's memory. However, the change in the response to Q4 (instruction at Berea College) is a direct reflection of the library's instruction program. Some of the students were enrolled in ENG 015 and ENG 100, which do not necessarily have library instruction included, while others were in GST 105 (Freshman Seminar) which had two formal library instruction sessions scheduled as a part of the course. More than half of the students had received instruction by the end of the fall semester, a reflection of the percentage of students enrolled in the various courses used to administer the survey (ENG 015, ENG 100 and GST 105).

In the second section, students were asked to respond to nine attitudinal statements about the library. (Q5a:Q5i - see Appendix B.2 for further detail. This Appendix is not available as a WWW document - please contact the authors for copies.) On the pre-test, most students seemed to feel fairly comfortable and confident in their library interactions, agreeing that they planned to do research and studying in the library, along with a little bit of recreational reading. They did not see socializing as a library activity.

In the post-test, the mean score for the statements regarding comfort and confidence increased, signifying less agreement with the two statements. Because most of Berea's students have had little exposure to a large library, even one the size of Hutchins, they may have found themselves a little overwhelmed. The size of the building and the collection, as well as the requirement to accomplish complex tasks with information sources (especially in Freshman Seminar), might all contribute to a student feeling less confident and comfortable in the library after the first semester.

The statements reflecting anticipated types of use of the library did not change very significantly from pre-test to post-test. The largest change occurred with the statement about studying in the library. Students unfamiliar with Hutchins may have thought it would be like monitored study halls in high school - quiet and a good place to study. However, once they became familiar with Berea's library, they found that the building (as it then existed) had some significant noise problems, making it more difficult to concentrate and study than they had initially supposed. When asked at the end of the semester about anticipated study in the library, the change in students' responses may reflect experiences with noise problems during the semester. The 1991 addition and renovation have helped to solve the noise level problem and to create an atmosphere more conducive to study.

The last two attitudinal statements reflected a student's perception of librarians. The scores on these two statements did increase, however the majority of answers were still "agree" or "strongly agree."

Section Three

There were three parts to this section: 1) ten questions matching types of information with sections or major library finding aids - serial record, card catalog, etc. 2) ten questions matching specific types of information with specific types of references/library resources - almanac, bibliography, dictionary, etc. and 3) three completion questions - name a periodical, recognize use for geographic source (atlas, globe or map), and recognize definition of an autobiography. (See Appendix B.3 for detailed results from completion questions.)

The overall mean score for the first part of section three increased from 3.33 to 4.595 (highest possible score = 10). The question that had the best gain from pre-test to post-test was question 8, which dealt with the serial record, a term students might not have known in high school and one that might vary from library to library. The percentage of correct answers on the pre-test was 4.7, while the corresponding figure for the post-test was 47.6. This question had the lowest correct response rate, not only for part one, but also for part two on the pre-test and moved to the third highest rate on the part one questions and eighth out of 20 overall on the post-test. Experience in using the library over the semester and instruction in use of the library were important in increasing the correct response rate for this question. One question from part one did show a decrease in percentage points; the question dealing with location of the text of Kentucky laws (Q.15) decreased from 29.6 to 24.9 percent correct.

On the second part of section three, the mean score increased from 5.5 to 6.089 (highest possible score = 10). Interestingly, the questions in this part retained their ranking in percentage correct scores from pre-test to post-test, while showing overall an increase in scores. Three of the questions from part two did loose points from the pre-test to the post-test, but they all had scores of above 70% correct initially and showed, at most, a decrease of 6.1 percentage points.

In comparing part one (sections of the library and library finding aids) and part two (specific types of references/library resources), the questions from part one showed the greatest increase. Since these questions dealt with terminology and sections of a library (Hutchins in particular), they could be seen as being more general and therefore, perhaps, more easily improved through introduction to and experience with a particular library. Questions from part two dealt with specific types of information and the types of sources needed to answer the need. While the mean score for the second part was higher than that for part one on both the pre-test and the post-test, the overall improvement was not as great for part two. Of the seven questions from parts one and two that had an increase of 10 percentage points or more, five were from part one and two were from part two.

The highest percentage correct for any particular question was 93.8 on the pre-test. This question asked the student to identify the card catalog as the most appropriate source to consult to find a particular book.

In part three, which consisted of the completion questions, students had the highest success rate. With a possible high of three (3), the mean for the pre-test was 2.24 and 2.36 for the post-test. Sixty-nine (69) percent or above answered correctly for each of these three questions.

In looking at the specific responses to these three questions, the composition of the students in Freshman Seminar classes must be taken into account. Because Freshman Seminar generally includes some sophomores, the specific responses on this section include those of students who did not take the pre-test. The analysis given above for the first two parts of section three could be made based upon scores for identical groups on the pre-test and post-test. The specific answers for the last three questions (part three of section three) were not indicated in the database, only whether correct or not. Specific answers were gathered from all returned surveys (actual paper copy) and were not identified with a particular respondent. Additional work remains to be done in order to identify only those answers identified as belonging to respondents of both pre-test and post-test.

Keeping that fact in mind, what follows is, at best, a tentative and very general analysis. The answers given for Q.26 (title of a periodical) showed greater diversity in the post-test. Seventy-four (74) distinct answers were included on the pre-test, while 99 were listed on the post-test. The type of periodical listed also showed a change from pre-test to post-test. Time, Newsweek, Sports Illustrated, and U.S. News and World Report were still among the top five answers, but 72 titles were included on the post-test that were not on the pre-test. This shows an awareness of an increasingly diverse and more scholarly level of periodical. (Titles unique to the post-test include Studies in Short Fiction, Business Week, The Economist, Hastings Center Report, Journal of Housing, etc.)

A more indepth analysis of the last two questions on the survey will not be made until a distinction can be made between those responses given by the group identified as having completed both the pre-test and the post-test and those returning only one or the other. For detailed information on part three, see Appendix B.3.

Also examined in this section of the evaluation was the possible correlation between pre-test and/or post-test score and a variety of factors, as self-reported on the student information form. Of the various factors examined, only that of high school grade point average seemed to have even a slight correlation with overall score on either pre-test (.295, covariance of 1.535) or post-test (.333, covariance of 1.988). The following figures are a result of testing for correlation between scores on the various sections, as represented by Sum. 1, Sum. 2, etc. (See Appendix B.1 for Questions included in each Sum.)


PRE-TEST /POST-TEST CORRELATION

 
                                  PRE-TEST

                     Sum. 1        Sum. 2       Sum. 3         Sum.4 
                                                              (total)

         Sum. 1                      .319         .329          .774

         Sum. 2                                   .246          .799

         Sum. 3                                                 .571


                                  POST-TEST
                     Sum. 1        Sum. 2       Sum. 3         Sum.4
                                                              (total)

         Sum. 1                      .378         .224          .824

         Sum. 2                                   .329          .801

         Sum. 3                                                 .503



                              ONE-TAILED T-TEST
       
             t-value        sign. level        pre-test       post-test
             (score)                             mean           mean

 Sum. 1       -8.844           .0005             3.33           4.595

 Sum. 2       -4.209           .0005             5.500          6.089

 Sum. 3       -1.995         .01<>.025           2.24           2.36

 Sum. 4       -8.188           .0005            11.03          12.91
(Total)

df=192	critical value (alpha=.0005) = 3.291


Detailed and indepth analysis of the data gathered through this process will continue in order to gain more fully from the evaluation process. Replication of this study will also be considered in order to assess the impact of changes in the B.I. program.

Susan_Henthorn@berea.edu
mroyse@utk.edu

Blue Line

Hutchins Library Bibliographic Instruction Program Evaluation Home Page
Hutchins Library Home Page | Berea College Home Page


Updated 3/23/01
Mail comments or questions to susan_henthorn@berea.edu