Documenting the Information Needs of Students and Faculty for ATS Standards Using Student Bibliographic Analysis
Abstract: The question asked in this paper was whether student bibliographies can be used to assess the usage of a library’s collection and thus answer the ATS standard of documenting information needs. The goal was to analyze ten percent of student bibliographies submitted in papers during the past four semesters. This analysis showed that 88% of the citations in this sample could also be found in our two database vendors, the Digital Theological Library (80%) and EBSCO’s eBook Religion Collection (46.4%). Additionally, vendor-supplied statistics did not always parallel student citations.
The Association of Theological Schools (ATS) Standard 6.5 calls for “Documenting that the information needs of the school’s students and faculty are met in ways that are appropriate to the school’s educational mission, degree programs, and educational modalities” (ATS 2020, 10). Because the Memphis Theological Seminary (MTS) Library is entirely digital, circulation statistics for print books are no longer available. Therefore, we have relied on vendor reports from our two major database suppliers, the Digital Theological Library and EBSCO’s eBook Religion Collection to supply the equivalent of circulation statistics. However, only one of our databases, EBSCO’s eBook Religion Collection, can provide title-level statistics, and our collection of EBCO ebook titles is much smaller than the Digital Theological Library.
Authors of a previous study noted that their project was time consuming: they reviewed 78 student research papers, yielding 441 bibliographic citations for the purpose of determining quality (Heller-Ross 2003). This project was also time-consuming. However, the primary purpose was not to determine quality, although that was a factor. Instead, this study sought evidence that students used our databases, particularly the Digital Theological Library. This study was limited to the past four semesters in the academic years 2023 and 2024 and covered 127 citations and 21 student research papers.
To retrieve title-level statistics for the Digital Theological Library, the primary challenge was to match our database title holdings with what was found in the student bibliographies. This examination of student bibliographies found a high level of correspondence (80%) between the titles cited in the papers and the titles found in the Digital Theological Library. Combined with the matches found in EBSCO’s eBook Religion Collection, the resulting coverage was 88.1%. By itself, EBSCO’s eBook Religion Collection matched 46.4% of the citations. Only 33% of the citations were found in both databases. Even fewer were not found in either database, 11.8%.
Although the student bibliographies showed that most of the resources they cited could be found in our two largest databases, a study of these citations could not unequivocally show that the students used our databases for all of their citations.
A similar statistical uncertainty would also be true for print books checked out versus print books cited in student papers. A print title found in the local public library would look the same as a print title from a seminary library when cited in a student bibliography. Seminary circulation statistics for that title might show heavy use, but it cannot link the student’s use unequivocally with the copy of the title found in the seminary. Thus, when matched, the association between titles found in our databases and those found in student papers is an inference.
The conclusions in this study are based on the following: First, if a citation appears in the paper, it meets the student’s information needs. Second, if it is posted in Canvas, it likely meets the professor’s expectations. Last, if the citation also appears as a resource in the databases owned by the MTS Library, it is an instance of the library fulfilling the students’ and faculty’s informational needs.
Scrutinizing student bibliographies removes the uncertainty that is associated with vendor reports. EBSCO’s eBook Religion Collection usage reports of ebook titles do not equate one-for-one with citations found in student papers. While students do cite the highest-accessed ebooks in the Religion Collection, student citation counts can be much lower.
For example, the third edition of Newsom’s Women’s Bible Commentary, which is not in the Digital Theological Library, was accessed 16 times in EBSCO’s eBook Religion Collection during the semesters studied, but it was only cited twice in the papers examined. Myers and Enns’ 2009 Ambassadors of Reconciliation, found only in EBSCO’s eBook Religion Collection, was accessed 20 times but only cited twice in student papers. Lastly, Rasell’s 2022 title, Way of Abundance, Economic Justice in Scripture and Society, found only in EBSCO’s eBook Religion Collection, was accessed 29 times but only cited twice.
As might be expected, both the Digital Theological Library and EBSCO’s eBook Religion Collection reported an increase in ebook usage once MTS book stacks were closed.
For the Digital Theological Library, ebook download statistics increased nearly threefold in the studied period, from an average of 150 to nearly 400 per month. Evenly distributed across all our approximately 200 students, that would mean two ebook downloads per month or eight ebook downloads per semester per student.
Some additional findings are worth noting.
In conversation with students, they say that they often buy the books they want. They identify a book from our databases as a promising resource by downloading it and browsing its contents, like the preview option for some books in a Google search. If they like it, they buy it. A few of our students report buying all the books used for their papers even after finding them online.
This preference for paper usually comes out in a reference interview. When informed that we no longer circulate print books, they want to know if the whole ebook is downloadable. When told that it is partially downloadable, they are sometimes disappointed and say they will purchase the book.
For other seminaries considering a move to an all-online library, it is worth knowing that seminary students may have a stronger affinity for print than other students in higher education. Based on personal observations, a comparison of marginalia and underlining in seminary library print books is much more extensive than in other types of academic libraries. Encountering these marginalia for the first time is reminiscent of encountering similar kinds of marginalia found in medieval manuscripts. For some MTS print books with several decades of usage, the exegetical student commentary constitutes a lengthy and sometimes heated dialogue. Unless students print out parts of an ebook, underlining and adding written commentary is not possible. Thus, they may prefer to purchase the book rather than be restricted to partial downloads.
A similar phenomenon occurs when digital chapters of ebooks and articles are pushed to students struggling with doing their research. These forwarded resources may seem especially relevant, but they are often missing when examining the student’s final paper.
With fluctuating enrollments, tracking simple download reports from vendors may not be useful. However, tracking average downloads per student could be a good statistic for documenting that information needs are being met, especially if compared with other seminary libraries.
EBSCO’s eBook Religion Collection reports are reliable documentation of browsing behavior by themselves, something the profession has long sought by posting “do not shelve” signs in the stacks. Even the Digital Theological Library’s count of total downloads gives a limited picture of student browsing activity. However, examining student bibliographies is better still. They are a record of what the students found useful after all the browsing and downloading was done.
There are additional reasons for examining student bibliographies.
This last semester, for instance, we saw a sudden increase in the use of peer-reviewed articles in master’s-level student papers. This appears to be the result of the Google Chrome “Angular 15” error at the end of the semester that prevented students from using the Digital Theological Library in either Chrome or Microsoft Edge. That this simple error would have that impact was surprising. The explanation is that the other database available to students at MTS is EBSCO’s eBook Religion Collection, and our collection of ebooks in the Religion Collection is small. So, when students could not find any eBooks in the Digital Theological Library and too few eBooks in EBSCO’ eBook Religion Collection, they looked at Atla Religion Database with AtlaSerials PLUS and found many relevant articles. Consequently, that semester, citations for peer-reviewed articles became more abundant in student papers.
An examination of student bibliographies also shows student collaboration. This is most evident when several student papers for the same class include several or all the same bibliographic citations. It also suggests that buying additional copies might be advisable if multiple students use one digital copy.
Opportunities for education about citing were also evident. Whether or not they should take the form of individual tutoring sessions is open for discussion.
If the student finds the resource from another location, this might mean that the seminary library should buy the title. Four ebook titles not owned at the time this study was completed were purchased as a result of student citations.
Examining student papers also provides an opportunity to evaluate databases. In our case, students still use EBSCO’s eBook Religion Collection even though it contains few ebooks. Anecdotal evidence suggests that they use the Religion Collection in part because they are more familiar with it. In this sample, however, students also found journal articles in Atla Religion Database with AtlaSerials PLUS that cannot be found in the Digital Theological Library.
This study also showed that students are still using Google searches. This is evident in the number of open access sources used in their papers that were also not in our databases. Although only three such open access sources were used, they were peer-reviewed and current.
For other librarians interested in pursuing this kind of analysis, looking at just one class will give a better idea of what is happening or not happening in the classroom. For instance, at least one student in this sample shows evidence of avoiding digital resources. The temptation is to ignore this kind of paper and pick the best papers from the class because the quality of the citations is easier to match in the database. However, this is a disservice to struggling students. They are more likely to come to us for help than the accomplished students, and when they do, it would be good to have some foreknowledge of their abilities.
A hazard of tracking individual titles cited in papers or as part of a download report from a vendor is the “hidden citations” that we know are there but do not appear in our statistics gathering (Meng 2024). As an example, in biblical studies, the most cited book should be some version of the Bible. However, students often use their own Bibles rather than those provided by their seminary, with the consequence that the most cited book is generally not provided by seminaries and will not show up in our statistics. In a similar vein, St. Augustine of Hippo is often not cited in papers that refer to the concept of original sin. Many other titles could be mentioned, but the point is that title-level statistics will only give a partial view of the information needs of students and faculty.
If used long term as a method for documentation, citation analysis sample size must be predetermined and adhered to. The sample in this study, 127 citations and 21 papers, represents about 20% of the papers generated by students and 10% of our student population for the period studied. This may or may not be a sufficient sample size to meet the criteria in the ATS Standard.
Lastly, students are using artificial intellignce to write their papers. A few student papers showed evidence of AI generation and were consequently not used for this study. What is important is that none of these AI-generated citations can be verified. Shabby citations in student papers may turn out to be one of the indicators of AI usage.
References
Association of Theological Schools (ATS). 2020. Standards of Accreditation. Association of Theological Schools Commission on Accrediting.
Heller-Ross, Holly. 2003. “Assessing Outcomes with Nursing Research Assignments and Citation Analysis of Student Bibliographies.” Reference Librarian (37) 77: 121–136. Preprint. https://soar.suny.edu/handle/20.500.12648/1189.
Middleton, Anne. 2005. “An Attempt to Quantify the Quality of Student Bibliographies.” Performance Measurement and Metrics 6 (1): 7–18.
Meng, Xiangyi, Onur Varol, and Albert-László Barabási. 2024. “Hidden citations obscure true impact in science.” PNAS Nexus 3 (5): 1–10. https://doi.org/10.1093/pnasnexus/pgae155.