Greetings Paulo Roberto Almeida,
New items have been posted in H-Diplo.
Please help us keep H-Net free and accessible. $25 from each of our subscribers would fund H-Net for two years. Click here to make a tax-deductible donation online.
H-Diplo Forum 2021-2 on Scholars and Digital Archives: Living the Dream?
H-Diplo Forum 2021-2
H-Diplo Forum on Scholars and Digital Archives: Living the Dream?
6 October 2021 | https://hdiplo.org/to/Forum-2021-2
Commissioning Editor and Chair: Richard H. Immerman
Editor: Diane Labrosse
Production Editor: George Fujii
Contents
Introduction by Richard H. Immerman, Williams College
Essay by Matthew Connelly, Columbia University
Essay by Kaeten Mistry, University of East Anglia
Essay by Christopher J. Prom, University of Illinois at Urbana-Champaign Library
For the vast majority if not the entirety of readers of this forum, the appeal of digital archives is almost irresistible. Generations of historians and other researchers, particularly those with decades-long careers, have spent countless days and weeks applying for funds, planning schedules, booking flights and accommodations, and then travelling in order to spend days or months poring through folders and boxes. Make no mistake. These journeys are often rewarding and exciting, and I’ve always found the sight of a cartload of boxes to be energizing. I even enjoy their smell. Yet these treks are also very expensive, time consuming, and difficult to balance with teaching schedules, family vacations, and other commitments and opportunities that are integral to our lives. Then came the COVID-19 pandemic and the compulsory closure of archival facilities throughout the United States and the world. As I am writing the introduction to this forum, some but not all are reopening, in many cases slowly and encumbered with many restrictions. At many, access by researchers is severely restricted. Many historians are all but paralyzed. Graduate students cannot complete theses and dissertations; junior faculty cannot finish books even as the tenure-clock keeps running; grant recipients have time and money but no place to go. The scholarly enterprise is crippled.
The intrinsic appeal of digital archives is that they can provide a solution to such challenges. Rather than having to secure the time and money necessary to travel, or waiting until a closed facility opens and even extends its hours, we can do our research from home. I vividly recall a conversation I had with my good friend and wonderful historian, the late Nancy Tucker, back in the 1980s. Nancy was focusing on the U.S. presidential library system, a small piece in a much larger archival mosaic. She was bemoaning the logistical nightmare created by the need to travel from Independence, Missouri to Abilene, Kansas, to Waltham (soon Boston), Massachusetts to Austin, Texas to conduct research for her current project (this in addition of course to Washington, D.C., and other domestic and international sites). She suggested that the Society for Historians of US Foreign Relations and other historical organizations collectively propose a consolidation of the presidential library system. Nancy knew her proposal was a non-starter. Nevertheless, it reflected a widespread sentiment. There must be a better way for underpaid, resource-starved, and time-constrained historians to access their essential documents; they should not have to book flights, rent cars, and/or take buses in order to get to and shuttle between West Branch Iowa, Hyde Park, New York, and Independence, Missouri.
One can only imagine how Nancy would have reacted when she learned that President Barack Obama decided that he would not deposit his “papers” in a library, in Chicago or anywhere. They would all be digital, held in a cloud, and managed by the National Archives and Records Administration out of Washington. And the evidence points toward Obama’s decision becoming a model for all successive presidents (as with everything else, Donald Trump is likely to remain a flame-throwing wild card). The archival component of any future presidential library will be digital.
Much the same may hold true for America’s National Archive. A June 28, 2019, memorandum jointly issued by Russell Vought, at the time the Acting Director of the Office of Management and Budget, and Archivist of the United States David Ferriero directs agencies to digitize all their remaining paper records by December 2022.[1] The National Archives and Records Administration (NARA) will no longer accept paper records after that date, at which time it will manage all its permanent records electronically. Many agencies will receive waivers. Nevertheless, albeit perhaps delayed due to the pandemic, all are busily turning their records into digital archives.
There are drawbacks to these projects to facilitate access (and conserve space) that go beyond the time and cost of digitizing paper records and making discoverable those and the growing volume of records that are ‘born digital’ (a key concern of these essays).[2]Traveling hither and yon domestically and internationally is expensive and often difficult. Nevertheless, where else could one find other researchers, at times an entire community of researchers, with similar interests and questions? Informal conversations in cafeterias and boarding houses frequently proved invaluable. And then there are the archivists, many of whom have unparalleled knowledge of and insight into the collections, both those that the researcher has identified as crucial to the project at hand and related ones that otherwise would remain unfamiliar to him or her. In the digital world, researchers can both access documents and outline their interests and communicate their questions to archivists. But that’s not the same as a face-to-face conversation, nor can an email or a telephone call replace the discoveries found by accident when combing through folders housed in one large archival box of documents. Further, the jury remains out as to whether an archivist at a digital archive will over time develop the same expertise in the holdings as one at a brick-and mortar archive. It’s too early to tell.
There’s another unknown: without a “natural” home for their papers, where will private individuals “deposit” their papers in the digital age. We all know of discoveries of valuable archives in basements and archives. Will those historical actors whose papers do not fall under federal guidelines keep their papers on private servers? Delete some or maybe even all of them? How will historians know if they are retained, and if so, whether they have been curated? What incentive will public actors have to leave their papers to a library or a historical society, let alone a federal depository?
The following essays do not and cannot address each of these questions. But they do identify, explain, and evaluate the many opportunities and challenges that digital archives present. And they do so from diverse perspectives and expertise. Matthew Connelly is a pioneer in the digital archive universe. His introduction to archival holdings began while he was a graduate student and evolved from his profound interest in information technology. Rather than pursue an archival track, he experimented with and tested these developing technologies—hardware, software, and a variety of services and databanks in the course of his study and research. In doing so, he created his own “personal archive,” while recognizing that this collection at best resembled a digital library.
The more expert he became, the more sensitive Connelly became to the inherent problems, most notably those of locating or discovering documents within the collections—finding needles in the haystack. Key-word searches yielded not only limited results but also often inaccurate and misleading ones. He explains in his essay how this appreciation of this phenomenon led him to examine, using the State Department Central Files (RG 59) as his source base, how a voluminous digital archive might actually promote secrecy as opposed to discoverability and transparency. The explosion of born-digital documents has produced a bonanza of riches for historians but also is marked by serious and in some cases intractable impediments to harvesting them. Connelly does not minimize the potential benefits to historians of the riches, but he underscores the impediments. He also proposes ways, using himself as an example, that historians can mitigate these impediments, by developing more sophisticated understandings of if not extensive expertise in data science and by collaborating with data scientists and archivists. Most important, historians must increase their awareness that the digital turn holds as much potential to turn into our nightmare as it does our dream.
Kaeten Mistry’s essay provides us with a glimpse into that nightmare. Mistry is among the few historians whose scholarship has required deep immersion in the archives of the Central Intelligence Agency. That archive is virtually entirely digital. Indeed, it may well be the largest digital archive in the world (some paper CIA records are on deposit at NARA II in College Park, MD, but most of these remain classified or otherwise off limits to non-agency personnel). For that reason, digitalization should have been a boon for a historian like Mistry, who is based in England. He should in theory have gained much more ready access to it. He explains that initially the archive was in fact accessible solely by consulting the CIA Research Search Tool (CREST) that could be found only on set number of computer terminals housed at National Archives II in College Park, MD.
That problem (not just the need to travel to College Park but also the limited number of terminals) suggests to Mistry a theme that runs through his essay: that the CIA created a digital archive, and only a digital archive, to deter if not to prevent scholars from discovering its contents. That situation changed less than half-dozen years ago, when the CIA uploaded all its declassified records to an Electronic Reading Room that its website hosts and made CREST available from any home computer. Yet because the archive manifests so many of the problems pinpointed by our essayists, it remains, to use Mistry’s words, “while not entirely useless… not useful in any meaningful way.” Probably the most glaring problem is the search engine, which is based on key words. But a key-word search will invariably turn up hundreds if not thousands of documents that lack context, chronological order, or even logic. The essential metadata is absent. The researcher can never evaluate how complete or comprehensive the search’s yield is on any topic. The multiple redactions in the documents exacerbate the problems, but they are not the fundamental cause. Mistry judges the inutility of the CIA’s digital archive purposeful, arguing that it is rooted in the agency’s commitment to secrecy. While the Agency proclaimed the creating of its digital archive as progress toward CIA transparency, the practical consequences for researchers have been the opposite.
Christopher Prom’s expertise and experiences contrast sharply with Mistry’s. While still a Ph.D. candidate, he began working as an assistant archivist. That turned into his career, and his career trajectory paralleled that of the digital archives. Indeed, among a handful of scholars steeped in the craft of both history and archival management (for this reason the American Historical Association appointed him to its inaugural NARA Review Committee), Prom, as he writes, became a charter member of “group of like-minded professionals who are working in the corners of our digital economy to address an issue that most people assume has been solved, but hasn’t: how to preserve a digital record of society.” The goal is to build “discoverable, usable, and sustainable digital archives.”
Prom’s essay identifies what achieving that goal requires, the progress being made, and the obstacles and challenges archivists confront. A digital archive, Prom’s contribution reveals, is far more than a collection of digitized documents, whether they are scans or were born digital. Unless expertly created and carefully managed, the digital archive runs the risk of obscuring contextual information (metadata) that is essential to understanding a record and identifying its relationship to other records. It can also conceal gaps in the record. In other word, it can turn into a CIA-like archive.
Joseph Wicentowski’s essay demonstrates unambiguously how different the outcome can be when the objective of an archivist, or an archive’s creator, is to serve the researcher as fully as possible. Connelly and Mistry refer positively to the State Department’s Foreign Relations of the United States series in their contributions, as would almost any reader of this forum. Wicentowski, who works in the State Department’s Office of the Historian (OH) as the digital history advisor, is most responsible for the series’ digitization. Because of his expertise and attention to detail, in fact, and the dedication of his OH colleagues, the digitized FRUSvolumes that are now available on the office’s website (history.state.gov) not only meet Prom’s definition of a digital archive, in contrast to a collection of digitized documents, but they contrast sharply with that of the CIA.
With commendable specificity Wicentowski explains not only how to use the FRUS archive, but also the methods and technology that went into making it such a valuable resource for all historians (not just those who concentrate on U.S. foreign relations). His essay illustrates the vast potential of digital archives, but warns that reaching that potential requires vast amounts of time, effort, and know-how. Many H-Diplo readers and subscribers have used the digital editions of FRUS. An added value of Wicentowski’s essay is that it provides a manual on how best to exploit the full spectrum of the archive’s capabilities.
Because plain text format will not support images and graphs, with the exception of this introduction, H-Diplo opted to publish this forum exclusively as a PDF, accessible via https://issforum.org/to/Forum-2021-2. You will learn a great deal by doing so.
Participants:
A former president of the Society for Historians of American Foreign Relations, Richard H. Immerman is ending more than a decade-long tenure as chair of SHAFR’s Historical Documentation committee. He is also ending an equally long tenure as chair of the State Department’s Advisory Committee on Historical Diplomatic Documentation (the HAC). He continues as chair of the American Historical Association’s recently established NARA Review Committee. Retired from Temple University, Immerman will serve as Williams College’s Stanley Kaplan Distinguished Visiting Professor in American Foreign Policy in 2021.
Matthew Connelly is a professor of international and global history at Columbia. He is co-director of the Institute for Social and Economic Research and Policy, and principal investigator of History Lab, an NSF and NEH-funded project to apply data science to the problem of preserving the public record and accelerating its release. His publications include A Diplomatic Revolution: Algeria's Fight for Independence and the Origins of the Post-Cold War Era (Oxford University Press, 2002), and Fatal Misconception: The Struggle to Control World Population (Harvard University Press, 2008). His current book project, to be published by Random House, is titled “The Declassification Engine.” Matt has written research articles in Nature Human Behaviour, the Annals of Applied Statistics, Comparative Studies in Society and History, The International Journal of Middle East Studies, The American Historical Review, The Review française d'histoire d'Outre-mer, and Past & Present. He has also provided commentary on international affairs for The Atlantic Monthly, The New York Times, The Washington Post, and Le Monde, and has hosted documentaries for BBC Radio.
Kaeten Mistry is Associate Professor of American History at the University of East Anglia. Among his publications are The United States, Italy and the Origins of Cold War: Waging Political Warfare (Cambridge: Cambridge University Press, 2014), with Hannah Gurman, Whistleblowing Nation: The History of National Security Disclosures and the Cult of State Secrecy (New York: Columbia University Press, 2020), and articles in Diplomatic History, Journal of American History, and the Washington Post. His current project explores the culture of modern secrecy.
Christopher J. Prom is Associate Dean for Digital Strategies in the Library at the University of Illinois at Urbana-Champaign and holds a Ph.D. in History from the University of Illinois. He is a Fellow of the Society of American Archivists and previously served as its Publications Editor and Chair of the Publications Board. He is currently directing two grant projects: Email Archives: Building Capacity and Community and Email Archiving in PDF: From Initial Specification to Community of Practice.
Joseph C. Wicentowski is the Digital History Advisor at the Office of the Historian, U.S. Department of State. He completed his Ph.D. in History at Harvard University in 2007 and joined the Office of the Historian the same year. He co-authored XQuery for Humanists (Texas A&M University Press, 2020).
Notes
[1] Russell T. Vought and David Ferriero, Memorandum for Heads of Executive Departments and Agencies (M-19-21), “Transition to Electronic Records,” June 19, 2021, https://www.archives.gov/files/records-mgmt/policy/m-19-21-transition-to-federal-records.pdf.
[2] Why Do Historians Still Have To Go To Archives?” Contingent Magazine, March 25, 2019, https://contingentmagazine.org/2019/03/25/mailbag-march-25-2019/.
