born digital archives

AIMS White Paper now available

2012-01-20T08:35:00.000Z

After a huge amount of effort the AIMS White Paper has finally been finished and is now available online.

The White Paper is intended as a framework for guide good practice in terms of archival tasks and objectives necessary for success. It builds upon the experiences of the four partner institutions - the universities of Hull, Stanford, Virginia and Yale - to process a range of collections with an array of format and media issues and using different software – we were keen to make this software agnostic and have gone back to the archival principles at the heart of the processes.

In many areas we found many similarities with existing practice with paper records and for some aspects we found there were multiple ways of achieving certain goals and we didn't want to be prescriptive in any way. So instead it highlights key decision points and aspects of policy that may be determined at an institutional level and is intended to help people making the same journey that we have made – finding out about projects, tools & case studies and starting to build knowledge, skills & infrastructure.

Although the publication of the White Paper officially marks the end of the AIMS project the institutions intend to continue collaborating and sharing their experiences on this blog.

We welcome your comments and feedback to the White Paper on this blog – whether you have implemented the framework or just found the guidance useful.

Day of Digital Archives – some personal reflections

2011-10-06T16:55:00.001+01:00

To mark the Day of Digital Archives I thought I would add a personal note about the “journey” I have made in the last two years. It was about this time in 2009 that it was announced that the AIMS Project was being funded by the Andrew W. Mellon Foundation and that I would be seconded from my post as Senior Archivist to that of Digital Archivist for the project.

At the time I had considerable experience of digitisation but very little about digital archives. So I began reading a few texts and following references and links to other sources of information until I had a pile of paper several inches high of things to read. At first there was a huge amount to take-in – new acronyms, especially from the frightening OAIS, and plenty of projects like the EU funded Planets initiative. It seemed that the learning would never stop – there was always another link to follow another article to read and it was really difficult to determine how much was making sense.

Talking to colleagues who were already active in this field also revealed how little digital media we actually had at the University of Hull Archives – just over two years ago we literally had a handful of digital media whilst others were already talking about terabytes of stuff. Fortunately the AIMS project sought to breakdown the workflow into four distinct sub-functions and placed emphasis on understanding the process compared to ‘traditional’ paper archives which reduced the sense of being overwhelmed by it all.

Since then I feel I have come along way – I have attended a large number of events and spoken at a fair few and quickly become both familiar and comfortable with the language. I do appreciate the time I have been able to dedicate solely to the issue of digital archives and that many colleagues are embracing this “challenge” without this luxury.

The biggest recommendation I can make is to start having a play with the software – many of the tools that we use at Hull University are free – Karen’s Directory Printer for creating a manifest of records including checksums that have been received; FTK Imager for disc images etc etc. Nor do you have to wait for digital archives or risk changing key metadata whilst you are experimenting – you can use any series of digital files or old media that are lurking in the way of a drawer. We have also created a forensic workstation and shared our experiences via this blog.

Once we had started to experiment, we created draft workflows and documentation and refined this as we experimented further – all tasks from photography of media to using write-blockers do become less daunting the more frequently you do them. Having learnt from many colleagues we have started to add content to the born-digital archives section of the History Centre website. I have also used some of my own email to play with the MUSE visualisation tool to understand how it might allow us to provide significantly enhanced access to this material in the future.

Although the project funding has now finished and I have returned to my “normal” job I do think that digital archives has now become part of my normal work and each depositor is now specifically asked about digital archives and in public tours of the building we explicitly mention the challenges and the opportunities of digital archives. We don’t have all of the answers yet – archiving e-mail in particular still scares me, but don’t feel as daunted as I did two years ago.

A Tale of Two conferences

2011-09-04T09:58:00.000+01:00

Last week I was fortunate to be part of the AIMS team presenting our work at the SAA Conference in Chicago. Despite the Saturday 8am start of our session and the impending threat of Hurricane Irene well over 150 delegates turned-out to hear our presentation which included both an introduction to the AIMS framework and reporting our practical experiences through case studies. If you missed it or want to relive it the presentations are available online via Slideshare.

On Friday I spoke at the ARA conference in Edinburgh – the theme of which was advocacy and as part of a Data Standards Group I spoke about the skill set that I had acquired during my change of role from archivist to digital archivist as a result of the AIMS project.

Although the two presentations were different in content and context they both included the same message – an attempt to breakdown the perceptions and myths surrounding born digital archives. In talking about skills in Edinburgh I sought to highlight the relevance of the traditional archive skills in the digital age and to encourage more individuals to do something.

It also raised the question – something that arose in the AIMS unconference in Charlottesville and the UK workshop in London, of when will digital archives become “the norm”. We don’t know the exact answer to this, but I do know it is necessary if we are to successfully manage the challenges of born-digital archives and strive to meet the increasing expectations of our users.

Friday also marked the end of a six month contract during which Nicola Herbert has helped us with the practical elements of digital preservation at Hull. I would like to thank Nicola for her hard work and direct users to her guest blogs on photography of media and write-blockers.

AIMS@SAA Part Two: SAA Session 502

2011-09-02T21:11:00.000+01:00

SESSION 502 - Born-Digital Archives in Collecting Repositories: Turning Challenges into Byte-Size Opportunities
SAA 2011
Chicago, IL
Aug 27, 2011

As the endnote to their foray into the SAA 2011 Annual Meeting, the AIMS Digital Archivists delivered a presentation on the AIMS project on Saturday morning. Although we were competing with Hurricane Irene’s effect on travel schedules, an 8 a.m. Saturday timeslot, and presentations from our colleagues Michelle Light, Dawn Schmitz and John Novak’s on delivering born-digital materials online as well as the Grateful Dead Archivist and a member of the band Phish, attendance was pretty darn good! We were pleased to be able to speak with some colleagues after the session and facilitate a few discussions during the question and answer portion of the session.

The presentation itself gave a brief overview of the project and then focused on the AIMS framework, or the four areas we’ve identified as key functions of stewardship for born-digital materials: Collection Development, Accessioning, Arrangement and Description, and Discovery and Access.

We’re very happy to share our slides here through slideshare. Remember, this is just a taste of what’s to come in the white paper this fall, so keep checking the blog for updates!

Slide are posted here after the jump!

Saa Session 502 Born Digital Archives in Collecting Repositories

View more presentations from AIMS_Archives

AIMS@SAA Part One: CREW Workshop

2011-09-02T21:06:00.002+01:00

CREW: Collecting Repositories and E-Records Workshop
SAA 2011
Chicago, IL 8/23/2011

The AIMS partners hosted a workshop in the run-up to the 2011 SAA Annual Meeting in August. 45 participants from the US and Canada joined us in exploring the challenges, opportunities and strategies for managing born-digital records in collecting repositories.

The workshop was organized around the 4 main functions of stewardship that the AIMS project has focused on: Collection Development, Accessioning, Arrangement and Description, and Discovery and Access. In addition to the AIMS crew (no pun intended) presenting on the research done through the AIMS project, several guest presenters showcased case studies from their own hands-on approaches to managing born-digital materials. Seth Shaw, from Duke University discussed the evolution of electronic record accessioning at Duke University and his development of the Duke Data Accessioner. Gabriela Redwine discussed work done in arrangement and description at the Harry Ransom Center at the University of Texas at Austin. Finally, Erin O’Meara showcased work done at the University of the North Carolina at Chapel Hill to facilitate access to born-digital records through finding aid interfaces.

In between presentations the participants engaged in lively discussions around provocative questions and hypothetical scenarios. At the end of the event, the AIMS partners felt they had gained just as much from the day’s activities as they hoped the participants had. Ideas that were discussed and case study examples will help strengthen the findings of the white paper due out this fall.

See the workshop presentations after the jump!

All of the workshop participants have graciously allowed their presentations to be available here through slideshare. We hope these provide some of highlights of the afternoon for those who couldn’t join us.

Workshop Presentations:
Intro by Bradley Daigle, University of Virginia

AIMS workshop: Introduction

View more presentations from AIMS_Archives

Collection Development by Glynn Edwards, Stanford University

AIMS Workshop pt .1: Collection Development

View more presentations from AIMS_Archives

Accessioning by Kevin Glick, Yale University

AIMS Workshop pt. 2: Accessioning

View more presentations from AIMS_Archives

Accessioning Case Study by Mark Matienzo, Yale University

AIMS Workshop Case Study 2: Re-accessioning at Yale

View more presentations from AIMS_Archives.

Accessioning Case Study by Seth Shaw, Duke University

AIMS Workshop Case Study 2: Accessioning Evolution

View more presentations from AIMS_Archives

Arrangement and Description by Simon Wilson, University of Hull

AIMS Workshop pt. 3: Arrangement and Description

View more presentations from AIMS_Archives

Arrangement and Description Case Study by Peter Chan, Stanford

AIMS Workshop Case Study 3: Arrangement and Description Case Study - Stephen Jay Gould Papers

View more presentations from AIMS_Archives

Arrangement and Description Case Study by Gabriela Redwine, Harry Ransom Center at the University of Texas at Austin

AIMS Workshop Case Study 3b: Born-Digital Materials at the Ransom Center

View more presentations from AIMS_Archives

Discovery and Access by Gretchen Gueguen, University of Virginia

AIMS workshop pt. 4: Discovery and Access

View more presentations from AIMS_Archives

Discovery and Access Case Study by Erin O'Meara, University of North Carolina at Chapel Hill

AIMS workshop Case Study 4: Discovery and Access to Hybrid Collections

View more presentations from AIMS_Archives

Forensic Workstation pt3

2011-08-30T15:58:00.001+01:00

A guest posting from Nicola Herbert, Digital Project Preservation Assistant at Hull University Archives

Once we had the forensic workstation up and running (see part 1 and part 2 in this on-going series) we installed MS Office and Mozilla Thunderbird (for working with Outlook .pst files). We also installed FTK Imager, Karen’s Directory Printer, DROID and the MUSE e-mail visualisation tool (in beta, but provides a very interesting perspective on the data). We are also planning to purchase Quickview Plus, a piece of software that enables viewing a range of file formats without requiring the original software on your PC.

We had already played around with these tools on our normal PCs and had run them on files copied from digital media prior to setting up the workstation.

Having received our two Tableau write-blockers we were eager to combine the separate processes we had developed into an integrated workflow. We have two write-blockers, one for USB devices (T8-R2) and one for internal hard drives from PCs and laptops (T35es). Simon’s visit to Jeremy John at the British Library had whetted our appetite for getting our mini digital forensics lab in operation.

USB devices
After a thorough read-through of the instructions we tested out the USB write-blocker first. Setting it up is relatively simple; the vital thing is to make the connections between device and write-blocker, write-blocker and forensic PC before switching on power to the write-blocker. The forensic workstation recognises the USB device as normal, and off you go.

We then run FTK Imager to create a logical image of the device. We tested the various formats and settings available and eventually decided that creating true forensic images would raise too many trust issues with potential depositors with regard to us being able to restore deleted files. For this reason we will create ‘Folder contents only’ forensic images which recreate the device as it would appear in normal use. From here we are exploring our options for exporting the files from the disk image, but we have found that the exported files display an altered Accessed date – any comments/suggestions on this issue would be gratefully received.

We also create directory listings of the contents with MD5 and SHA-1 checksums. From the disk image and directory listing we can start to consider the arrangement for the collection, using Quickview Plus to preview file contents.

Our second write-blocker can be used with IDE and SATA hard drives...but more of this in part 4!

Forensic Workstation pt2

2011-08-22T10:37:00.000+01:00

When we moved from the University campus to our new joint facilities with Hull City Archives and the Local Studies Library we took the opportunity to upgrade many of our PCs – leaving a few older specimens “just in case” anybody was so desperate that they were willing to accept a machine that was reluctant to start-up!

Recently the library has been re-organising its stock and space-utilisation ahead of a major refurbishment. Our old PC was discovered in the basement and ear-marked for disposal (well recycling really but disposal is less ambiguous). It was at this point, and with a new-found digital archives perspective, that I realised the potential of this machine to become our first digital forensics workstation. With an internal 3.5” floppy drive, CD drive and 2 USB ports this was a combination that seemed to promise possibilities for dealing with a range of media but also the chance to transfer the files once they had been extracted. The PC with slightly grubby keyboard and monitor were shipped to their new home at the History Centre.

I had by this time, started to identify requirements for a new PC to act as a workstation for the capture of hard-drives and other large volume of material. This request intrigued a colleague Tom in ICT and a visit was duly arranged, Tom was really interested in our work and offered to help. Tom took our PC and returned it a few days later - with a clean version of the Windows XP image installed aswell as an internal zip drive added.

Tom has also promised to put aside a couple of internal 3.5” floppy drives as an insurance policy for the drives failing as Jeremy Leighton John at the British Library had reported mixed results when using the external USB floppy drives. Having two workstations, one old and one new, will give us an option for dealing with some media formats; a USB drive for 3.5" floppy drives and an external 250MB zip drive. The latter was found when clearing-out an old cupboard and came with all cables and even its original installation CD proving that assembling a forensic workstation does not have to cost a fortune and I have heard several tales of kit assembled via ebay purchases.

In the interim our Tableau write blocks had arrived and we installed a few bits of software ...but more on this to follow in part 3.

AIMS at SAA

2011-08-16T19:21:00.004+01:00

Today's post is just a brief announcement...The AIMS team will be taking part in two events at next week's Society of American Archivists Annual Meeting. The first is a workshop we've developed to provide an opportunity for archivists and technologists to discuss issues related to collection development, accessioning, appraisal, arrangement and description, and discovery and access of these materials. Unfortunately, space issues have required us to limit registration and it is now full. However, we promise to post a longer recap to this blog after the event.

No such limitations exist for our other SAA event, a presentation entitled Born-Digital Archives in Collecting Repositories: Turning Challenges into Byte-Size Opportunities, which will be given August 27th at 8 a.m. At this presentation the AIMS Digital Archivists will describe a bit of the high-level framework being developed by the AIMS project to characterize archival workflows for born-digital materials in archival repositories.

We hope to see you there!

Digital Forensics for Digital Archivists

2011-08-12T16:27:00.013+01:00

I’ve been very fortunate here at UVa to have at my disposal some wonderful resources for getting up to speed with born-digital theory and practice. First and foremost, UVa is home to Rare Book School which has offered a course on Born Digital Materials for the past two years (and I’ve just learned will offer it again in 2012). I was able to take this course in July along with 11 fellow classmates from around the country. A week and a half later I was then off to the headquarters of Digital Intelligence, Inc. makers of our Forensic Recovery of Evidence Device (FRED) for Computer Forensics with FRED. This was a two day course covering basic digital forensic skills as well as the FRED system.

Mulder and Scully are concerned about the viability of this forensic evidence gathered next to UVa's FRED...

Given my great bounty, and my belief in professional karma, I’ve decided to give a brief overview of both of these classes here on the blog followed by my thoughts on a potential Digital Forensics for Archivists class/workshop that I’d really like to see developed, by myself or whomever! Two major classes out there that I have not taken are the DigCCurr Professional Institute and SAA’s electronic records workshop. Anyone with experiences in those classes, please add your comparisons in the comments.

RBS L95 — Born Digital Materials: Theory and Practice

Overall, I’d say this class has the perfect name: there’s an almost equal amount of theory and practice. That may sound like faint praise, but it’s really not. It’s something that too few workshops or classes get right. Instructors Naomi Nelson and Matt Kirschenbaum deserve much credit for a well constructed week that built practice on top of theory.

For someone new to the field of the born-digital it’s a great foundation. Concepts like metadata, preservation, “the cloud,” essential characteristics, physicality/materiality and digital humanities are combined with real-life examples from libraries, archives, and the university. This overview allowed us to attack the fundamental question of the class: what should we be trying to accomplish when we attempt to “save” (or steward, curate, safeguard, preserve, “archive”) born-digital materials.

On the practical side of things, digital forensics is covered and students get the opportunity to do a few lab exercises with emulators, floppy drives, and older models of equipment. The syllabus and reading list provide an excellent bibliography for further research.

It’s a relatively high-level class and therefore a great way to get started or a great way to get administrators thinking intelligently about the issues they need to face. I think that a more practitioner-focused and through digital forensics curriculum in the archives or cultural heritage setting could complement the course very nicely.

Computer Forensics with FRED training

University of Virginia decided to invest in the FRED technology last year and has not regretted it. While the FRED can do lots of neat things, I feel it is important to note that many or all of the same things can be done with other hardware and software, it just takes a bit more persistence. Similarly, despite the name a lot of this course dealt with basic data and file system concepts, as well as a little bit about some of the specific hardware most commonly found. In the future, DI is going to be splitting this up into two classes: Digital Forensic Essentials and Digital Forensics with FRED. The first part is a two day course and covers the hardware, data, and system stuff. The second is a one day class that covers the specifics of FRED. Although the first class will be more expensive than the current combined class is, it would be of more interest to those in the archival world.

As it is geared for law enforcement, a lot of time was spent on detected deleted, fraudulent, or hidden material. While all the cops in the room thought that this would be of no use to me, I disagreed. I need to know what I am collecting (whether inadvertent or not), whether it is authentic, and how to communicate with donors to decide how to deal with it. In addition, if we can get donors to agree to let us transfer backup or deleted versions of manuscripts, we’ll gain a wealth of information about how the final version evolved. Knowing that such recovery is possible is one of the more glamorous promises of digital forensics.

We also learned how to create and navigate disk images. While some of this stuff was fairly easy for me to pick up beforehand from Peter Chan’s tutorials, the extra practice and insight was very useful.

Digital Forensics for Archivists

Based on my experiences in these two classes, I would propose a Digital Forensics for Archivists workshop geared specifically for those interested in incorporating forensic techniques into the capture and processing of digital materials. The outline of topics I would expect to see on the syllabus below is probably a bit ambitious for a one-day workshop and would certainly have some hurdles to overcome related to provisioning hardware for all. However, these are the areas I’ve come to think of as necessary for an archive to be prepared for the variety of media that we will be collecting for the continuing future.

Digital Forensics for Archivists

Hardware basics

IDE, SCSI, SATA, USB, Firewire
Floppy drives
Optical disks
Hard drives
Internal basics (motherboard, pci, power, etc.)

Operating Systems

DOS
Windows
MAC OS
Linux

File system basics

NTFS

HPFS

Forensic vs. logical copying

What happens to deleted data

How it can be recovered

Why you need to know…

Write blocking

How to achieve it

Image files

Types

Software

Uses

Emulation and Migration

Cost/benefit of each

Possible use cases for each

So what do you think? Pipe dream? Useful? Impractical? Let me know in the comments…

Forensic workstation pt 1

2011-07-25T10:55:00.001+01:00

A key part of dealing with born-digital archives is the ability to receive and process material without making changes to the underlying metadata including date created, date accessed etc – data that researchers will be looking to use and rely on. As archivists we place considerable emphasis on our roles as custodians and with digital material it is important that we treat the material carefully and appropriately. Fortunately there are tools that help us with the authenticity of born-digital files the most obvious of which is the checksum.

An important legacy of the AIMS project for us at Hull is working towards our ability to take born-digital material from depositors as a normal part of our work. A key component of this is a forensic workstation – by which I mean a PC (or two) through which material can be safely captured following a clear process, in-effect replicating the isolation room for receiving paper material. This will allow us to undertake a forensic examination – to check the material is what we expected or agreed to take, including the ability to generate a manifest of the material to send to the depositor, and that it does not include viruses etc.

There seem to be two main routes – to purchase FRED which stands for Forensic Recovery Evidence Device (other digital forensic workstation solutions are available). A second and more organic solution, and the one we intend to adopt at Hull, is to start with a new PC and to add appropriate hardware and software to this to provide the equivalent functionality. At the moment we are pondering a name for this with current suggestions including:
- Hal - Hull Archives Laboratory
- Harold – Hull Archives Recovery Of Legacy Data
- Hilary - Hull Investigator for Library and Archives RecoverY
- Dawn – Digital Archives WorkstatioN
but we are open to other suggestions until the machine is installed and formally named!

We don’t want to become a computer museum with an extensive range of hardware, software and operating systems environments for any possible eventuality. We do want a core ability to handle material we reasonable expect to receive – including material on 3.5” floppy disks, zip disks, hard drives etc. We intend to develop and extend our capacity as need dictates – if we get material in a format we will consider whether we need to support this ourselves or whether a suitable 3rd party is more appropriate.

Central to this is the need for write blockers which prevent you from writing or updating the files. Having read countless websites I felt I knew what they were supposed to do but had a nagging doubt that my knowledge was incomplete.

A tour of the British Library eMss Labs courtesy of Jeremy Leighton John (as featured on the BBC Radio 4 programme 'Tales from the Digital Archives' broadcast in May but still available online) confirmed the simplicity of theory and the fragility of the media – just having the hardware isn’t enough – you also need some luck that you have the correct drivers to read the specific version of the media. In the next few weeks I hope to place our order for the various bits and pieces and will update you on this exciting journey!!

Curators Workbench workshop

2011-07-04T10:28:00.000+01:00

I was fortunate enough to attend the Curator’s Workbench workshop at the British Library last week. It was a chance to see, have a play and discuss the tool with its developers Greg Jansen and Erin O’Meara from University of North Carolina. The tool is designed to aid with the accession, arrangement, description and staging of material prior to ingest into a digital repository. Essentially the tool has an interface designed for archivists can use.

The session featured a walk-through and chance to have a play with experts on-hand if you had a problem – only necessary as we had latest ‘unstable’ release including the latest enhancements to functionality and GUI. Stable versions are available for download via GitHub. I am especially smitten with the crosswalk feature providing a drag’n’drop interface for mapping the metadata with METS. There is also the date recogniser which allows you to map the date format to the ISO standard, though there could be issues if the data is in a variety of formats, ie 1984 would be transformed to 01/01/1984.

It has a different take to where arrangement and description occurs in the workflow to that intended for Hypatia in the AIMS workflow, but it does raise some interesting questions that I hope to explore in more detail over the next few months.

It was also interesting to hear features and functionality on their wish-list including disc images, multiple users, recording processing notes, PREMIS and so the list goes on!
The discussion that followed was really enlightening as it highlighted the different approaches that archives are currently adopting to the preservation of born-digital archives.

I picked-up some useful pointers to software and tools I haven’t used before – Bulk extractor, Google Refine, and came away determined to throw more stuff at Curators Workbench, to join the users discussion list (done) and to figure out some of the aspects we have avoided so far things like PREMIS and METS etc !

Photographing the digital: creating images of Hull University Archives’ digital media

2011-06-21T12:12:00.000+01:00

A guest posting from Nicola Herbert, Digital Project Preservation Assistant at Hull University Archives

Over the last few months I have been working with the AIMS team at Hull University. My role entails getting stuck into some practical processing of the born-digital collections in the Hull University Archives as well as planning aspects of digital preservation. A lot of our work so far has been to discover and document the material that we already hold in what we thought were purely paper collections and I have written a workflow for the discovery of these items and their preparation for ingest into Fedora. As part of this workflow we decided to photograph all of the removable media we currently have and create a process for photography of new deposits when they arrive.

Why bother?
By retaining photographs of the original media alongside content we will be able to provide an image of the appearance of the original media to researchers if they request it. For the foreseeable future we are storing the image files on a shared drive, but they will eventually be stored as an element of metadata with the digital files in our Fedora Repository. We will be dealing with large numbers of media items so need to ensure consistency in the way the media is photographed and information recorded from those images.

Process
Having not previously numbered the discs, we decided on a simple running number within each accession. Despite our familiarity with labelling paper material, it seemed more complicated with digital. Our conservator advised against sticking labels (even conservation grade) onto the plastic casing of a floppy or Amstrad disc. Though a specialist CD marker can be used to label CDs, we were reluctant to permanently mark the items! After a worryingly long thought process we decided to stick to the old faithful method of writing in pencil on the existing label or case.

I then started planning the process. Despite trying to anticipate the different elements of information to include for each media type, it was only trial runs photographing actual media that gave the full picture - i.e. that Amstrad discs have three aspects to photograph (Side A, Side B and the edge). Lots of seemingly trivial questions arose - like whether to photograph the case or whether to photograph a label if blank. Getting the process right from the start will save time in the long run.

We decided to create a ‘clapperboard’ to photograph with the items for a failsafe way to ensure easy identification. I decided on a reusable form printed on a transparency which we can label with a drywipe marker. Putting theory into practice needed several trial runs; after each one I adapted the form and the procedure.

In addition I wrote up detailed notes describing the procedure for each type of media we anticipate encountering. We worked out a sensible image quality – so to ensure legibility of the labels without clogging up our servers with unnecessarily large images. Once the photographs have been taken they are renamed and filed. We also maintain an inventory of the items and record the media and label information alongside it. This ensures that if we send items (like our Amstrad discs) away to a third party we can match them to our records when they return.

This process has been satisfying to complete and enables us to tick at least one thing off our to-do list. Anyone can get this part of the process completed – even for material which is stored on a shared drive, photography of the original media is a useful process.

AIMS: the UnConference

2011-05-18T20:51:00.012+01:00

Not two full weeks into my new job as Digital Archivist at UVa on the AIMS grant, I rolled up my sleeves to facilitate and host an unconference with my fellow Digital Archivists. Our unconference would be two full days of discussions, demonstrations, lightning talks, and networking with digital archivists from around the globe. At first the thought was a little terrifying – I’m not even fully sure I know what this job is yet, how could I actually lead discussions on the salient topics? But my fears were baseless: all the unconference attendees were thoughtful, articulate, and lively participants. I learned much more from them than they probably did from me.

The unconference was held on the 13th and 14th of May at the Omni Hotel in Charlottesville. The 27 participants represented libraries, archives, museums, and digital humanities centers across the US, Canada, and the United Kingdom. Despite the differences in our institutions, backgrounds, and training, we learned that we not only shared similar challenges, but also the same hopes for collaboration and innovation.

The first day started off with a round of lightning talks. Each participant had 5 minutes to present a topic, project, problem or idea that they were interested in talking about. The variety in the talks was remarkable to me, traversing the breadth and depth of all that can be thought of as “born-digital” and the many processes involved in managing it. The lightning talks were also great way to get an introduction to each participant as well as their perspective or the particular issues they were dealing with in their institution. A brief outline of each of the talks is available on the AIMS Unconference Wiki.

After lunch all participants joined in their first unconference discussion based around topic proposals submitted beforehand on the wiki. Voting on which proposals were of interest took place during lunch, then proposals were grouped into one of 5 major themes which were discussed over the course of the next two days:

Management: Unconferencers tried to answer the question “how do we make born-digital materials workflow part of “the norm” in our profession?” Discussion touched on the need for institution-wide strategies, the viability of cross-institutional models, and generational issues. A fuller set of notes from the discussion can be found on the wiki at the link above.
Tools: Discussion on tools ranged from the abilities and limitations of specific tools to the broader idea of how to collaborate and build better tools. Specific areas of tool development that were discussed included the management of restrictions and forensic tools developed specifically for the archival community.
Access: Discussions on access continued to grapple with the issue of restrictions and how facilitate access while limiting abuse or misuse. Discussion moved on to questions of migration and emulation as well as the possibility of collaborating with researchers on analysis tools.
Acquisitions: The entire spectrum of archival processing was really discussed in the fourth session: how can we capture and represent the context of digital lives? Enhanced curation techniques were advocated while the limits and ethics of digital forensics were discussed.
Outreach: The final unconference discussion dealt with outreach through collaborations among institutions as well as collaborations with research communities. The group also touched on the importance of work with LIS programs and potential future archivists to ensure that they are coming to new professional positions prepared to deal with these questions.

In addition to all the discussions, Saturday morning was devoted to tool demonstrations and small discussion groups. Tools like Archivematica, Rubymatica, Curator’s Workbench and Forensic Tool Kit were demonstrated, while other groups talked about imaging optical media, self-deposit repositories, web-archiving, and the potential development of a new tool for processing digital archives.

After all that unconferenc-y goodness, the participants got together for a final discussion on next steps: where do we go from here? After talking through some of the options the participants agreed to try to continue to meet bi-monthly for informal phone conferences and to develop wiki pages for communicating about several broad areas: Curriculum Development, Best practices and Policies, Tool Development and Digital Research Communities. A specific suggestion was made to organize a “Day of Digital Archives” similar to the “Day of Digital Humanities” that’s become an annual event with our DH colleagues.

In all, the conference was lively, informative, and maybe even groundbreaking. If only all jobs started like this…

Workshop on "Using FTK Imager and AccessData FTK to Capture and Process Born Digital Materials"

2011-05-05T16:56:00.001+01:00

On April 22, I conducted a 2-hour workshop on "Using FTK Imager and AccessData FTK to Capture and Process Born Digital Materials.” The purpose of the workshop was to give staff a hands-on experience in using FTK Imager and AccessData FTK. Eight colleagues from the Stanford University Libraries attended the workshop – primarily from Special Collections and University Archives and the Humanities and Social Sciences Group.

The workshop covered the following:

FTK Imager – how to:
1. Download and install the software (free software - http://accessdata.com/support/adownloads).
2. Create a forensic image of an USB flash drive.
3. Create a logical image of the same flash drive.

AccessData FTK – how to:
1. Load an image – for this workshop we used a sampling from the Stephen Jay Gould papers.
2. View technical metadata generated by the software.
3. Arrange column settings to see specific file attribute (e.g. duplicate files).
4. Search for social security numbers using pattern search.
5. Test the full-text search function.
6. Flag files with sensitive information with "privileged" tag (such as those with social security numbers, etc.)
7. Use the bookmark feature for hierarchical information and apply it to groups of files (e.g. series, subseries, etc.)
8. Label groups of files with user defined labels (controlled vocabulary for computer storage media, document type suggested in the workshop, subject headings or access rights, etc.)
9. View files with specific bookmarks and labels.

Many incoming collections are hybrid collections – containing both analog and digital material. The digital component will become even greater as we move forward. Empowering all archivists to use a tool such as AccessData FTK to process the digital materials would be very useful.

Data Management Planning?

2011-04-08T19:20:00.005+01:00

Guest blogger: Andrew Sallans

Following on Tom's generous invitation to write a post for the AIMS partner blog, I am finally getting around to doing so. Tom and I have been holding monthly discussions about our respective projects since sometime last summer, and have talked in great length about the commonalities between what my group (the Scientific Data Consulting Group) is dealing with in regards to research data management versus what the AIMS group is dealing with in terms of born-digital archive material.

http://www.lib.virginia.edu/

We have found that there are many areas of similarity, and that we face many of the same challenges, although we approach the problem quite differently and of course have entirely different terminology given our relative perspectives.

To get started, I have a pretty good understanding of the born-digital problem set, but have not been keeping detailed notes on the workflows and solutions that the AIMS group has identified as best practices throughout the life of this project. My intention for this post is to share the issues that we are dealing with in research data management and try to make some suggestions around areas where there may be overlap and opportunities for great information sharing and collaboration.

Starting this past January 18, 2011, the National Science Foundation (NSF) put into effect a new implementation of their pre-existing data management planning requirement. This revision now requires that researchers submit a 2 page data management plan (DMP) that specifies the steps they will take to share the data that underlies their published results. This DMP will undergo formal peer-review, require reporting in interim/final reports, and all future proposals. In effect, what one says must then be done, or else one runs the risk of losing future funding opportunities or worse, losing all funding for the institution from that particular agency. Although this requirement is focused on data sharing, it isn't possible for such an initiative to succeed without first addressing a mass of other data management issues, ranging from technical, to policy, to cultural. As we often point out these days, it is far easier to improve the process of data management up-front, in the operational process phase, than it is to begin thinking about how to share the data at the end of the project. I would expect that those attacking the born-digital archive problem can fully relate.

Here in the Scientific Data Consulting (SciDaC) Group in the UVA Library, we have been collecting and developing our local set of data management best practices for some time and have served as advisors to researchers in both the research data management and DMP development areas (they are of course interrelated, but sometimes have different levels of urgency). In doing so, we have developed what we call a "data interview/assessment" (based in large part upon the visionary work of others, Purdue's Data Curation Profile and work from the UK's Digital Curation Centre, to name a few), which is a series of questions that address many different areas of data management, including context, technical specifications (formats, file types, sizes, software, etc.), policies, opinions, and needs. We meet with researchers to have a conversation, educate them on emerging trends and regulations in data sharing, and listen to their concerns and challenges. In the end, we try to make recommendations on how they can improve their data management processes, and then we offer to connect them with people who can help with the specific details (if it isn't us). For the DMPs, we have a series of templates that are specifically configured for the respective program requirements. Again in this case, we do some education, then offer some feedback and advising on what qualifies as good data management decisions for a particular community. Behind all of these efforts, we know we don't know all the answers, but we do know most of the questions to ask and who we need to pull together to figure out the solutions. That's our basic operating principle.

So, sound a bit familiar? Based on conversations with Tom, and reading some of the posts in the AIMS blog myself, it sounds like we are up against some very similar challenges in regards to the front-end of the issue, around education, conducting inventories and assessments, and figuring out how to manage processes before it comes down to managing the information itself and providing access to it for others. Appraisal and selection is incredibly important to us, but is usually driven more by the type of data. As an example, reproducible data generated by a big machine might not be important to keep, but the instructions and context in which it is generated would be invaluable. On the other hand, data from natural observations (ie. like climate data) would be critical to save. These considerations are not always apparent to the researcher, as they often think within the context of their work, rather than others. I would expect that the back-end is even more similar, as we are all ultimately dealing with bits and bytes, formats, standards, and figuring out how to decide what to keep and how to do it.

Lastly, for now, I also would like to mention that I had the opportunity to attend the annual Duke-Dartmouth Advisory Council meeting at the Fall CNI Forum several months ago.

http://www.dartmouth.edu/~vox/0708/0218/infomgmt.html

As you'll read, this project aims to bring together stakeholders from all areas of digital information across the institution, to talk about and plan in a collaborative and strategic way. They aim to tackle the challenges of management, technology, policy, and hardest, culture. I was incredibly impressed by the vision of this undertaking, and hope that we can continue to refine our efforts at developing a collaborative digital information management strategy as well. In practical terms, we all need to try and be attentive to how our effort plugs-in with others around the institution. The issue of digital information management is undoubtedly a very big one, and requires coordination and collaboration across many experts in order to appropriately treat the various bits that we encounter. Doing so will hopefully also provide us with the ability to bring best practices from one challenge to another.

Andrew is currently the Head of Strategic Data Initiatives and the Scientific Data Consulting Group at the UVA Library.

Contact info: Andrew Sallans, Email: als9q@virginia.edu, Twitter: asallans

Digital Collaboration Colloquium

2011-04-01T10:04:00.000+01:00

On Tuesday I attended the Digital Collaboration Colloquium event in Sheffield organised to mark the end of the White Rose Libraries LIFE-SHARE Project.

The day included a number of talks about how institutions can collaborate including an interesting account of the Wales Higher Education Libraries Forum (WHELF) and experiences from the Victoria & Albert Museum. Although the majority of examples focussed on digitisation the principles and lessons learnt were all equally appropriate to a born-digital context.

As part of the day I presented a Pecha Kucha session on the AIMS project and some of the digital collaboration tools that we have found to be effective including Skype and GoogleDocs. In you are not familiar with this format it involves a presentation of 20 slides, changing automatically every 20 seconds and despite cutting the content quite heavily I still found myself chasing to keep-up with the changes. Other sessions looked at digitisation in-situ in a public setting – bringing behind the scenes in-front of the curtain, and other sessions on the Knitting patterns project at Southampton, the Addressing History project based at EDINA and the Yorkshire Playbills project.

The afternoon included a presentation form our hosts on the LIFE-SHARE project and their experiences of the collaboration continuum and a roundtable session that led to a good discussion between panel and audience. With alot covered in a relaxed and friendly atmosphere there was plenty of networking and I’m sure everybody took something from the day.

The presentations are available via slideshare

Personal Digital Archiving Conference 2011

2011-03-18T19:59:00.002Z

I had the good fortune to attend the 2011 Personal Digital Archiving conference at the Internet Archive, along with other colleagues on the AIMS project, including Michael Forstrom from Yale and Michael Olson, Peter Chan, and Glynn Edwards from Stanford. The conference was exceptional, and had a great range of presentations ranging from those on fairly pragmatic topics to the highly theoretical. There are a number of other blogs with comprehensive notes on the conference, and the conference's organizers have already provided a detailed listing of those. Instead, I'd just like to focus on what I considered the highlights of the conference.

Cathy Marshall's keynote was excellent. I have seen her speak before, and she presented an survey of her ongoing research into personal digital archives.
Jeremy Leighton John presented on work undertaken since the Digital Lives project at the British Library.
Judith Zissman presented on "agile archiving", similar to agile development, wherein individuals can continually refine their archival practices.
Birkin Diana presented on how Brown University is working to make their institutional repository a space for personal materials, and strategies that allow users to work on adding metadata iteratively.
Daniel Reetz presented on his DIY Book Scanner project, but also brought in detailed technical analysis about how image sensors in digital cameras work and how our brains process image data.
Jason Zalinger introduced the notion of Gmail as a "story-world" and presented some prototype tools and games to help navigate that world.
Cal Lee presented on introducing education about digital forensics to the archival curriculum.
Kam Woods also presented on applying digital forensics to the archival profession.
Sam Meister presented on the complex ethics of using forensics in acquiring and processing the records from start-up companies.

In addition, I presented with Amelia Abreu on "archival sensemaking", which introduces the notion of personal digital archiving practice as an iterative, context-bound process.

Processing Born Digital Materials Using AccessData FTK

2011-03-12T00:59:00.001Z

To follow up with my previous blog entry on "Surprise Use of Forensic Software in Archives", I have prepared a YouTube video "Processing Born Digital Materials Using AccessData FTK". I hope this video can give people more details on how FTK is being used at Stanford University Libraries. Take a look and let me know what you think.

I would like to say a few words on discovery and access even though it is not the topic of the video. After we process the files in FTK, one way to delivery the files is to store them in a Fedora repository and let people access our Fedora repository using a web browser through Internet. We have developed an alpha version of this model using files from the Stephen Jay Gould collection. Another way to provide access to the files is to let people use FTK to access the files in our reading room. I will write about that later.

Hope you enjoy the video.

http://www.youtube.com/watch?v=hDAhbR8dyp8

File type categories with PRONOM and DROID

2011-03-04T15:45:00.006Z

In order to assess a born digital accession, the AIMS digital archivists expressed a need for a report on the count of files grouped by type. The compact listing gives the archivist an overview that is difficult to visualize from a long listing. The category report supplements the full list of all files, and helps with a quick assessment after creation of a SIP via Rubymatica. (In a later post I’ll point out some reasons why pre-SIP assessment is often not practical with born digital.)

At the moment we have six categories. Below is a small example ingest:

Category summary for accession ingested files

data	3
moving image	1
other	2
sound	2
still image	26
textual	12
Total	46

Some time ago we decided to exclusively use DROID as our file identification software. It works well to identify a broad variety of files, and is constantly being improved. We initially were using file identities from FITS, but the particular identity was highly variable. FITS gives a “best” identity based meta data returned by several utility programs. We wanted a consistent identification as opposed to some files being identified by DROID, some by the “file utility” and some by Jhove. We are currently using the DROID identification by pulling the DROID information out of the FITS xml for each file. This is easy and required very little change to Rubymatica.

PRONOM has the ability to have “classifications” via the XML element FormatTypes. However, there are a couple of issues. The first problem is that the PRONOM team is focused primarily on building new signatures (file identification configurations) and doesn’t have time to focus on low priority tasks such as categories. Second, the categories will almost certainly be somewhat different at each institution.

Happily I was able to create an easy-to-use web page to manage DROID categories. It only took one day to create this handy tool, and the tool is built-in to Rubymatica. The Rubymatica file listing report now has three sections: 1) overview using the categories 2) list of donor files in the ingest with the PRONOM PUID and human readable format name 3) the full list of all files (technical and donor) in the SIP.

This simple report seems anticlimactic, but processing born digital materials consists of many small details, which collectively can be a huge burden if not properly managed and automated. Adding this category feature to Rubymatica was a pleasant process, largely because the PRONOM data is open source, readily available, and delivered in a standard format (XML). My thanks and gratitude to the PRONOM people for their continuing work.

http://www.nationalarchives.gov.uk/PRONOM/Default.aspx

http://droid.sourceforge.net/

As I write this I notice that DROID v6 has just been released! The new version certainly includes a greatly expanded set of signatures (technical data for file identifications). We look forward to exploring all the new features.

Arrangement and Description of born-digital archives

2011-02-22T15:59:00.000Z

For the last two months the Digital Archivists have been trying to define the requirements of a tool to enable archivists to arrange and describe born-digital archives. To do this we have stood-back and reviewed the traditional skills and processes and whether changes are required or appropriate to accommodate the particular issues surrounding born-digital archives.

The components we identified were as follows:
• Graphical User Interface – needs to be clean and easy to use
• Intellectual Arrangement - must be easy and instinctive for archivists to use
• Appraisal – born-digital archives need to be appraised as much as their paper predecessors
• Rights and Permissions – to enable the management of access to the born-digital archives and also to demonstrate to 3rd part depositors that the material is safe in your care
• Descriptive Metadata – a term we have been using to relate to description information and to explicitly distinguish this from the technical metadata about each file
• Import/Export functionality – to import/export data with other tools
• Reporting – to provide a range of "views" for managing the digital assets

Through a series of user stories and scenarios we have sought to clearly explain the requirement and how this might relate to other functionality.

This work has been under-taken predominantly through the use of GoogleDocs and created a document that we can all access and edit, create diagrams and include screenshots as necessary. Over weeks hundreds of comments have been added, and the text subjected to a comprehensive review and refinement process by numerous staff across the four partners.

Each institution has now scored and prioritised these features and as befits a collaborative initiative like the AIMS project allow us to identify a core group of features and functionality that we feel will be of greatest use to our institution and the wider archival community.

With the exception of intellectual arrangement most tasks and processes are not unique to archives so there is already a body of knowledge and experience in how to approach the task. For intellectual arrangement we have to be clear and precise about what we need and want we didn’t, for example a single intellectual arrangement when multiple versions would be possible in a digital environment.

Over the next few months we will be refining and reviewing these requirements, very much aware that there are only seven months of the project remaining. We also intend to discuss those aspects we identified as "critical" in future blog postings.

Tell us what tools you use with born-digital archives...

Surprise Use of Forensic Software in Archives

2010-11-19T18:08:00.008Z

When I first heard of the use of computer forensic in archives, I was excited and wanted to learn how these law enforcement techniques could help me do a better job in processing digital collections. After learning people are using computer forensic to copy disk image (i.e. an exact copy of a disk, bit by bit) and to create a comprehensive manifest of the electronic files of collections, I was a bit disappointed because software engineers have been using the Unix dd command for many years to copy disk images. Also, there are tools (e.g. Karens's Directory Printer) available to create comprehensive manifest of the electronic files of collections. Data recovery is another feature of forensic software some people consider useful for archivists/researchers. In my opinion, data recovery may be useful for researchers but not archivists. Without informed written consents from donors, archivists should NOT recover the deleted files at all. Also, in some cases, a deleted file doesn't appear as one file, but instead, tens or hundreds of segments of files. When most archivists don't do item level processing in paper collections due to limited resources, I can't image archivists performing sub-item level processing in digital collections. Computer forensic in criminal application usually look for particular evidences. Organizing all files in a disk drive is usually not their interest. In archives, we are organizing all files in disk drives, looking for particular items are not our duty. Computer forensic may be more useful for researchers when they want to look for particular items. All these lead me to the conclusion that computer forensic may not be very useful for digital archivists.

However, after attending a 2.5-days training on AccessData FTK (a computer forensic software), I started to see the potential of using forensic software to process digital archives. I found out that the functions (bookmarks, labels) which help investigators to organize the evidence they selected are equally applicable to the organization of the whole collection. The functions (pattern and full text search) which are used to found particular evidence are equally applicable to search for restricted materials. I can also use the software to replace a number of software I am using to processing digital collections. Although, 90% of the training is related to cracking passwords, searching on delete files, identifying pornographic images, etc., I found the 10% the course worth every cents Stanford spent on it. Of course, the ideal case would be a course tailored for the archival community, but unfortunately, there is no such course exists.

Now, I am using AccessData FTK to replace the following software I used in the past to process digital archives.
Karens's Directory Printer - to create a comprehensive manifest of the electronic files of collections
QuickView Plus - to view files with obsolete file formats
Xplorer - to find duplicate files, copy to folders
DROID, JHOVE - to extract technical metadata: file formats, checksums, creation/modification/access dates
Windows Search 4.0 - to perform full text search on files with certain formats (word, pdf, ascii);

I am using following functions, which I have not found software package to perform in a very user friendly manner, in AccessData FTK to process digital archives.
Pattern search (to locate files containing restricted information such as social security no, credit card no., etc.)
Assign bookmarks, labels to files (for arranging files into series/subseries, other administrative and descriptive metadata)
Extract email headers (to; from; subject; date; cc/bcc) from emails written in different email programs for preparing correspondence listing.

The cost of licensing the software seems high. But if you look at the total costs of learning several "free" software, the lack of support for such software, and the integrated environment you get in using on software, you may find the total costs of using commercial forensic software is cheaper than using "free" software.

Other Highlights from the DLF Fall Forum

2010-11-16T01:05:00.002Z

A few weeks ago I got the opportunity to attend the Digital Library Federation's Fall Forum in Palo Alto, California. This is the same conference for which Peter previously announced his session on born digital archives. In addition to Peter's session, there were a number of other sessions that were of interest to those working with digital archives.

I attended the working session on curation micro-services, led by Stephen Abrams of the California Digital Library, Delphine Khanna from University of Pennsylvania, and Katherine Kott from Stanford University. (Patricia Hswe from Pennsylvania State University was supposed to be one of the discussion leaders, but she was unable to attend the Fall Forum.) The micro-services approach is a philosophical and technical methodology for the architecture of digital curation environments. This approach values simplicity and modularity, which allows "minimally sufficient" components to be recombined. Furthermore, the strength of curation micro-services is the relative ease by which they can be redesigned and replaced as necessary. The slides from the beginning part of the session can be found here.

There was also a reading session at the DLF Fall Forum on "Reimagining METS." The session's discussion revolved around ideas put forth in a white paper distributed in advance of the conference. The majority (if not all) of the METS Editorial Board facilitated the discussion, which was very high level and incredibly interesting. Much of the discussion seemed to imply the requirement that METS actually needed to change. The most interesting potential idea that seemed to get a fair amount of traction was to consider whether METS should focus on its strength in packaging and abdicate some of its functionality to other standards that arguably do it better (e.g., OAI-ORE for structure).

On the last day, I went to the workshop on JHOVE2, which is the successor project to the JHOVE framework for characterization. JHOVE2 has an notably different architecture and expanded feature set, which expands characterization to include other processes, including identification, validation, feature extraction, and assessments based on user-defined policies. Additionally, users will be able to define format characterization and validation files for complex digital objects, such as GIS shapefiles. The presenters stated that future development for JHOVE2 will include a GUI to assist in rule set development. From the standpoint of a digital archivist, this tool will be essential in any of the further work that we do.

Donor survey web form is ready

2010-11-10T14:16:00.011Z

As part of our on-going work to ingest born-digital materials, we have implemented a donor survey web form. In the evolving AIMS work flow, the donor survey occurs pre-ingest. The data collected in the survey may become part of the Submission Information Packet (SIP). This is our first attempt at testing several ideas. We expect to make changes and we invite comments. Our donor survey web site is open to archivists and other people interested in born-digital work flows. See the account request information below.

I realized I could quickly adapt an existing application as a donor survey, if the application were flexible enough. A couple of years ago I created a mini Laboratory Information Management System (LIMS). The programmer can easily modify the fields in the forms, although users cannot add fields ad-hoc. The mini LIMS has its own security and login system, and users are in groups. For the purposes of the AIMS donor survey, one section of the LIMS becomes the “donor info”, another section becomes the “survey”.

Using the list of fields that Liz Gushee, Digital Archivist here at UVa, gathered while working with the other AIMS archivists, I put the donor’s name, archivist name and related fields into the “donor” web form. All the survey questions went into the “survey” web form. The survey will support distinct surveys from the same donor over time to allow multiple accessions.

Our next step will be to map donor survey fields to various standard formats for submission agreements and other types of metadata. While the donor survey data needs to be integrated into the SIP workflow, we haven’t written the transformation and mapping code. We are also lacking a neatly formatted, read-only export of the donor survey data. Our current plan is to use Rubymatica to build a SIP, and that process will include integration of the donor survey data. The eventual product will be one or more Hydra heads. Learn more about Hydra:

https://wiki.duraspace.org/display/hydra/The+Hydra+Project

Everyone is invited to test our beta-release donor survey. Please email Tom twl8n@virginia.edu to request an account. Include the following 3 items in your email:

1) Your name
2) A password that is not used elsewhere for anything important
3) Institutional affiliation or group so I know what group to assign you to, even if your group has only one person.

I'll send you the URL and further instructions.

On the technical side, there were some interesting user interface (UI) and coding issues. Liz suggested comment fields for the survey, and we decided to offer archivists a comment for every question. I used Cascading Style Sheets (CSS) to minimize the size of comment fields so that the web page would not be a visual disaster. If one clicks in a comment, it expands. Click out of it, and it shrinks back down.

The original LIMS never had more than a dozen fields. The donor survey is very long and required some updates to the UI. By using a JQuery function ajax() I was able to create a submit button that would save the survey without redrawing the web page. CSS code was required to make the save buttons remain in a fixed location while the survey questions scrolled.

The mini LIMS relies on a Perl closure to handle creation of the web forms, and saving of the data to the database. Calling functions from the closure creates the fields. A closure is similar to a class in Java or Ruby, but somewhat more powerful. The data structures from the closure are passed to Perl’s HTML::Template module to build the web pages from HTML templates.

The mini LIMS was originally created to use PostgreSQL (aka Postgres), but I’ve converted it to us SQLite. SQLite is a zero-administration database, easily accessed from all major programming languages, and the binary database file is directly portable to all operating systems. Perl’s DBI database connectivity is database agnostic. However, some aspects of the SQL queries are not quite portable. Postgres uses “sequences” for primary keys. Sequences are wonderful, and vastly superior to auto-increment fields. SQLite does not have sequences, so I had to create a bit of code to handle the differences. The calendar date functions are quite different between Postgres and SQLite, so once again, I had to generalize some of the SQL queries related to date and time.

There was one odd problem. Postgres and SQLite are both fully transactional. However, because SQLite is an “embedded” database, it cannot handle database locks as elegantly as client-server databases. Normally this is not a problem, and all databases have some degree of locking. In this instance I got a locking error when preparing a second query in one transaction. Doing a commit after the first query appears to have fixed the problem. I’ve worked with SQLite for several years an never encountered this problem. It could be a bug in the Perl SQLite DBD driver. Using the sqlite_use_immediate_transaction did not solve the problem.

The mini LIMS source code is available.

Digital Library Federation (DLF), Fall Forum, 2010

2010-10-29T17:51:00.015+01:00

Stanford, Emory and Duke will host a join forum discussing "Archiving" Digital Lives: Choices, Challenges, and Change in the up coming DLF Forum Nov. 1-3, Palo Alto. The panel will use ongoing work with the Salman Rushdie archive and Stephen Jay Gould archive as case studies. We expect a discussion mode in the forum. Anyway, I have prepared some powerpoint slides to show the workflow we developed at Stanford. Session leaders for our forum are:

Duke University: Naomi Nelson
Emory University: Erika Farr, Peter Hornsby
Stanford University: Peter Chan, Glynn Edward, Michael Olson

Update on the Donor Survey

2010-10-26T19:46:00.003+01:00

As our readers may recall, this past July, the AIMS archivists created a donor survey for born-digital archives. My colleague, Peter Chan, wrote fairly extensively on its origins and purpose; please go here to read up on the background of our survey.

A few months have gone by and the archivists have had the opportunity to think more about how we envision the donor survey fitting into both shared and institution-specific born-digital workflows. First of all, we all agreed that we wanted to move away, as much as is possible, from continuing to create paper-based forms and records regarding donors and content. Moving the donor survey to a web-based tool, complete with an SQLLite database back-end, seemed to be a good way to start (for technical specifics, please see Tom's forthcoming entry regarding the web form - coming up next!). In the web-based survey, we deliberately included a space for the archivist to record comments for each question and answer on the survey. We realized that by creating a place for the archivist to record their findings and/or elaborate on what was recorded by the donor/owner of the personal archive, we could make the process of determining the scope of the personal archive for transfer that much more transparent. As one of our senior archivists on the project pointed out, it's as important to know what was excluded from transfer and why as to have a trail of documentation as to what was transferred and why (especially if the processing of the collection follows many months later!). We hope that adding this feature to the survey will help with the recording of that process in a centralized location and perhaps serve as the digital equivalent to a donor file.

As to how the donor survey fits into our shared and institution-specific workflows, that is still a work in process. Generally speaking, it is intended that the data collected from the survey could be mapped to a submission agreement, which, in turn, would then be part of the SIP (submission information packet). We also intend to map portions of what had been collected from the survey and submission agreement in Archivists' Toolkit and Calm (collection management software from the UK) to form an accession record. Ideally, we want to have to enter/create data once and have it re-purposed as often as is needed throughout our workflow.

We invite you to test out our web survey and to give us your feedback. In our next entry, Tom will be posting a description of the technical side of the survey web form and he'll include a link for access. Other folks have been working on other versions of surveys for electronic records as well. If you're not already familiar with Chris Prom's blog, Practical E-Records, get a readin'. Chris recently posted a version of a donor survey; check it out here.

Liz Gushee
University of Virginia

Rubymatica as web application

2010-09-30T20:06:00.006+01:00

With a working command line script, it was time to create a web interface. The first step of this was to modify my main processing method so it could be called from an external script. Initially, Rubymatica processed every ingest (folders) in the origin folder. That script was upgraded to accept a command line parameter and only process a single ingest. Internally, the code was refactored in order to abstract processing into a sub-step after initial setup.

Ruby on Rails is a wonderful way to build web sites. Rails is convoluted, but there are many good examples on the Web. Basically, I followed one of the “hello world” examples, and added a method to the hello_world_controller to call the same entry point as the command line wrapper. This worked well, although it was processing the ingest in real time, so it was obvious that sooner or later the processing would have to be made asynchronous.

The web interface also needed code to report on the status and results of ingests in the system. The web site has been kept as simple as possible because we will eventually be using Rubymatica as a backend for a Hydra head. Even so, the simple web site needs several controller methods: offer_upload, do_upload, save_file, full_status, reset, get_file, get_log_xml, process_sip, show_logs, file_list, and report. As of this writing Rubymatica is unable to create a BagIT bag, so there will be at least one more method added to the controller.

Changing Rubymatica to become an asychronous process was interesting (at least to a programmer). In this sense “asynchronous” means execution of a program as a background task with no window or terminal session. The main web page has a link for pending ingests. Clicking one of these links starts the process. The program begins to run and is “forked” to a separate task. This forking happens very quickly. You see the main web page refresh with a status message that says (essentially) “Processing has started”. Meanwhile, the background task runs independently taking as long a necessary. This is important for two reasons. First, web browsers will “time out” after a minute or two giving you a mostly empty window with an error message. Web browsers can’t deal with long-running tasks. Secondly, programmers don’t want to lock the user into doing nothing while the independent background task runs. The user doesn’t need to stare at a blank page waiting for the task to complete. (As an aside, Linux programmers call tasks “processes”, but since the word “process” has many meanings in the mixed world of digital accessions, I’ve change it to “task” throughout this blog.)

Normally, (in Perl or Python with the Apache web server) programmers simply fork. However, Ruby is often run in a special web server, and forking actually forks the web server, not just the Ruby on Rails controller. That is bad. Fortunately, it was easy to “exec” the task, which means that a new task is created in the background. Using the ingest name supplied from the web request I simply exec’d a task using the command line that I had created earlier. This is simple, elegant, and robust.

You may be wondering how the web site is able to know the status of background tasks. In order to fill this need, the background task writes status messages (and some other administrative meta data) into a SQLite database which is present in each ingest. One of the web controller methods queries each ingest’s database, and displays the most recent status message in the main web page. Another controller reports the full list of status messages for a given ingest.

Rubymatica is near maturity. It has (generally speaking) an architecture typical of most web applications: code, HTML templates, and a SQL database. The code is Ruby on Rails. The HTML templates used by Rails are Ruby erb files. The SQL is a SQLite database in each ingest directory. Eventually there might be a SQL database with system-wide Rubymatica information, although SQLite will happily open multiple databases so a system wide database may not be necessary.

Rubymatica needs to gain a few features to be considered a SIP creation tool, and I’ll cover those in a later blog posting. However, there is one more interesting technical problem with checksums and ingests.

Normal practice validating files is to generate a file that is a list of checksums for all the files in a directory tree. We typically put the checksum file in the root directory of the directory tree. However, a real-world ingest may include a pre-existing checksum file, changing file names (due to detox), removed files (due to anti-virus filtering), and new files from extracting .tar and .zip files. We need to detect an existing checksum file, modify it as file names change, modify it when files are moved/deleted, and add to it when we extract .tar and .zip files.

Links:

http://en.wikipedia.org/wiki/Ruby_on_Rails

http://en.wikipedia.org/wiki/Fork_(operating_system)

http://en.wikipedia.org/wiki/Sqlite

https://wiki.duraspace.org/display/hydra/The+Hydra+Project

http://en.wikipedia.org/wiki/Checksum

http://en.wikipedia.org/wiki/Tar_(file_format)

Archivematica SIP ported to Rubymatica

2010-09-30T19:58:00.004+01:00

I thought some of you might enjoy a look inside the mind of a software developer converting key scripts from the Archivematica project into Ruby. The impetus of the conversion was to have a Submission Information Package (SIP) creation tool written in Ruby and ready to be integrated with a suite of web applications. Ruby is the language of choice at the University of Virginia (UVA) Library and we are in the process of rolling out a Hydra/Hydrangea web application technology stack. Archivematica is written in Python and shell scripts, and while a web interface is planned, it was months away and (as far as I know) will not be using Hydra. At UVA, the processing steps beyond SIP creation will happen in other systems, so scope of the conversion fit into a 2 to 4 week time line. After checking Google for existing projects, we settled on the name “Rubymatica”.

I am very grateful to the Archivematica developers for creating a working product based on Linux. Archivematica uses a “desktop explorer” user interface where moving an ingest to specific folders causes the Archivematica processing scripts to run. This architecture is easy to understand, and fairly easy to port to another programming language.

For the first phase, I rewrote shell and Python scripts in Ruby as a single group of methods in one script with a single entry point. I ran my script from the command line. I had to work out origin and destination folders, because Archivematica moves the ingest into a temporary folder, and moves it a second time to a final destination. Since I would have a unique subfolder for each ingest, I didn’t need an intermediate temporary folder. I made additional modifications to the Archivematica directory structure for clarity and programming sanity. For instance, Archivematica creates some meta data files in the same directory as the ingested files. My code was simpler if the meta data for a given ingest always resided in it’s own folder. (Early versions of Rubymatica were processing the meta data files as part of the ingest because the files were mixed together in the same directory.)

Ingesting a collection of files involves traversing the directory tree of the ingest. This traversal happens several times. For example, .tar and .zip files need to be extracted, therefore the directory tree is traversed searching for files to extract. This process is iterative in that each newly-extracted directory must be traversed. Archivematica uses the Python script called Easy Extract. I recoded Easy Extract as a recursive Ruby method, which was nontrivial and required a couple of days work. The directory tree also has to be traversed when calling “detox” to clean file names, and traversed again for virus checking with ClamAV. To keep my sanity, I created a method specifically to traverse the directory tree. In keeping with the theme of clarity and simplicity of algorithms, the directory tree is crawled several times rather than trying to do everything in one pass. This works well and fits the programmer mantra: avoid premature optimization.

Creating the METS XML took several days of difficult work. I used Nokogiri’s Builder class since it has powerful XML tools. Nokogiri is a Ruby module or a “gem” in Ruby parlance. The Builder class of Nokogiri is wonderfully powerful, but almost entirely undocumented. Archivematica uses a Python class to build the METS XML, but the parallels between Python’s XML tools and Nokogiri Builder are tenuous. With some help from the very bright, talented, and helpful Andrew Curley here at UVA, I was able to bend Nokogiri Builder to my will. It was a huge battle and Nokogiri nearly crushed me. I’ve created example code for Nokogiri Builder which can be found at:

http://defindit.com/readme_files/nokogiri_examples.html

Now that I have a working command line script, it is time to think about creating a web interface. I’ll cover that process in my next blog.

Links:

http://archivematica.org/wiki/index.php?title=Main_Page

http://en.wikipedia.org/wiki/Ruby_(programming_language)

http://en.wikipedia.org/wiki/Tar_(file_format)

http://en.wikipedia.org/wiki/Python_(programming_language)

http://en.wikipedia.org/wiki/Shell_script

http://en.wikipedia.org/wiki/Clam_AV

http://en.wikipedia.org/wiki/Directory_tree

Can you read 5.25 inch double sided / high capacity IBM / MS DOS formatted diskettes?

2010-08-25T21:19:00.016+01:00

I believe many archives have old computer storage media such as 5.25 and 3.5 inch. floppy diskettes and zip disks in their collections. Since you can still buy 3.5 inch. floppy disk drives with USB interface in the market, reading 3.5 inch. floppy diskettes is not a problem. Also, because some Zip drives use ATAPI / USB interface, connecting a zip drive to your pc is not a problem either (ATAPI is still widely used to connect CD/DVD drives.). The problem is 5.25 inch. floppy disk drives which use 34-pin floppy disk drive connectors (NO USB/ATAPI version). The 34-pin floppy disk drive connector doesn't exist in the motherboards of some modern personal computers (e.g. the FRED we have at Stanford.). Even a motherboard has a 34 pin floppy disk drive connector, some BIOS can recognize 3.5 inch. floppy disk drive but not 5.25 inch. floppy disk drive (e.g. the retired Dell pc in my offce).

In order to read 5.25 inch. double sided or high density IBM / MS DOS formatted diskettes, one option is to get a Catweseal card to put in a spare PCI slot in your pc. The Catweseal card has a 34-pin floppy disk drive connector for you to connect to a 5.25 inch. floppy drive. According to the manual, it can be configured to read Commodore 64 disk, extended density format disk (2.88MB), CP/M format disk (8-inch floppy for PDP-11 machines), IBM / MS DOS format disk. One limitation is that you have to use the "Imagetool" software come with the Catweseal card to read and write disk images. You cannot browse the contents of a disk and see what is inside before you make the disk image. You have to create the disk image and export the files in the disk image to see the contents of the disk. Also, Imagetool does not have an option to create logical disk image (slack space not captured / deleted files not copied). At Stanford, we create mostly logical disk images. Another option is to get a pc with motherboard and BIOS capable of connecting and recognizing 5.25 inch. floppy disk drive. I checked the specifications of several new pc in the market and found no mention about 5.25 inch. floppy disk drive (I will be very surprised if I find the specifications mention that.). I have checked the specifications of new motherboards to see whether they can connect and recognize 5.25 floppy disk drives. (If yes, I can buy the motherboard and build a pc around it.) The specifications will tell you whether the motherboards have a floppy disk connector but no details on whether they recognize 5.25 floppy disk drives. One day, I opened a retired pc of mine and discovered that it has a 34 pin floppy disk connector in the motherboard. I bought the pc to my office and connected the 5.25 floppy disk drive to the motherboard and it WORKED!!!! I can see the contents of 5.25 inch floppy disks using Windows Explorer. And I can use FTK Imager to create logical disk images. After that, I connected an ATAPI Zip drive (taken from the retired Dell pc in my office) to the same machine and it became a standalone capture station for me!!

For people who don’t have 5.25 inch drive in their office, they can try to get one from eBay. But keep in mind that the drives you bought from eBay may be more than 20 years old and they may stop working anytime. In fact the two 5.25 inch. floppy disk drives Stanford bought from eBay last year have stopped working. Another source of 5.25 inch. floppy disk drives is people around you. They may have one in their garages. In fact, I mentioned Stanford had many 5.25 floppy disks and the problems we were facing to a manager from Konica Minolta in a conference in July. He told me that he had one 5.25 inch. floppy disk drive in his office and asked whether I wanted it or not!! I have just received the drive on last Fri. and it is now working for me :)

CALM Digital Records meeting

2010-07-28T12:03:00.005+01:00

The last week has been a busy one with two deposits of born digital material (16GB representing more than 27,000 digital files) and a meeting with users of the CALM software to discuss digital records.

Adrian Brown (Parliamentary Archives) convenor of the meeting, hosted by The National Archives, reported on the main findings from a survey of CALM users conducted at the end of 2009. It was clear from the meeting that many archivists were actively investigating the options and issues surrounding a digital repository but that the lack of a digital repository with-in their organisation and the need for training were huge obstacles that needed to be overcome.

I gave a brief outline of the AIMS project and presented a diagram to highlight our current thinking about how Fedora and born-digital material can be integrated into our workflows. [This model is currently still conceptual but we will be working with Axiell to progress this – comments welcomed]. Natalie Walters (Wellcome Library) highlighted their work and how she had found that many of the professional archive skills used to handle and manage paper archives still applied in the born digital arena. Malcolm Todd (The National Archives) talked about four key aspects to digital repository technology modularity, interoperability, sustainability and cost effectiveness all of which are being actively embraced by the AIMS project.

Malcolm Howitt and Nigel Pegg (Axiell) spoke about their plans to extend CALM to link to digital repositories and it is hoped that we can work closely with them on this.

The rest of the meeting was spent discussing and identifying issues surrounding cataloguing and metadata; accession and ingest; user access and best practice. A number of common themes emerged:
• That the differences between paper and digital archives are often exaggerated with issues like provenance and integrity key to both
• That depositor’s perception of digital archives is very different to paper and that by acting promptly archivists is the only way we can avoid technological obsolesce and a digital dark hole in the historical record
• The need for archives staff to be actively involved in the digital repository and not leave it for ICT staff to develop/manage exclusively
• That born digital archives may open-up the archives to new audiences
• A desire to share experiences, documentation etc for the wider benefit of the profession
• A need for more opportunities for “hands-on” experiences with born-digital archives and repository to increase familiarity with-in the archives profession

Introduction

2010-07-27T15:00:00.000+01:00

The Born Digital Archives is the blog of the AIMS team. We hope to stimulate dialog about practical solutions to archiving materials that originate in digital form. We invite all concerned archivists to chime in with questions and comments via the "comments" to each post. AIMS is inclusive with the intention to create open source solutions that are useful to both small and large institutions.

AIMS is “Born Digital Collections: An Inter-Institutional Model for Stewardship". Funded for two years by The Andrew W. Mellon Foundation, the four partners are The University of Virginia Library, Stanford University, the University of Hull, and Yale University. The purpose of AIMS is to implement best practices. AIMS will create and deploy open source software to manage the all steps in the acquisition, conservation, and eventual dissemination of digital collections. Given that we all have somewhat different administrative, hardware, and software needs, AIMS will strive to use portable open source tools which integrate reasonably seamlessly, and allow archivists a flexible workflow. We plan to process several important collections of born digital media, using Hydra to handle discovery. Hydra is based on the Fedora Commons Repository Software.

The AIMS digital archivists have the mandate to nurture a global community by publishing our lessons online, writing manuscripts, attending conferences, and generally being evangelists for born digital. We are optimistic that we can create (or discover) workable solutions to real-world problems involved with processing and preservation of digital objects. The work flow is wide ranging. Archivists often work with donors early in the process. Many collections pose technical and intellectual challenges such as arrangement or presentation. Legal aspects are involved in authority and access. Collections are sometimes ingested into content management systems. Of course, the eventual goal is to make the documents available via discovery by the general public, as well as accessible to scholarly research.

The AIMS team includes a software engineer, and our team is working closely with other software developers. We use normal software development conventions and open source software that runs on commodity hardware. We are agnostic about user interface and operating system so that our solutions will be portable, sustainable, easy to use, and accessible to the broadest possible audience.

Surveying Born Digital Collections

2010-07-16T23:47:00.009+01:00

The idea of creating a survey form was initiated by Glynn Edwards (my direct supervisor) at Stanford in May. Glynn would like to have something to guide the discussion of collecting digital material from a donor in July. She started with the Paradigm records survey (published by the Bodleian Library, Oxford University) and asked people at Stanford for comments. After gathering the comments, I posted the revised survey for the AIMS team to discuss. The main comment was the donors (mostly technical non-savvy) might not be able to understand all the technical terms. I have to thank Michael Forstrom at Yale University for sharing his revision of the Paradigm survey which avoid the use of technical terms.

The main difference between the original Paradigm records survey and the AIMS Digital Material Survey are:

1. Additional technology questions (e.g. web based backup, mobile device, social networking sites, document sharing sites, etc.)
2. Division of the survey into 2 parts: Part I is designed to be a prompt sheet for phone / face-to-face interview with donors by curators / digital archivists. Part II is to be filled out by digital archivists regarding technical details of the tools used to create digital material.
3. Usage of non-technical terms.

I think the survey should be sent before the actual interview as "something for the donor to start thinking about". If the donor is willing to reply before the interview, it helps the digital archivist to prepare as well. In fact, I sent the survey to a donor in July and she replied before the interview mentioning that she used Eudora for her emails. Since I was not familiar with Eudora, the answer helped me to get prepared for the interview as well.

Finally, I have to thank Susan Thomas, project manager of the Paradigm and the futureArch project, for her comments on the AIMS Digital Material Survey and her sharing of the experience in using the Paradigm records survey.

We would like to seek your comments on the survey as well. If you are going to discuss with donors on personal digital archives, why not download the survey and give it a try. Even if you are not collecting personal digital archives in the near future, take a look and tell us what you think.

Click below for the survey:
AIMS Digital Material Survey– Personal Digital Archives

Digital Lives Research Seminar

2010-07-07T11:35:00.002+01:00

On Monday I attended the Digital Lives Research Seminar Authenticity, Forensics, Materiality, Virtuality and Emulation and the presentations will be appearing online soon via the Digital Lives pages

There was a packed programme of speakers with a huge array of experience, of direct relevance to the AIMS work were the following:

Helen Broderick, British Library described her work as Curator, Modern Literary Manuscripts including cataloguing the born-digital material in the Ronald Harwood archive. The paper part of the collection had already been listed by a colleague and Helen encouraged hybrid collections to be tackled as single entity and this is what I intend to do with the Stephen Gallagher material at Hull.

Helen described using QuickView Plus software to view and used two screens (one to display the digital file and the second to record descriptive notes). Other thorny issues to be tackled include email and how this could be made available to others without infringing Data Protection and other privacy concerns.

Seth Shaw, Duke University gave an account of the current work at Duke, openly admitting that work on arrangement and description was very sporadic! They are looking to standardise their policies, documentation etc with the search interface another element on his todo list! It was clear that practice was being shaped by their experiences echoing the underlying element of the best practice guidelines that AIMS will produce based upon our combined experiences.

It was good to see colleague Michael Olson, Stanford University who gave an account of the forensics work at Stanford including the approach adopted for the Stephen J Gould material and outlined the resources in the Forensics Lab.

Gabriela Redwine, Harry Ransom Centre (University of Texas) provided an update on the forthcoming Computer Forensics and Born-Digital Content in Cultural Heritage Collections (see http://mith.info/forensics/) due to be published later this year. It came as no surprise to those present that the biggest challenge the research had identified was legacy hardware and software; other challenges included trust and authenticity. This led to a discussion around some of the ethical issues surrounding born digital materials and that we should be looking to multiple sources of information to build-up a complete picture (metadata, creator and forensics).

Erika Farr & Naomi Nelson, Emory University gave a fascinating account of their work on the digital material in the Salman Rushdie archive and the multi-disciplinary approach to tackling this collection. After discussion and consideration they agreed to respect the hybrid nature of the material; to balance the needs of the researcher and the donor but also the desire to provide an authentic ‘experience’. They had originally distinguished between paper and born-digital material with separate agreements but quickly revised this to one based on content and NOT format. They discussed with the donor his relationship with the PC and how he used it whether he customised parts etc to understand this aspect better. They were even able to recover files from a laptop he had accidentally damaged. The use of an emulator does give a totally different perspective to the born digital material that simply allowing access to the content can ever do. Whether this approach is always possible or practical remains to be seen.

Our host Jeremy John, British Library described their approach and workstream including imaging the disk and creation of digital replicates – viewed via original software and emulator and then facsimile versions for user viewing. He encouraged using hash values generated by two systems as additional level of verification. The British Library policy was to disk image wherever possible and were actively using emulators using a virtual machine based on the original hardware OS.

I was able to give a quick introduction to the AIMS project and from the questions that followed some of our work regarding access and use is of particular interest to others.

Jeff Ubois highlighted the main issues that arose from the Personal Digital Archiving Conference 2010 earlier this year including complexity of media with the need to compare donor agreements, interface design, suitability of tools re Facebook etc identified for future consideration and action. He also spoke about the public/private boundaries and mentioned a Research Libraries Group project ‘Good Terms’ about engaging with public companies for digitisation programs.

There's a New Course in Town

2010-06-29T18:19:00.003+01:00

Hello Fellow Digital Practitioners,

Given the multiple postings on the Digipres lately in regard to education for digital preservation and curation professionals, I thought it was good timing to plug a new course offering from the University of Virginia's Rare Book School. Fresh off the press (so to speak) is "Born Digital Materials: Theory & Practice", taught by Matthew Kirschenbaum, Associate Director of the Maryland Institute of Technology in the Humanities (MITH), and Naomi Nelson, former Interim Director for the Manuscript, Archives and Rare Book Library (MARBL) at Emory University and incoming Director for the Rare Book, Manuscript, and Special Collections Library at Duke University. I was lucky enough to be a member of the beta class although Matt & Naomi are such pros that one wouldn't have known that this was their first time around with this course.

As articulated in the course description, this class is aimed at an audience of archivists, librarians, curators and other individuals tasked with managing collections of born-digital content. Over a five-day period, topics such as materiality of digital objects, preservation metadata, emulation, migration, authentication of electronic records, and access strategies for these types of collections was discussed. Lectures and discussions were interspersed with hands on labs involving conducting donor surveys, using hex viewers, installing emulators and getting to use VINTAGE PCS AND MACS FROM DAYS GONE BY to enable file sleuthing! Needless to say, great fun was had by all. And we learned stuff too.

I would urge any and all who are engaged in digital preservation and stewardship to enroll in this course when it is next offered (likely Summer 2011). A great strength of the course are the complimentary interests and professional experiences that both Matt and Naomi bring to the course. It was a great asset that we also had a small class of colleagues anxious to share their real-world experiences as well. Don't let the reading list or computer forensics scare you - Enroll!

DigCCurr Professional Institute 2010

2010-06-03T19:14:00.016+01:00

During the week of May 16th – 21st, I attended the DigCCurr Professional Institute: Curation Practices for the Digital Object Lifecycle, at the University of North Carolina at Chapel Hill. The week-long course was entirely focused on best practices for digital preservation and curation with a balance of theory and hands-on labs using some of the common tools for the assessment of files and for the creation of curation workflows. The participants were a wide-array of librarians and archivists from all parts of the United States. The faculty (listed below) are, for the most part, well known in digital curation circles. I found this week of study to be entirely relevant to my work for the AIMS project and I would make the recommendation that this course would be highly useful to those engaged in archival processing, preservation and/or data curation.

The DigCCurr faculty are as follows:

UNC Chapel-Hill: Dr. Helen Tibbo, Dr. Cal Lee, Dr. Richard Marciano, & Carolyn Hank University of Michigan: Dr. Nancy McGovern University of Toronto: Dr. Seamus Ross University of Cologne: Dr. Manfred Thaller

Highlights of the Institute, aside from meeting a bunch of energetic and forward thinking librarians and archivists, was the focus on the importance of creating policies for one's institution for digital preservation & curation and the opportunity to get an overview of some of the tools that have been developed to help stewards of digital content manage their collections responsibly.

For those of you keen on drafting policies in regard to digital preservation and digital curation, both ICPSR and OpenDOAR provide examples of policies that can be used as guidelines for your institution. Curious about the readiness of your institution's repository to manage digital curation activities? DRAMBORA (Digital Repository Audit Method Based on Risk Assessment) is a tool meant for assessing risk associated with an institutional repository in terms of its readiness to manage curation activities. TRAC (Trusted Repositories Audit & Certification) provides a suite of tools for the audit, assessment, and potential certification of digital repositories. It establishes documentation requirements for an audit, outlines a process for certification, and establishes a framework for determining the sustainability of digital repositories. Auditors come from the outside of your organization.

I could go on (but I won't) since the week was packed with great hand-on labs and lectures. If you're interested in getting more information you can certainly contact me. One last thing I'll add is this; although this wasn't part of the formal program, I did have the opportunity to learn about a new project underway in Vancouver called Archivematica. It's an open source set of tools knit together for the management and preservation of born digital collections and archives. The 0.6 alpha version was just make available for download a couple of weeks ago - Check it out! And if you want an assessment from January 2010, check out Chris Prom's blog.

Practical Approaches to Electronic Records

2010-05-25T10:31:00.000+01:00

On Friday I attended the excellent Practical Approaches to Electronic Records event in Dundee. The programme included thought provoking discussions from Dr Ian Anderson (HATII) and Malcolm Todd (TNA) who both stressed the importance of demonstrating ‘value’ and ‘relevance’ to our organisations, and the need to develop new partnerships with colleagues working in digital forensics, ICT departments and universities to tackle the challenge of digital preservation. WillIam Kilbride (Digital Preservation Coalition) offered some interesting personal reflections on digital preservation and the conclusion that it is not about ‘data’, ‘access’ or ‘risk’ but about people and outcomes.

The afternoon was especially timely as it featured two demonstrations of ingest tools - something the AIMS project is currently working on. Viv Cothey showed us the work he has done at Gloucestershire Archives on the SCAT tool and this was followed by Peter Cliff demonstrating the BEAM Ingest tool being developed at the Bodleian. Both tools have adopted a modular approach to utilise many of the excellent and widely adopted 3^rd party tools such as PRONOM, Jhove and FITS and this is the obvious route to follow as we seek to create an ingest tool that is integrated with the Fedora digital repository.

The day ended with Chris Prom summarising his work to identify and compare many of the open source tools that are available. He encouraged everybody to get involved with a software project and listed the elements that he thought made an excellent Open Source project citing archivematica as a good example as it provided regular updates, clear documentation, availability of source code and support wiki.

ECA 2010

2010-05-17T13:58:00.003+01:00

A few weeks ago I was fortunate enough to present a poster about the AIMS project at the 8th European Conference on Digital Archiving at Geneva. The poster attracted a fair amount of interest and in particular the intention to share the tools and best practice with other institutions – many of the other sessions and some posters presenting what were quite institution-specific solutions.

Reflections on the conference

– though with 7 simultaneous sessions there was only some much any one person could see!

Programme & keynote speeches

First session after the initial keynote welcome was an interesting session “Envisioning Archival Description” by Victoria Lemieux & Geoffrey Yeo about using visualisation software to represent data and reflect relationships between words, as tested using the British Cabinet papers. There wasn’t enough time for a live demonstration, but the potential of offering an intuitive visual mechanism for looking at large volumes of data was really interesting. Information about word frequency for example could also be used by the archivist to assist with describing the records. The $64,000 questions are the cost of the software and what data / formats this is appropriate for?

There was also a good overview from Robert Sharpe of Tessella about obsolescence, characterisation and preservation planning with particular emphasis on the Planets Framework of tools.

Jaap Kamps “What makes a User Click?” was a timely reminder of the different search strategies between experts and novices and something we must return to when considering making born digital archives accessible on-line.

Jane Stevenson gave a review of the Archives EAD editor for the Archives Hub that can now include links to digital objects but emphasised that the files need to be stored locally – they are not a repository. Also reminded me that we (Hull University) owe the Hub some data!

Day 1 ended with a session by Daniel Pitti, who I met many years ago as EAD took-off in the UK, giving an account of a new 2 year project which seeks to create tools to automatically extract authority records using the new Encoded Archival Context-Corporate Bodies, Persons, and Families (EAC-CPF) standard. The project will start with an initial body of more than 30,000 EAD finding aids and look to combine this with more than 10m name authorities from the Library of Congress and OCLC. The aim is to take a step beyond personal name indexes and show a network of relationships (eg between correspondents). With the University of Virginia being one of the project partners it is hoped that AIMS can collaborate with this project further down the line.

Days 2 and 3 to follow...

Digital Forensic Lab at Stanford University

2010-05-11T17:03:00.007+01:00

Hi,

I am so lucky to work as digital archivist at Stanford. One of the reasons is that Michael Olson from our Digital Library System Service group set up a digital forensics lab last year at Stanford. We have Forensic Tookit and EnCase installed in a FRED (Forensic Recovery of Evidence Device) in the lab. Michael also maintains a blog about Digital Forensic at Stanford University.

DigCCurr

2010-05-11T15:08:00.003+01:00

Hello,

I'm getting geared up to attend the 2010-2011 DigCCurr Professional Institute at UNC-Chapel Hill next week. I'm looking forward to meeting my peers in the field, learning more about the life cycle of digital objects and getting some hands-on experience with tools to assist with their management over the long haul.

I'll be posting more about this institute once I've actually attended it; in the meantime, here's a bit of info:

DigCCurr Institute

Liz