The development model we chose for the Data Vault is to get us all in a room (Robin, Tom, Claire, Mary, Stuart) and to collaboratively develop the proof of concept system over a few days.  We were kindly hosted by the University of Manchester IT services in their Sackville Street building.

We started by looking at the skeleton framework that Tom and Robin had worked on, and then assigned areas of code to each person to write.  For example work was required on the user interface that the user sees, the broker in the middle that manages the system, and the backend workers that perform the archiving.

All of the code is stored openly in github, and is open source with an MIT license:

• https://github.com/DataVault/datavault

Work is now continuing following the hackathon to complete a few areas of remaining code before the next Jisc Data Spring programme meeting where we can share the system with others.

On 10th of June, the Data Library team ran two workshops in association with the EU Horizon 2020 project, FOSTER (Facilitate Open Science Training for European Research), and the Scottish Graduate School of Social Science.

The aim of the morning workshop, “Good practice in data management & data sharing with social research,” was to provide new entrants into the Scottish Graduate School of Social Science with a grounding in research data management using our online interactive training resource MANTRA, which covers good practice in data management and issues associated with data sharing.

The morning started with a brief presentation by Robin Rice on ‘open science’ and its meaning for the social sciences. Pauline Ward then demonstrated the importance of data management plans to ensure work is safeguarded and that data sharing is made possible. I introduced MANTRA briefly, and then Laine Ruus assigned different MANTRA units to participants and asked them to briefly go through the units and extract one or two key messages and report back to the rest of the group. After the coffee break we had another presentation on ethics, informed consent and the barriers for sharing, and we finished the morning session with a ‘Do’s and Dont’s exercise where we asked participants to write in post-it notes the things they remembered, the things they were taking with them from the workshop: green for things they should DO, and pink for those they should NOT. Here are some of the points the learners posted:

DO
– consider your usernames & passwords
– read the Data Protection Act
– check funder/institution regulations/policies
– obtain informed consent
– design a clear consent form
– give participants info about the research
– inform participants of how we will manage data
– confidentiality
– label your data with enough info to retrieve it in future
– develop a data management plan
– follow the certain policies when you re-use dataset[s] created by others
– have a clear data storage plan
– think about how & how long you will store your data
– store data in at least 3 places, in at least 2 separate locations
– backup!
– consider how/where you back up your data
– delete or archive old versions
– data preservation
– keep your data safe and secure with the help of facilities of fund bodies or university
– think about sharing
– consider sharing at all stages. Think about who will use my data next
– share data (responsibly)

DON’T
– unclear informed consent
– a sense of forcing participants to be part of research
– do not store sensitive information unless necessary
– don’t staple consent forms to de-identified data records/store them together
– take information security for granted
– assume all software will be able to handle your data
– don’t assume you will remember stuff. Document your data
– assume people understand
– disclose participants’ identity
– leave computer on
– share confidential data
– leave your laptop on the bus!
– leave your laptop on the train!
– leave your files on a train!
– don’t forget it is not just my data, it is public data
– forget to future proof

Our message was that open science will thrive when researchers:

• organise and version their data files effectively,
• provide comprehensive and sufficient documentation for others to understand and replicate results and thus cite the source properly
• know how to store and transport your data safely and securely (ensuring backup and encryption)
• understand legal and ethical requirements for managing data about human subjects
• Recognise the importance of good research data management practice in your own context

The afternoon workshop on “Overcoming obstacles to sharing data about human subjects” built on one of the main themes introduced in the morning, with a large overlap of attendees. The ethical and regulatory issues in this area can appear daunting. However, data created from research with human subjects are valuable, and therefore are worth sharing for all the same reasons as other research data (impact, transparency, validation etc). So it was heartening to find ourselves working with a group of mostly new PhD students, keen to find ways to anonymise, aggregate, or otherwise transform their data appropriately to allow sharing.

Robin Rice introduced the Data Protection Act, as it relates to research with human subjects, and ethical considerations. Naturally, we directed our participants to MANTRA, which has detailed information on the ethical and practical issues, with specific modules on “Data protection, rights & access” and “Sharing, preservation & licensing”. Of course not all data are suitable for sharing, and there are risks to be considered.

In many cases, data can be anonymised effectively, to allow the data to be shared. Richard Welpton from the UK Data Archive shared practical information on anonymisation approaches and tools for ‘statistical disclosure control’, recommending sdcMicroGUI (a graphical interface for carrying out anonymisation techniques, which is an R package, but should require no knowledge of the R language).

Finally Dr Niamh Moore from University of Edinburgh shared her experiences of sharing qualitative data. She spoke about the need to respect the wishes of subjects, her research gathering oral history, and the enthusiasm of many of her human subjects to be named in her research outputs, in a sense to own their own story, their own words.

Links:

• Scottish Graduate School of Social Science
• FOSTER Open Science
• Research Data Management Training – MANTRA course

Rocio von Jungenfeld & Pauline Ward
EDINA and Data Library

Meeting our Needs

Since the early 2000s we have been looking for suitable software to manage our archives in a holistic manner. We began to deliver online catalogues at this time via various project initiatives, with metadata encoded as EAD/xml, but this only dealt with resource discovery and was quite cumbersome. Moreover, along with other digital developments, the work inhabited one of a number of parallel silos.

As time moved on, we got better at developing systems to move different elements of work from the analogue to the digital but were still some way off developing or finding a comprehensive, robust and sustainable way to join things up in a meaningful way. This changed when we began to investigate Archivists’ Toolkit in 2011. Although we had looked at it in one of its earlier versions, we were surprised to see how much subsequent developments had brought it quite close to ticking everything on our wish list. It was lacking a resource discovery layer but a successor product, ArchivesSpace, was already planned and would include this.

From Archivists’ Toolkit to ArchivesSpace

We therefore began looking at Archivists’ Toolkit in more detail, assessing issues such as functionality and usability but also those of sustainability and interoperability. It scored very highly, high enough for us to be able to make the business case to commit to ArchivesSpace and obtain the internal funding to sign up as Members.

The involvement of the profession in the development of ArchivesSpace has been and continues to be crucial. What has been developed is not just other people’s idea of what the product needs to be but what we as archivists actually require. Although heavily influenced by the predominant US partners and the specifics of US practice, it has been developed in way that is equally intelligible to others and easily customisable to reflect local needs and terminology.

Priorities and Impact

We originally focused on moving our behind-the-scenes work over but then switched to frontloading our resource discovery, migrating existing EAD xml files and also retro-converting a wide range of old spreadsheets, databases and similar. In terms of impact, this both provides evidence that our business case was sound but, most importantly, meets growing user expectations of what and online catalogue should deliver.

Phase one saw the delivery of nearly 17,000 catalogue records along with over 22,000 authority terms. We still have more to add, along with a whole range of management metadata about accessioning, locations etc. This will feature in Phase 2.

Because the source metadata has been drawn from a variety of legacy sources, there are issues of consistency and quality to be addressed. These are outstanding issues which could never be solved just by getting the metadata into ArchivesSpace. However, with all the metadata now in one place we can now look to quantify and rectify them. Experience told us that’s users would often rather have partial metadata rather than no metadata at all so we chose to go for a warts and all approach, only correcting what was obviously erroneous at this stage.

Community and Participation

We are proud to have signed up as the first European partner and the support we have had from a growing community of ArchivesSpace users and developers. This discussion is also two-way, with us feeding ideas back for future development.

Locally we are also more fully integrated into developing solutions that deliver all our collections online, through a suite of applications and interface that work together, improving user experience and improving how we manage the collections themselves.

Next Steps

We still have lots to do with the system to leverage the full functionality of the system and fully showcase our amazing archives collection. So watch this space.

View the online catalogue.

Read about this from a technical perspective

The Statistical Account presents a fascinating picture of eighteenth-century Scotland. It gives a rich and often detailed view of the working lives of ordinary people, and was of particular importance to my area of research which was concerned with the origins of eighteenth-century popular political consciousness and the relationship between Presbyterianism, literacy and political activity. Hence sources of information relating to occupations and social status were vital, and the OSA provided considerable insight as to the range of occupations in which heads of families were employed in villages and towns throughout Scotland. The on-line version of the OSA with its text searching facility was invaluable in pinpointing specific information and saved hours of manual searching.

As one aspect of my thesis was the relationship between religious controversy and popular political consciousness expressed through patronage disputes, it was important to determine what kind of people involved themselves in such disputes. Patronage disputes were widespread across eighteenth-century Scotland and were significant, in part because they displayed clear opposition to the élite and to government policy, but also because they involved whole communities, with opposition from heritors, elders, and heads of families, essentially including people from every stratum of society, and every occupation.

Little Dunkeld Kirk Perthshire Scotland, built 1798

Using the on-line search facility I was able to search by occupation and place, and by comparing the occupations in the OSA with the occupations on book subscription lists, it became evident that the types of occupation followed by heads of families had not changed over the century. This provided a clear guide to the kind of people who were at the heart of eighteenth-century patronage disputes and later political activity. For example, at Campsie in Stirlingshire, where there was a disputed presentation in 1784, there were ninety-six heads of families: twenty-eight were feuers who farmed their own lands, fifty were tenants, seven of whom were chiefly employed in grazing, and the remaining eighteen were masons, carriers or road makers. At Crieff in Perthshire, heads of families included apothecaries, physicians and surgeons, mantua makers, bakers, masons, slaters, barbers, shopkeepers, butchers, carriers, carters, messengers, clergymen, midwives, clockmakers, millers, coopers, saddlers, dyers, schoolmasters, distillers, excise-officers, shoemakers, farmers, smiths, spinsters, stocking makers, fiddlers, tailors, gardeners, weavers, wrights, hecklers, writers, inn-keepers, labourers, manufacturers, and gentry. The variety of occupations underline the social status of those involved.

The OSA also offered numerous examples of ordinary peoples determination to access education, books, pamphlets and newspapers, as well as their desire to associate, meeting together in clubs for discussion and debate. The accounts from parish ministers testify again and again to their parishioners’ over-fondness for books about ’controversial divinity’, as well as a wider interest in books generally. Many ministers commented on the range of people expressing interest in newspapers and current affairs, for example, the Revd Andrew Duncan of Wigton noted that, ‘An attention to publick (sic) affairs, a thing formerly unknown among the lower ranks, pretty generally prevails now. Not only the farmers, but many of the tradesmen, read the newspapers, and take an interest in the measures of government’. The Revd John Bruce remarked that, in Forfar, ‘A spirit of enquiry and a taste for reading … (with) subscriptions to the Encyclopedia Britannica, the Bee, and several periodical and other publications, scientific, religious, moral and political, are more numerous of late’. The minister at Little Dunkeld in Perthshire, noted that ‘Newspapers and other periodical publications find their way to every corner of the parish’, and this interest in current affairs was not a recent phenomenon but one which had existed since at least the time of the American War of Independence. He also commented on the formation of several reading clubs. Such observations emphasised the significance of reading, and many ordinary people’s involvement in political activity was enabled by the coincidence of widespread literacy encouraged by a Calvinist education, from which they had not only gained that skill, but the ability to reflect, form opinions, and question authority, capacities which could be brought to bear in the wider context of assessing their position in Scottish society. Ministers’ observations throughout the OSA highlighted the availability and variety of reading material, and how common the practice of reading and book buying was, a practice which encouraged debate, and contributed to popular involvement in the radical political movement of the 1790s.

I hope this brief description has highlighted what a wonderful resource the OSA is with its wealth of information and ease of use through on-line access and search facilities.

Dr Val Honeyman

We hope you have enjoyed this post: it is characteristic of the rich historical material available within the ‘Related Resources’ section of the Statistical Accounts of Scotland service. Featuring essays, maps, illustrations, correspondence, biographies of compliers, and information about Sir John Sinclair’s other works, the service provides extensive historical and bibliographical detail to supplement our full-text searchable collection of the ‘Old’ and ‘New’ Statistical Accounts.