About Pauline Ward

Research Data Support Assistant Library and University Collections University of Edinburgh

New feature in Edinburgh DataShare: the REST API

Ever wanted to get a table of the details of all the datasets on DataShare to do with Scottish history? Or matching some other criteria, possibly on specified fields? If so, the new API (Application Programming Interface) can help.

DataShare now has a REST API, which you can use to query our metadata. An API makes the database’s contents accessible for requests from external servers, through a command-line, which allows external users to script such requests. The DSpace API also provides its own web-based query client and report client. These pages allow users to use a graphical interface to quickly build a query and see the results in a table, all in the browser.

The DataShare REST API page starts with a link to our plain-English explanation of how the API can be used:

Edinburgh DataShare DSpace REST API 

We would like to hear from anyone who wants to use the API. Please try it out and let us know what you find useful! Email us at data-support@ed.ac.uk .

Examples using the graphical query builder

I wanted to find datasets where I could add a link to the associated publication. This is a bit of a challenge for us, since users typically deposit their data with us under embargo before the associated paper has been published, and we do not have an automatic way to detect when or whether an associated publication has appeared.  I used the query builder to find the IsReferencedBy value for deposits accessioned in 2017. The plain-English guide on the wiki provides the steps I went through to do so:

How to use the DataShare REST API 

This feature may be of use to colleagues who support organisational units at University of Edinburgh which don’t align precisely with the Collections structure of DataShare – the API lets you query on multiple collections through the reporting tool. We’d love for colleagues to contact us if their teams have published a new paper containing a data citation of their DataShare deposit, so we can add the details of the publication to the DataShare Item’s metadata, resulting in a hyperlink appearing on the dataset landing page.

I wanted to find datasets with an embargo date in December. This is a challenge for us because users often set their embargo expiry date to Hogmanay, which means their one-week reminder would arrive on Christmas Day right in the middle of the university’s winter break. But many other fields contain dates with December in them, so it has not been practical for me to search for this using the graphical interface. So I used the API to search specifically in the dc.date.embargo field. See the screenshot below. The API helped me find the datasets whose embargo date we needed to extend, or else lift the embargo outright, allowing us to contact the depositors in good time to ask them whether a paper had been published or more time was needed.

Screenshot of the output of the REST API

Results showing datasets with an embargo date in December 2021

Thirdly, to demonstrate the power of this tool relative to the non-specific Search I chose a topic with very common words to show how to use the query builder to focus in on results avoiding spurious matches.

Using the existing ‘Search’ function on the homepage I searched for ‘history Scotland’. This produced 39 matches, some of which have nothing to do with historical research or Scotland, but merely mention a funder “NHS Research Scotland”, and mention the history of the research field in passing to provide a little context. Most of the matches are interesting, but some are not relevant.

Whereas when I set the API query builder to search for ‘history’ in the research area (subject classification), and ‘Scotland’ in the field for geographical metadata ie dc.coverage.spatial. This provided me with a short list of high quality matches, three datasets of historical research to do with Scotland – see the screenshot. This is a useful tool for narrowing a search.

Screenshot showing the input, and the output on the API query builder webpage

A search for two very common words in specific fields produces high quality results

Enabling the API

The REST API is a feature of the underlying DSpace repository software. Our sysadmins configured the API with great care to block certain commands and enable only the ‘GET’ commands that are needed for appropriate queries using DSpace config settings (further info DSpace 6 Documentation on the Lyrasis wiki ).

The Future

In the international DSpace repository community, we’re aware the API is used for integration with at least one CRIS (Current Research Information System) and quality tool applications (Andrea Bollini, 4Science, private communication). We understand the API of the newer DSpace 7 contains significant changes compared to that of DSpace 6, which we’re using for Edinburgh DataShare.

We’re aware of only a few examples of the API being used by individuals for occasional metadata queries. But we will watch with interest to see how the DSpace 7 API will be used.

 

Pauline Ward
Research Data Support Assistant
Library and University Collections
University of Edinburgh

Diverse climate change data in DataShare’s newest thematic collection

‘Code red for humanity’ was the galvanising message of the sixth report of the Intergovernmental Panel on Climate Change published on 9 August. The report draws on thousands of academic research projects. Research data is vital to understanding the nature and scale of the challenge of climate change, and the necessary deployment and application of solutions.

We decided to draw together the datasets relating to climate change to showcase them on a single Collection page on Edinburgh DataShare, our research data repository. In part this was prompted by a new deposit from Oliver Escobar in the School of Social and Political Science – data from citizens’ assemblies debating wind farms. Our DSpace repository allows us to ‘map’ an Item to Collections other than the one to which it belongs, resulting in the dataset being listed in more than one Collection. Edinburgh DataShare contains a wealth of research datasets from an extremely diverse array of academic disciplines, reflecting the strengths of the University of Edinburgh, and so it is with our climate change research:

Climate Change Collection

We added many datasets from our School of Geosciences: one dataset from Ian Goddard and Professor Simon Tett demonstrated how urbanisation has affected temperatures in the UK, and includes a map showing heat islands around our major cities. Professor Tett said:

“To truly understand how climate change might impact society we need to bring together many datasets developed by many researchers so that other researchers can use them for their own studies. DataShare enables this.”

Goddard, Ian; Tett, Simon. (2018). “Software and data used in the study ‘How much has urbanisation affected temperatures in the United Kingdom'”, 1990-2017 [software]. University of Edinburgh. https://doi.org/10.7488/ds/2370.

Climatological data and toolkits for public engagement around climate and natural resources came from Professor Marc Metzger – including various kinds of maps, a board game and posters showing natural resources.

One dataset was a description of an artwork, a quilt representing global temperature measurements. Posters on the wall show the years, so as to provide a time axis for the temperature data represented in the colours of the patches in the quilt:

Photo of quilt hanging on the wall of an exhibition space

World temperature quilt on display at the Data-X exhibition

Zaenker, Julia; Vladis, Nathalie. (2017). Feel The Heat – A World Temperature Data Quilt, [image]. University of Edinburgh. EDINA. https://doi.org/10.7488/ds/1998.

Another theme was renewable energy – we included data from our School of Engineering on tidal turbines, and recent wave buoy experimental data:
The big 3-0-0-0: DataShare reaches three thousand datasets

All this raises the question – why bring these data together, what for? Do the datasets measuring and defining the problem really belong with the research working on technology to reduce greenhouse gas emissions? To answer this, I think the analogy of our other thematic collection on Covid-19 is apt. To develop and implement effective treatments and public health responses to Covid-19, we do need to understand a great deal about the cause, the pathogen and the pathology it creates. We should strive to break down barriers between domains of knowledge. So yes, to tackle climate change more effectively, we should all seek to better understand the underlying processes and the behavioural and technological solutions we must employ.

By bringing together research data from diverse teams in a single DataShare Collection, we empower the user to browse those datasets using the ‘facets’ feature in DataShare, or indeed a text search within the Collection. The user can filter by geography, by data creator’s name, by keyword, funder (see the screenshot below) or they can choose their own search term. When they reach an individual dataset, the breadcrumb trail at the top of the page can lead them into the original Collection where the dataset was first deposited, leading them to other work from the same research group, centre or School. This is one small way for the curation team to enhance the findability of the data. Scientists tell us there are challenges posed by the plethora of formats and programming languages used, even within disciplines. We hope that by making the connections and common themes between these different strands of research from different disciplines more visible, we make the data more findable, and perhaps hope to inspire new research questions or approaches.

a screenshot from the Collection page

DataShare’s facets

A word about DataShare’s structure: we find our depositors prefer to place their data in a Collection that reflects the organisational placement of their research group – typically the Collection represents the research group, and sits within a Community representing a research centre, sitting within a Community representing a School, sitting within a top-level Community representing a College:
DataShare structure

If you would like to suggest a theme for a new thematic Collection on DataShare, please contact the Research Data Support team:
Research Data Service | Contact

The RDS team, like all the University of Edinburgh’s teams, has a remit to address climate change as the university is committed to contributing to the UN’s Sustainable Development Goals, including no. 13 “Take urgent action to combat climate change and its impacts”:
Social Responsibility and Sustainability | The University of Edinburgh

We can all learn more about how we can take that urgent action effectively on the university’s amazing and inspiring “Climate Solutions” MOOC, available on edX:
Climate Solutions | edX 

I recommend anyone to take this course – it is free of charge, it’s fun, it is easy to fit around other commitments. I’ve nearly completed the coursework and already passed thanks to my quiz scores, got my nice PDF certificate signed by Professor Dave Reay. The Climate Solutions MOOC inspired me to create the Climate Change thematic Collection and it has really opened my eyes to the scale and nature of the challenge, and many actions we all need to take to contribute to halting the rise in global temperatures. Everyone has their part to play.

Pauline Ward
Research Data Support Assistant
Library & University Collections
University of Edinburgh

The big 3-0-0-0: DataShare reaches three thousand datasets

Quote

Confetti banner says "3,000th deposit!!!" 2021-08-02

Timestamp showing the accession of the deposit on the 2nd of August.

We’re thrilled Edinburgh DataShare has just ingested its 3,000th deposit:

Davey, Thomas; Draycott, Samuel; Pillai, Ajit; Gabl, Roman; Jordan, Laura-Beth. (2021). Wave buoy in current – experimental data, [dataset]. University of Edinburgh. School of Engineering. Institute for Energy Systems. FloWave Ocean Energy Research Facility. https://doi.org/10.7488/ds/3105.

The depositor was Dr Tom Davey, Senior Experimental Officer in the School of Engineering, who said:

“It is a pleasure for us all in FloWave to see one of our datasets achieve this milestone for Edinburgh DataShare. This is also the tenth DataShare upload making use of experimental outputs from the FloWave Ocean Energy Research Facility. Providing a reliable and accessible repository of our project outputs is not only important for our funders, but also promotes new research collaborations and builds lasting impact for our experimental programmes. This particular project will aid in the understanding measuring wave and currents at deployment sites for offshore renewable energy technologies, and adds to the existing FloWave portfolio of datasets in the field of wave energy, tidal energy, advanced measurement, and remote operated vehicles.”

You can explore more data generated at FloWave in the IES DataShare Collection:

Collection – The Institute for Energy Systems (IES) (ed.ac.uk)

Although the ‘wave buoy in current’ dataset is under temporary embargo, currently set to expire on the 5th of September, it is possible to request the data using DataShare’s request-a-copy feature in the meanwhile. Embargoes may be extended, or lifted early, usually reflecting publication dates.

You might also enjoy this hilarious and very popular video about FloWave:

 

Pauline Ward

Research Data Support Assistant

Library & University Collections

University of Edinburgh

DataVault ‘at 100’ – stories from our users

The DataVault now contains over one hundred datasets, with a combined size of over one hundred terabytes. These are the stories of a handful of our happy customers, talking about the benefits of using the DataVault…

1.    Low-cost long term storage delivers flexibility

Emily Clark (PI) and Mazdak Salavati (Core Scientist)

Emily and Mazdak heard about the DataVault at an event at the Roslin Institute organised by Colin Simpson, local project officer, and expert user of the DataVault. They needed to archive a large amount of genomics data – more than three terabytes. They wanted to save money and liberate their active storage for other data by moving the files from DataStore to DataVault. As DataVault is much cheaper than DataStore (once you’ve used up your initial free DataStore quota). We also held an initial meeting with Colin to answer their questions about how they would be billed (via an eIT), who would be able to deposit and retrieve data and how. Emily wanted flexibility around the billing, and the control to restrict access. We discussed the usefulness of splitting the data into separate deposits, to enable subsets of the data to be exported back to DataStore. As you can see from the public metadata, link below, the information has been worded in such a way as to manage the expectations of the reader appropriately. Mazdak created the vault, gave Emily access, and then deposited the data as a series of separate deposits, over a few days. We then issued the eIT for payment.

Mazdak says:

“Engaging early and reading the documentation was key for the use of DataVault. Having a research data management (RDM) plan from the beginning of your project helps with almost every aspect of dissemination of it i.e. publication, grant reports, collaborators and stakeholders etc. Understanding the types of storage and their cost makes this planning much easier for both the PIs and the data processors involved. Moreover, curating the metadata associated with biological datasets is much easier once the RDM plan is based on a streamlined platform such as DataVault. The clever use of low-cost long term storage solutions can free up lots of flexibility both in consumables and computational resources if considered from the start of every project. The DataVault is maintained and supported by very dedicated folks at the University who would their best to help and accommodate research needs. Talk to them in the earliest point possible to discuss your RDM plan and take advantage of their support.”

You can see the public details of the data by clicking on the DOI:

2.    A safe place for personal data

Professor Sue Fletcher-Watson

Sue had a set of video footage gathered as part of her work with children with autism, specifically the Click-East clinical trial. The audio-visual files, a little over a terabyte, were stored on an external hard drive, so she wanted to have them safely backed up, but did not have sufficient spare storage in her DataStore area. Sue had learned about the Edinburgh DataVault when chatting with a member of our team at an IAD event, so she knew DataVault would be an appropriate home for this sensitive data to be stored for the ten year period to which the participants’ parents had consented; the DataVault encrypts the data and stores three copies, and is cheaper than buying additional DataStore storage. Information Services provided Sue with a temporary ‘staging area’ on DataStore free of charge to accommodate the transfer of the data from the external hard drive, first onto the DataStore staging area and then into the vault she created, after first creating a Pure record which we validated. The Research Data Support team now has a similar dedicated staging area on DataStore which we can make available to those users who need the space temporarily for a DataVault deposit or retrieval. Of course, datasets should be split into deposits of an appropriate size so that the retrieval need not occupy too much space on DataStore. Sue successfully deposited the data. And later was able to use the staging area again to retrieve the data.

“The support I got from the DataVault team was exemplary and really helped me with this first deposit. I now have complete confidence that these valuable data are safe and secure. I’ll certainly be using DataVault again”

3.    Facilitating restricted sharing and citation of clinical data

An anonymous Edinburgh researcher

One research team contacted us about getting a DOI (Digital Object Identifier) for their pseudonymised clinical data, so they could cite the data in their manuscript, and so they could share the data on a restricted basis with their reviewers. We advised that while the reviewers would not be able to access the DataVault directly, the researchers could use DataSync to share an encrypted copy of the data with them, while protecting the anonymity of the reviewers, by sending the link for DataSync to the journal, for forwarding on to the editors. We helped the researchers describe their data in Pure. The researchers archived the data into DataVault. We minted a DOI on the Pure record, which the researchers then added into their manuscript and is now included in the finished publication. Of course, clicking on the DOI link does not give users direct access to the data – it merely takes them to the metadata, the description, on Pure’s public portal, the Edinburgh Research Explorer, where they can find the information they need to make a request for the data. Thus the research team still has the control, so that they can decline a request, or they can require such external researchers to sign a Data Sharing Agreement, undertaking not to attempt to re-identify any participants nor to share the data further with others.

A note on Data Protection

It is important to keep in mind that Principal Investigators are responsible for understanding and complying with data protection law and their own funders’ and collaborative partners’ requirements. The DataVault should be used as part of good practice in research data management throughout the research data lifecycle. We strongly encourage researchers to make a Data Management Plan for every project; the Research Data Support team is happy to review your Data Management Plan, provide feedback, advise whether the DataVault would be a suitable solution, and help include the associated costs in your research bid.

Pauline Ward
Research Data Support Assistant
Library and University Collections