Following on from the Jisc Research Data Spring the universities of Manchester and Edinburgh have been continuing to develop DataVault.  Both institutions are currently planning local implementations, whilst working together to continue develop the software.

On 28th March, the project team met in Edinburgh to discuss their implementations and tasks for the next couple of months.

Tasks on the DataVault development list:

– BagIt Libraries, resolve issues with BagIt libraries: removing empty directories, renaming files, creating manifests in memory

– Stand alone packager, to package deposits external to the DataVault web application

– Verification of deposits, e.g. checking that the number of files and filesizes are correct

– Deposits via API, for browser upload, requires further investigation into authentication and chunking of files

– User Roles/Groups/Sharing Archives, definitions of roles and implementation within the DataVault

– Closing Vaults, in what circumstances are vaults closed?

– Integrations with Pure and Dropbox

We will also be moving our documentation and issue tracking into the DataVault GitHub repository https://github.com/DataVault/datavault

It’s been a busy summer for the DataVault, with further presentations taking place.

First up was a short trip to Dublin to the Open Repositories 2016 conference in June. The presentation was scheduled to be part of the 24×7 session: 24 slides to be presented in 7 minutes – that’s a mere 17.5 seconds per slide (rather than the usual rule-of-thumb of a minute or two!) and with the slides auto-advancing for a bit of added fun!. Thomas Higgins and Stuart Lewis presented, and gave an overview of the project, the platform, and gave a demonstration.

A copy of the slides is available for download.

Then in August, Mary McDerby and Stuart Lewis attended the Repository Fringe in Edinburgh, where Stuart presented the DataVault in a session led by the DCC’s Angus Whyte, alongside Rory Macneil from ResearchSpace, looking at the subject of research data workflows, and what this means for systems such as the DataVault which sit within those workflows.

We’re now in the third phase of the DataVault project, and as previously discussed, we are using fortnightly development sprints to undertake the remaining development tasks.  Following our monthly project meeting yesterday, we now have draft sprints to take us up until the end of June, and the first full release of the software!

Keep an eye out for a future blog post: we’re scheduled to hold an event for potential early adopters of the DataVault system in their own institutions.  29th June, central London!

Between now and then we have planned a further four sprints (sprints 3, 4, 5, and 6).  We plan these in details at the start of each sprint, but right now we have indicative backlogs for the next three.  Not only will these involve further developments to the software, but also test installations at our institutions to allow more thorough testing of the software in-situ, especially once fully configured into local systems such as Shibboleth and PURE.

All of the Jisc #DataSpring projects have also been reviewed by both the SSI and the DPC in terms of sustainability and good practice from an open source perspective.  We’re glad to report a relatively good result, but there are a few areas where we can improve – so we will also be addressing those in the coming weeks.  These include better documentation, links about how to contribute to the project, and clearer contact details.

Phase three of the DataVault project is now earnestly underway.  Two weeks ago we held the first of our monthly DataVault project meetings, with Mary McDerby and Thomas Higgins (University of Manchester) visiting us in Edinburgh.  One of the changes we are making to this final six month phase is to move to fortnightly development sprints.

We’re now almost at the end of the first sprint, so reviewed the outputs during a Skype call today.  We use Trello to manage a backlog of tasks, which then get selected for the next sprint. Most of the development tasks are now completed or almost finished.  We also started planning the stories to be tackled in the next sprint, however due to Easter, this will be a little shorter than normal:

In the first sprint (sprint 1) we undertook some rationalisation of the API (as it has grown arms and legs over time), added an enhanced auditing feature (using the PREMIS ontology), added an improved runtime configuration option, and provided better error reporting for a problem that occurs when the CSRF tokens timeout.  As always, all of this can be seen in the project’s github account.

In the next sprint we’ll be fixing a few bugs and some of the terminology used (replacing ‘restore’ with ‘retrieve’ when getting some data from the DataVault), and adding better application logging.

It’s been a while since we last posted a project update, so here is a round-up of recent progress:

• We are currently at the 11th International Digital Curation Conference in Amsterdam, and will be presenting a paper on the Data Vault project to the conference in the ‘digital preservation’ stream of the conference.
• A demo system is now available where you can see how the system works, and deposit + retrieve some example data sets.
• Jisc have generously provided phase three funding to the Data Vault project.  This funding will last for six months, and will provide resources to complete the development of the first complete version of the Data Vault software.

Over the next few months we will be hosting a series of workshops, with the aim of engaging with the community to refine the Data Vault platform solution developed so far.

The first workshop is being held in Manchester on 7th October 2015 – for further details and to register see:

Eventbrite Data Vault Community Event, Manchester

The second workshop is being held in Edinburgh on the 5th November 2015 – for  further details and to register see:

Eventbrite Data Vault Community Event, Edinburgh

Now that the Data Vault project has received second round finding, we thought it was time for a makeover!

We’ve been working hard with our designers, and now have a project logo and URL:

http://datavaultplatform.org/

We’re pleased to announce that following the second round of project pitching in London on the 13th and 14th of July, the Data Vault project has been awarded further funding.

The first round of funding provided three months of effort, during which time the project developed a working proof of concept system that could archive and restore data.  At the pitching session, team members demonstrated the prototype, and explained what would be achieved in the second round:

• Critical success factor: Deliver a first version of a complete data vault platform
• Implement the remaining requirements:
  • Authentication and authorisation
  • Integration with more storage options
  • Management / monitoring interface
  • Example interface to CRIS (PURE)
  • Development of retention and review policy
  • Scalability testing
• Two community engagement events (Manchester: 7th October, Edinburgh: 5th November)

We are grateful to Jisc for providing this second round of funding which will provide effort from August to November.

Blog post by University of Manchester project developer Tom Higgins:

Yesterday I gave a short presentation on the Data Vault project at an event in Lancaster:

https://www.eventbrite.co.uk/e/research-data-management-solutions-for-your-needs-tickets-17100593335

I based this on the original pitch with a few updates reflecting the work we’ve done over the last couple of months.

Here’s some of the feedback and questions from the event – I think a lot of these are more relevant for “phase 2 and beyond” than the current prototyping:

• How does the Data Vault differ from iRODS? Perhaps the policy model from iRODS could be useful or iRODS could serve as a back-end. There was a comment that iRODS may be more useful where the researcher’s workflow is known and can be encoded into the system (e.g. it’s deeply involved in the day-to-day active data).
• Archivematica (being explored by a project in York) can handle many preservation activities but has a specialist user interface which is not suitable for researchers to use directly. Perhaps a Data Vault could be used to ingest data and hand it over the Archivematica for preservation.
• How would a Data Vault handle sensitive data? Would it be need to be certified? What if the “back-end” was using a certified storage system – would that ease the burden at all? I mentioned that perhaps both a “general” and a locked-down “sensitive” instance of the software could be run in parallel.
• How could a Data Vault handle a dataset that is changing over time? Perhaps snapshots could be captured periodically – would this use a lot of storage space?
• Could data be ingested from instruments automatically? I think this is an interesting one because the researcher will presumably want to access the data on active storage too (e.g. just ingesting into the vault isn’t particularly useful since you’d then need to pull it back out to actually work with the data, but you may want to have a frozen copy of the raw data too).
• How could a Data Vault handle complex data e.g. from a database or an object store? In the simple case a user could export their data (e.g. in a backup format) and store that data (similar to how they might back up a database to a USB drive). Does it make sense for the a vault to try to understand complex data?

Here are some examples of “Active” and “Archive” systems which might be useful targets for integration:

• Box
• Hitachi Content Platform
• DuraCloud
• iRODS
• Archivematica