Invenio Blog

Whoops! The Better-Late-Than-Never March 2025 InvenioRDM Partner Meeting Summary

2026-03-20T00:00:00Z

As the InvenioRDM partners were making their way back home from 2026's annual partner meeting, we suddenly realized we had not published a blog post summarizing 2025's partner meeting. Whoops! Turns out, we were so busy building and improving InvenioRDM for you that it completely slipped our minds. Before we catch you all up on the most recent meeting, here is a summary of the great things that the InvenioRDM partners accomplished in 2025.

Meeting highlights

Kai Wörner, Deputy Head of the Center for Sustainable Research Data Management at Universität Hamburg, and his team, hosted the InvenioRDM community for five days from March 24 - 28, 2025. In addition to the local team's hospitality, participants experienced the university's cutting-edge research and the city's rich maritime heritage, cultural diversity, and the enchanting Miniatur Wunderland. Meanwhile remote participants enjoyed a seamless experience through a Zoom/Owl setup provided and maintained by Steve Diggs, an InvenioRDM collaborator currently with the Scripps Institution of Oceanography.

Instance rodeo & Open space technology

The first day was devoted to an "InvenioRDM-Instance-Rodeo," in which maintainers of local instances showcased their implementations, sharing their local innovations and customizations.

Next, utilizing proven methodology, Open space technology was used to enable participants to propose and vote upon topics for discussion. The topics included:

Glacier/cold storage handling
Full-text search
Automated metadata extraction from files
User experience on the deposit form
EduGAIN interfederation
Accessibility issues
Self-contained HTML reports
RO Crate previews
Invenio-stats
Translations
Thesis Fields
Scope of the Partner Meeting and Community topics
Maintainers playbook

Tangible outcomes

A few of these topics have had concrete effects visible today, including:

The deposit form now offers enhanced customizability, allowing users to hide or show fields, reorder components, and easily locate elements for override.
Available metadata fields have been expanded to include new optional fields, such as thesis, improving content description and FAIR compliance.
Joining the maintainers team is now more accessible with the introduction of the new Maintainers Handbook.
Partner Meeting structure decision: maintain the first two days for presentations and general information so those considering the software can attend just the first 1-2 days for a general overview.

Thanks

Many instance presentations from the Rodeo can be viewed in the InvenioRDM Partners Workshop 2025 Zenodo community. Many thanks to the local hosts at Universität Hamburg for their hard work and hospitality, and thanks also to Steve Diggs!

Repositories and libraries: Invenio features InvenioILS!

2026-01-01T00:00:00Z

For years, the Invenio community has been known for powerful open-source repository solutions - especially InvenioRDM. But many institutions need more than a repository: they need a modern, flexible platform that also supports collections, loans, acquisitions, and patron services.

InvenioILS brings these capabilities into the Invenio family, offering a fully open, customizable Integrated Library System built on the same modular framework as InvenioRDM.

A modern, feature-rich ILS (Integrated Library System)

InvenioILS delivers the essential tools libraries rely on:

Circulation & patron services: loans, renewals, requests, notifications, and configurable policies.
Cataloguing & metadata: flexible JSONSchema-based records, holdings, items, and fast discovery powered by Elasticsearch/OpenSearch.
Acquisitions & ILL: vendor management, orders, budgets, document delivery, and interlibrary loan workflows.
Clean, responsive UI: a modern interface designed for both librarians and patrons.

Flexible and extensible by design

InvenioILS is highly modular and, like InvenioRDM, based on Invenio Framework. It becomes easier to share the expertise between these two products. Institutions can customize metadata, circulation rules, UI components, and system behaviour - or integrate with external services through the REST API. This flexibility allows libraries to adopt InvenioILS “as is” or tailor it into a specialized solution that fits their workflows and identity.

CERN: A Showcase Installation

An example of an advance deployment of InvenioILS is at CERN - CERN Library Catalogue, where the platform powers daily library operations and serves thousands of users.

Smooth SSO integration

InvenioILS connects seamlessly to CERN’s institutional Single Sign-On, providing automatic user provisioning, role synchronization, and a unified login experience across CERN services.

Bulk e-book importer (CERN customisation)

CERN uses an automated bulk importer to process large vendor MARC packages. Metadata is transformed, validated, and ingested with minimal staff intervention - turning what once took days into an efficient, scalable workflow.

Mobile self-checkout & campus navigation (CERN customisation)

CERN’s library offers mobile self-checkout, enabling patrons to borrow items directly from their phones. Through InvenioILS APIs, the system also integrates with CERN’s internal navigation app, guiding users to shelf locations and showing real-time item availability.

Reliable at scale

The CERN installation demonstrates how InvenioILS can support large user bases, extensive collections, and demanding operational environments - all while integrating with CERN Institutional repository, based on InvenioRDM.

Part of a product family

InvenioILS complements InvenioRDM and the broader Invenio framework, providing a cohesive platform for managing both library collections and research outputs. With shared components and infrastructure, institutions benefit from consistent authentication, indexing, APIs, and development practices across products.

Invenio does not only power repositories.

It powers libraries as well - with power of Free Open-Source Software

Join the Invenio community

InvenioILS and InvenioRDM thrive thanks to a global community of libraries, research organizations, developers, and service providers. Whether you’re evaluating Invenio, looking to contribute, or simply curious, you’re warmly invited to join. The InvenioRDM community meets regularly to discuss features, share insights, and collaborate on the future of open knowledge systems - we would like to recreate this approach for InvenioILS and hear your feedback. Your voice is welcome - come help, shape what’s next.

datasafe: Implementing a Dark Archive with InvenioRDM

2025-10-15T00:00:00Z

We've recently completed a major overhaul of our research data archive, datasafe, which is now built on InvenioRDM. Our team at the University of Münster's University and State Library (ULB) has been working to implement this open-source software, tailoring it to meet the specific needs of our researchers.

datasafe is designed as a dark archive, where researchers can deposit their datasets for long-term preservation. Once archived, the data will be retained for a minimum of 10 years, and up to 15 years, after which it will be automatically deleted. To streamline the process, metadata can be easily imported from our local Current Research Information System (CRIS).

One of the key benefits of using datasafe is the ability for researchers to demonstrate compliance with funder requirements for data storage. Upon archiving their data, users receive a certificate that confirms their datasets have been securely stored, making it easier to report to funding agencies.

A notable feature of our implementation is the heavily customized frontend, which provides a minimalist and modern interface. By simplifying the user experience, we've created an intuitive platform that meets the unique needs of a dark archive, where data is primarily stored for preservation purposes rather than dissemination. This approach allows researchers to easily deposit and manage their data, while also ensuring the long-term integrity and authenticity of the archived materials.

Another significant improvement is the increased upload limit, now up to 5 TB in size. This community-driven feature, which our team helped implement, enables researchers to upload large files directly through their web browser into the underlying S3 storage backend, eliminating the former need for an SFTP server. This will greatly benefit researchers working with large datasets, allowing them to archive and preserve their data in a single, convenient location.

By building on InvenioRDM, we're able to leverage a community-driven solution that is actively maintained and developed by a global network of institutions and experts. We're pleased to be part of this community and look forward to sharing our experiences and feedback to help shape the future of the platform.

For those interested in learning more about our implementation of InvenioRDM, you can contact our team at forschungsdaten@uni-muenster.de!

Ensuring the Future of Digital Repositories in West and Central Africa: A Case Study on BAOBAB and Sustainable Repository Development

2025-08-11T00:00:00Z

In today’s rapidly digitizing research landscape, the ability to preserve and share knowledge openly is not just a technological challenge, it’s a matter of equity and innovation. Yet, across much of West and Central Africa, many institutions still struggle with inadequate infrastructure for digital preservation. As I shared at a recent OR2025 session, this is where BAOBAB comes in.

Why Repositories Matter

Open Access repositories are critical tools for fueling collaboration, driving innovation, and reducing information inequality. But for many African universities and research centers, significant barriers, from unreliable digital systems to a lack of preservation strategies, hinder their ability to contribute to and benefit from the global research ecosystem.

Introducing BAOBAB

Launched in 2024 by WACREN, the West and Central African Research and Education Network, BAOBAB is an InvenioRDM-based regional repository service designed to close these gaps. As of now, it supports more than 14 institutions across 8 countries, including public universities, research councils, and technical institutes. What sets BAOBAB apart is its ability to offer a centralized infrastructure while preserving local institutional autonomy, a model that ensures scalability without sacrificing ownership.

Tackling the Challenges

Before BAOBAB, institutions operated in silos. Many lacked reliable backup systems, and technical capacity varied widely. This often led to duplicated efforts and even loss of valuable data. BAOBAB’s shared, cloud-based platform reduces the technical burden for individual institutions, while a community-driven governance model ensures the service evolves with user needs.

What’s Inside the Repository?

BAOBAB has quickly become a treasure trove of African research. From PhD theses and conference papers to datasets, the repository ensures materials are enriched with multilingual metadata and accessible through persistent identifiers. That last point is key.

Why ARKs?

BAOBAB adopts ARKs (Archival Resource Keys) as its persistent identifier system. Why? Because they’re:

Cost-effective and flexible
Free from vendor lock-in
Interoperable with existing and future PID systems

Each item in BAOBAB automatically receives an ARK, ensuring it can be reliably found and cited over the long term.

Building for the Long Run

But identifiers alone don’t guarantee preservation. BAOBAB is developing a comprehensive digital preservation strategy, including:

A formal preservation policy
Offsite backup strategies
Regular file fixity checks
Compliance with international metadata standards

These efforts aim to make the repository resilient, even in the face of funding or infrastructure challenges.

Learning from Global Peers

BAOBAB didn’t start from scratch. It was inspired by global leaders in repository development, including:

LA Referencia in Latin America
HAL in France
The Africa Open Science Platform

These models were carefully studied and adapted to local realities—proving that global solutions can be tailored to local contexts.

Measurable Impact

Since its launch, BAOBAB has archived over 408 documents, reduced duplication across participating institutions, and improved the visibility of African research on a regional and international scale. That’s not just a technical win, it’s a step toward equity.

A Blueprint for the Continent

BAOBAB is more than a platform, it’s a blueprint for sustainable repository development across the continent. It shows that with thoughtful design, community involvement, and the right infrastructure, even resource-constrained regions can lead in Open Access.

Final Thoughts

Sustainable, equitable, and locally owned repositories aren’t just possible, they’re essential. BAOBAB’s success offers hope and a roadmap for other regions aiming to strengthen their research infrastructure. At WACREN, we believe the future of African research lies in collaboration, openness, and resilience.

Let’s build it together.

InvenioRDM v13.0 released

2025-07-29T00:00:00Z

We are excited to announce the release of InvenioRDM v13.0 on July 23, 2025! This new version brings a host of new features and improvements, building upon the strong foundation of previous releases.

Try it

Want to try the new features in v13.0? Just head over to the demo site. If you want to install it, you can follow the installation instructions.

Release Notes

See the full release notes for details on all the new features, and the upgrade guide for instructions on how to upgrade from your v12 instance to v13.

What's new?

InvenioRDM v13.0 focuses on enhancing administrative capabilities, improving community features, and introducing powerful new tools for data management and organization. Here are some of the major highlights, be sure to check the release notes for a comprehensive list of new features and improvements.

Communities enhancements

The Communities functionality has been significantly expanded with exciting new features:

Themed communities

Communities can now be customized with unique fonts, colors, and homepages. These themes apply to all community pages, including records and requests, allowing for a more branded and distinct user experience.

Subcommunities

InvenioRDM v13.0 introduces hierarchical relationships between communities through subcommunities. This allows for structured organization, where records from child communities are automatically indexed in parent communities. A dedicated "Browse" page lists subcommunities and collections, providing a clear overview.

Collections

Collections are a powerful new feature enabling dynamic, query-based groupings of records that automatically update. Collections provide dedicated pages for records matching specific criteria and can be organized hierarchically by subject, resource type, funding program, or other metadata fields. This is ideal for grouping content by research disciplines, publication dates, or funding programs.

Curation checks

It is now possible to configure automated checks in your communities to provide instant feedback on draft review and record inclusion requests. Checks provide feedback to both the user and reviewer that submissions to your community are compliant with your curation policy.

Files uploader & S3-compatible storage

The new file uploader delivers a faster, more intuitive, and modern file upload experience. It also enables advanced features such as multipart file transfers with S3-compatible storage backends.

FAIR Signposting

With v13, you can now enable support for FAIR Signposting level 1 in addition to the automatically enabled level 2.

Sitemaps

InvenioRDM v13 introduces the automatic generation of sitemaps to help search engines and other crawlers discover and index your repository's content. Sitemaps are even automatically linked in your robots.txt.

DOIs on demand

You can now let users to choose if they need a DOI or not when uploading.

New metadata fields

We have introduced new metadata fields that will allow you to capture more useful information when uploading:

A dedicated copyright field is now available, ensuring clear and comprehensive copyright information.
We've added a new Thesis set of optional metadata fields. We've also reorganized the thesis section, grouping thesis fields together.
The edition field has been introduced under the Imprint set of optional metadata fields, providing a way to specify the edition of a book.
A new identifiers field, composed of id and scheme, has been added to the Meeting set of optional metadata fields.

Dashboard: shared with you

You can now easily find records and requests shared with you.

Compare revisions

A brand new compare revisions feature has been introduced, allowing administrators to audit record updates and track changes over time. This provides a transparent history of modifications, crucial for data integrity and compliance.

Audit Logs

A significant addition is audit logs, a new feature designed for tracking system activities and providing a detailed record of events accessible from the administration panel.

Jobs

This release introduces a new jobs feature, providing a comprehensive way to manage asynchronous tasks via the UI or REST API. Jobs are triggered via the admin UI or REST API, run using Celery, and support logging, argument validation, and result tracking.

Jobs can be very useful to setup automatic recurrent fetching of ROR and ORCID databases or other vocabularies.

Breaking changes

Make sure to read the Breaking Changes section in the release notes.

Requirements

InvenioRDM v13 requires:

Python 3.9 (end of life October 2025), 3.11 and 3.12
Node.js 18+
PostgreSQL 12+
OpenSearch v2.12+

Questions

If you have questions related to the InvenioRDM v13.0 release, don't hesitate to jump on Discord and ask us!

Credit

The development work of this impressive release wouldn't have been possible without the help of these great people (name or GitHub handle, alphabetically sorted):

Adrian Moennich
alejandromumo
Alex Ioannidis
Alzbeta Pokorna
Anika Churilova
Austin Sharp
Brian Kelly
Carlin MacKenzie
Christoph Ladurner
Cristian Pogolsha MBP
Dan Granville
David Eckhard
David Glueck
ducica
Eduard Nitu
Emil Dandanell Agerschou
Eric Newman
Eric Phetteplace
Esteban J. G. Gabancho
Fatimah Zulfiqar
Felipe Carlos
Florian Gantner
Furkan Kalkan
Guillaume Viger
Hrafn Malmquist
Ian W. Scott
Janne Jensen
Javier Romero Castro
Karl Krägelin
Karolina Przerwa
libremente
liptakpanna
Martin Fenner
Martin Obersteiner
Matt Carson
Maximilian Moser
mb-wali
Michael Groh
Mirek Simek
Miroslav Bauer
mkloeppe
Nicola Tarocco
Pablo Panero
Pablo Saiz
Pablo Tamarit
Panna Liptak
roll
Saksham Arora
Sam Arbid
Sarah Wiechers
Tom Morrell
Werner Greßhoff
Will Riley
Yash Lamba
Zacharias Zacharodimos

KTH Data Repository: Launching an InvenioRDM-based Institutional Repository

2025-03-11T00:00:00Z

KTH Data Repository - we added user guides and updated the KTH web-pages on research data management when the new repository was launched.

On the 29th of January 2025, the KTH Data Repository was launched through a team effort supported by management and policies at the KTH Royal Institute of Technology, and by partnering up with the InvenioRDM community.

A Bit of History Behind the Launch

Back in 2021, KTH had an institutional repository for publications, theses, and student reports, but no repository for research data. A recently established cross-functional research data team offers curation support for researchers depositing data to either a KTH community in Zenodo or the Swedish National Data Service data catalog, DORIS. But we had challenges the services at the time didn’t meet:

Researchers often document data at the last minute before publishing results, resulting in both stress and lower quality of metadata.
Composing good quality metadata requires researchers’ domain-specific understanding of the data as well as understanding of FAIR data principles.
Researchers prefer to keep documentation and data private or share data within a limited group before results are published. That group most often consists of collaboration partners as well as KTH staff.
KTH has a legal obligation to keep a registry of data underlying scientific results. Some types of data can’t be shared outside the organisation; other types can only be shared if legal agreements are signed.
Data-driven research areas produce much larger data-sets than the current limits to deposit in the institutional repository and most external repositories.

Since the team also supported researchers writing data management plans, we knew that in a few years there would be a large number of projects in need of long-term archival storage compliant with funder requirements on FAIR data. Many of those projects also had data that could only be shared with restricted access.

In 2022 a project aiming to find out how research support could become more accessible was conducted and a group of researchers were interviewed. Conclusions from the interviews were combined with basic technical requirements collected via the KTH digitalisation platform. This resulted in a pilot project where InvenioRDM was selected as the candidate that best fulfilled the requirements collected by the digitalisation platform and criteria for a FAIR data repository according to the KTH guidelines for managing research data. The pilot was funded after a decision from a steering group which allowed for an IT-developer to join the team, and KTH also decided to join the InvenioRDM project as a partner.

The pilot was extended in 2023, when test users from different research communities tried out the system and provided feedback during demos, and more user stories and insights to the pilot team. 2024 was the year of management decisions on long-term maintenance and work on deployment, where we could get access to devops expertise for deploying on Kubernetes thanks to shared interest with the Scilifelab research infrastructure. Kubernetes allows for scaling services running on the platform and ensures high availability, so we hope that contributions in the area may be of use for other partners.

Where are we now?

It has been very fruitful to be part of the global InvenioRDM Open Source Community, and we are very grateful for the support we’ve received from the community. For KTH, the most interesting features that came with InvenioRDM were the ability to create communities, the easy-to-use deposit interface, and the APIs. The APIs are crucial since many research groups are using automated workflows. For data governance reasons, we needed to customize InvenioRDM so that all data must belong to at least one community. We also see the need in the near future to make changes to the log-in mechanism, where collaborative research makes it necessary to enable log-in for collaboration partners but with stronger means of verification of the identity of users in a way that complies with the eIDAS regulation.

We have received quite a lot of interest since the launch both from KTH researchers interested in using the repository but also from other universities and research infrastructures in Sweden. We are open for discussions with anyone interested in the KTH data repository - so don't hesitate to contact us if you want to learn more!

FreiData: InvenioRDM at the University of Freiburg

2024-12-09T00:00:00Z

Image by Sandra Meyndt/University of Freiburg

When InvenioRDM launched on 27th of October 2023 at the University of Freiburg under the label FreiData, campus storage options which minted DOIs for publications already existed. For over a decade, Freiburg has hosted an institutional repository called "FreiDok“, which is also used as a documentation system for the university's publication activity. In its policies for handling research data and for Open Science, the university recommends that researchers prioritize visibility to their key target audience as the most important criterion when publishing research data. This often leads researchers to choose externally operated repositories.

Why did the University of Freiburg still decide to establish its own instance of InvenioRDM? And why did the university's management commit to sustainable operation?

Motivated by the principles of scientific accountability enshrined in the state higher education law of Baden-Württemberg, as well as globally increasing scholarly adoption of Open Science best practices, University of Freiburg has embraced the publication of research results (i.e., data) as a fundamental mission of the university. In its Open Science policy, the university has set the goal of independent operation at the infrastructure level, acknowledging that local repository control allows more flexibility for researchers. While the established FreiDok service could accommodate research data, on a technical level, it could only handle smaller data volumes. These limits were established at a time when larger volumes of research data were not typically intended for publication. Therefore, there was a need for an addition to the existing service to connect storage systems with greater capacity in the background. During the same period, an object storage system with several petabytes of capacity was put into operation. This storage is operated in the same server room from which the local instance of InvenioRDM is delivered.

Automation and FAIRness are crucial desired features for storage of Freiburg's large-volume research datasets. Both local research data and larger Open Science policies require Findability and Accessibility of datasets, Interoperability, and clear guidelines for data Reusability. For all these features to be delivered as concisely and easily as possible is key. InvenioRDM meets the challenge in various ways, including automatically issuing DOIs for deposits, use of the DataCite metadata schema, use of open protocols, clear licensing options, and having a corresponding REST API for every function available on the web interface. These are vital tools for allowing researchers and institutions to map the lifecycle of research data. However, launching the repository is just the first step. The next step is to engage researchers to use InvenioRDM.

FreiData and User Engagement

Data quality control and user control of datasets were clear local use cases and needs for the InvenioRDM instance. An InvenioRDM feature which accommodates both needs is the ability to create communities in a self-service manner, where registered users can come together. Community owners can personalize their space and assign their communities distinct names via a brief abbreviation. The abbreviation is appended to the domain name of the InvenioRDM instance and serves as the hostname for the group's deposited research outputs.

In a community, various roles with increasing permission levels are predefined: Reader, Curator, Manager, and Owner. In addition to controlling community branding and membership, those with higher permissions can initiate more focused tasks, such as assigning review of a resource to another community member before its record is published and issued a DOI. The nuanced roles allow the establishment of group-specific workflows through which internal review and quality control can be achieved for all deposits. Such review processes allow each community to set quality standards for its FreiData publications. The criteria for these quality standards can be defined and documented by the community itself.

In this way, communities organized around specific subjects leverage their own expertise to ensure high-quality data deposits. The resultant benefits to individual researchers, to the communities that they form, and to both data depositors and data re-users, are the underpinnings of growing user engagement with FreiData. And if this engagement continues to increase, it bodes well for the long-term sustainability of FreiData.

Despite desire for local repository control, University of Freiburg never intended to tax its capacities to the point of offering all repository IT services in-house. From the beginning, the operational planning of "FreiData" included the intention to make its InvenioRDM installation available to others. The DataPLANT consortium, a partner within the National Research Data Infrastructure (NFDI), operates another instance of this InvenioRDM installation under its own domain. Partners in this installation offer each other mutual support, leading to sustainable operation.

The computing center of the University of Freiburg supports work on its installation through its involvement in the InvenioRDM Developer Community.

Zenodo/InvenioRDM participation at GREI Annual Meeting, Chicago

2024-11-19T00:00:00Z

Merchandise Mart, River North, Chicago, IL by w_lemay, CC BY-SA 2.0 via Wikimedia Commons

On September 19-20, 2024, members from the seven participating repositories in the Generalist Repository Ecosystem Initiative (GREI) met in Chicago for the GREI Year 3 Annual Meeting to celebrate the achievements of Year 3 of the National Institutes of Health (NIH)-funded, data sharing and open science-focused, four-year initiative. The event was held at Chicago’s historic Merchandise Mart at MATTER, a global healthcare startup incubator, community nexus, and corporate innovation accelerator.

The GREI program’s primary mission is to establish a common set of cohesive and consistent capabilities, services, metrics, and social infrastructure across generalist repositories. The program’s secondary mission is to raise general awareness and help researchers adopt FAIR principles to better share and reuse data. This is particularly important for the health research community to fulfill data sharing requirements per the NIH’s Policy on Data Management and Sharing. Generalist repositories are essential infrastructure to enable data sharing when discipline-specific or institutional repositories cannot be identified or do not exist.

The plans for the third year of GREI have been nearly fully implemented, and the group that gathered in Chicago last month reported on outcomes and completed planning exercises for priorities for Year 4. The Year 3 plan is published and available on GitHub, and is broken down into eight objectives. Below is an outline of how Zenodo representatives contributed to each objective, foundational work which often aligned with improvements being made for the InvenioRDM development and user communities:

Desirable Characteristics for All Data Repositories (for sharing scientific data)

Zenodo team members co-led the effort to update the Generalist Repository Comparison Chart, a tool to help researchers select the best generalist repository to meet their needs. The new version is projected for Spring 2025
Zenodo team members co-led the team which created the Generalist Repository Selection Flowchart, a tool designed to guide users through a series of considerations while selecting the best repository to share data

Consistent Metadata

Zenodo and InvenioRDM continue to adhere to the recommended GREI standard, which incorporates the DataCite metadata schema. In addition, InvenioRDM already has capability for further suggested metadata enhancements, such as incorporation of LCSH and MeSH subject headings and CRediT contributor role taxonomy roles

Search & Browse

Work is ongoing to explore a cross-GREI-repository search option. Meanwhile, InvenioRDM has already achieved a further recommendation: incorporation of ROR, the identification registry for research organizations

Analytics & Reporting

Zenodo team members are co-leading the effort to share the Make Data Count usage metrics with the wider community via DataCite

Use Cases

Zenodo team members co-led the task group responsible for updating the Use Case catalog with two new use cases: one outlining a scenario where a researcher utilizes a generalist repository in lieu of a local institutional repository, and one scenario where a researcher must deposit portions of a heterogenous dataset into several different repositories, one of which is a generalist repository

Connecting Digital Objects

Zenodo team members are working on the effort to map DataCite relationTypes to all types of non-data research objects, and to submit any needed updates to DataCite via API
InvenioRDM has also completed work on the FAIR signposting recommendation, thanks to Guillaume Viger’s work on signposting

QA/QC

Zenodo team members are contributing to a comprehensive review and report on approaches to handling personal or sensitive data, as well as a comprehensive review of how QA/QC on data is done at each GREI repository

Training & Community Engagement

Zenodo team members are actively involved in the multiple GREI webinars, conference proposals, and blog posts produced each project year

The GREI-Zenodo team members look forward to continuing our efforts through all the above projects, and new ones in Year 4, to provide an interoperable, robust repository option to US-based health researchers. Additional GREI milestones that can positively impact InvenioRDM development will be reported on in future telecons.

This work was supported by the National Institutes of Health (NIH) Office of Data Science Strategy/Office of the NIH Director pursuant to OTA-21-009, “Generalist Repository Ecosystem Initiative (GREI)” through Other Transactions Agreement (OTA) Number OT2DB000013.

InvenioRDM v12.0 released

2024-08-20T00:00:00Z

We are happy to announce the release of InvenioRDM v12.0! Released on August 1, 2024, it is the first InvenioRDM release since v11.0 was released on January 23, 2023, and it includes not only many bug fixes and small improvements but also several major new features listed below.

Since the v11.0 release several InvenioRDM partners have launched InvenioRDM instances in production, including Zenodo, which migrated to InvenioRDM in October 2023. A number of these partners have migrated already to InvenioRDM v12.0 or plan to do so in the coming months.

German translations for v12 will be coming in v12.1 targeted for release in October 2024. We hope that other language translations will follow suit. Work on InvenioRDM v13.0 has already started and you can track it here.

Try it

Want to try the new features in v12.0? Just head over to the demo site: https://inveniordm.web.cern.ch. If you want to install it, follow the installation instructions at https://inveniordm.docs.cern.ch/install/.

Release Notes

See the full release notes at https://inveniordm.docs.cern.ch/releases/v12/version-v12.0.0/ and the upgrade guide at https://inveniordm.docs.cern.ch/releases/v12/upgrade-v12.0/.

What’s new?

Usage statistics compliant with MakeDataCount and COUNTER

A major new feature in this release is the integration of invenio-stats, a powerful and flexible extension for measuring usage statistics of InvenioRDM records. Usage statistics are compliant with the MakeDataCount and COUNTER standards. Usage statistics are displayed in the record landing page and record search:

Record inclusion in multiple communities

A record can now belong to multiple communities. Including a record in multiple communities will let different curators change the files or metadata of the record.

Powerful and reliable record access

Giving and requesting access to records have seen a complete overhaul in this release. Record owners or curators can share them directly with other users or with groups, as well as control whether, with whom, and how access can be demanded.

Notifications

A notification system has been introduced. Users can now receive email notifications depending on their preferences when they are involved in certain activities.

Moderation of users and records

The administration panel now includes a "User Management" section to suspend, block and delete users, and undo all those actions. Records can also be deleted (with a grace period for appeal or undoing), which empowers administrators to enforce institutional policies and fight spam.

DOIs for concept records and no DOIs for restricted records

InvenioRDM now mints a concept DOI for every record by default, similar to what Zenodo has done for many years. Along with this update, restricted records will now stop minting a DOI upon publication thus keeping private records truly private.

Even better previewers

Audio and video previewers come by default. Tiling support for the International Image Interoperability Framework (IIIF) API standards is available. Text previewing has been made much more resilient.

External resources integration (e.g. GitHub)

The landing page for a record can now provide nicer visuals for configured related works. A configuration variable can be set to highlight some of the referred platforms.

Skippable community submission review

With InvenioRDM v12, you can adjust whether a record review is always required for your community, or if curators, managers, and owners can submit a record without the review process.

Banners

With the addition of invenio-banners, you can easily add and manage important alerts and messages on your InvenioRDM instance, such as system maintenance notifications or announcements.

Static pages - administration panel

Another notable addition is the integration of invenio-pages with the administration panel. It exposes a convenient UI to create and manage static pages.

Breaking changes

Make sure to read the Breaking Changes section in the release notes.

Limitations and known issues

Translations for v12 will be coming in v12.1 targeted for release in October 2024.
Sharing a secret link to a restricted record in a restricted community does not provide access to the record yet. Work on this is tracked here.

Requirements

InvenioRDM v12 requires:

Python 3.9, 3.11 or 3.12
Node.js 18+
PostgreSQL 12+
OpenSearch v2

Support for older versions of Elasticsearch/Opensearch, PostgreSQL and Node.js has been phased out.

Questions

If you have questions related to the InvenioRDM v12.0 release, don't hesitate to jump on Discord and ask us!

Credit

The development work of this impressive release wouldn't have been possible without the help of these great people:

CERN: Alex, Anna, Antonio, Carlin, Fatimah, Javier, Jenny, Karolina, Lars, Manuel, Nicola, Pablo G., Pablo P., Pablo T., Yash, Zacharias
Northwestern University: Guillaume
TU Graz: Christoph, David, Mojib
TU Wien: Max
Uni Bamberg: Christina
Uni Münster: Werner
Front Matter: Martin
KTH Royal Institute of Technology: Sam
Caltech: Tom

Conference Spotlight: InvenioRDM Workshop Day at Open Repositories 2024

2024-07-19T00:00:00Z

The 19th annual Open Repositories conference took place June 3 - 6, 2024 in Göteborg, Sweden. Attendees benefitted from hundreds of presentations on innovations in the world of digital repositories, all focused around this year's conference themes of Community, Transparency, and Sustainability. In addition, ten workshops were presented on June 3rd, the InvenioRDM Workshop being one of only two to receive a full-day allotment.

At our third annual OR InvenioRDM Workshop, over a dozen of our community members collaborated to present and discuss the software, and to showcase features and customizations currently in place at our own institutions. These include:

Caltech's guide to customizing templates, vocabularies, and the deposit form
NYU's multiple storage configuration
University of Bamberg's UI-customizations, custom subjects and vocabularies, and deployment concept
TU Wien's innovations on authentication and digital preservation
University of Münster's use of Devbox to provide a streamlined InvenioRDM installation and development experience
Northwestern University's implementation of Signposting in InvenioRDM
CERN's demo of the Zenodo and CDS installations, as well as a v12 features demo
KTH Royal Institute of Technology's customizations of InvenioRDM

In the afternoon, we offered two separate tracks. In the first we hosted self-organizing technical discussions, including a hands-on demonstration of Devbox presented by project partners from Uni Münster. We also encouraged participants to install InvenioRDM on their machines with help from technical experts, many of whom had already spoken in the earlier showcase session.

In the second track, we focused on community discussions with two additional presentations:

The InvenioRDM Community: what it is and how to join
Supporting Transparency and Compliance with InvenioRDM, a presentation from Northwestern on how InvenioRDM's features support compliance with the increasing number of public access and data sharing requirements for researchers

An informal survey of our 30+ participants afterward revealed that the workshop was generally very favorably received. Participants enjoyed the general introduction to the platform, as well as feature demonstrations and the opportunity to speak with members of the community. While some participants found installation challenging, others found deployment easy. Overall, participants found the experience to be valuable, with some encouraging us to offer the workshop again next year with minor tweaks.

Our active community participated in more than just the InvenioRDM workshop, however. Project partners also participated in:

The Repository Rodeo: Alex Ioannidis on behalf of Zenodo
A Data Repositories and Lessons Learned presentation from KTH on the KTH data repository based on InvenioRDM
An outline of Zenodo's role in the Generalist Repository Ecosystem Initiative, presented by Northwestern University, in Empowering Global Progress: GREI Coopetition's Role in Standardizing Transparency, Community, and Sustainability Initiatives
24x7 session on Integrations for Sustainability and Transparency: NYU's presentation We Can Work It Out: Cross-Functional Collaboration on Repository Strategy
A Developer Track session, Repositories and Computation: Crossover Episode presented by TU Wien

At the end of the conference, it was announced that the next Open Repositories will be hosted in Chicago from June 15 - 18, 2025. We will aim to host an InvenioRDM workshop once again, and we hope to see you there!

InvenioRDM Partner Meeting / Developer Workshop: Action Items

2024-04-24T00:00:00Z

From March 18 - 22 the InvenioRDM partner community met in Münster, Germany, to discuss current challenges and plans for the future of the project. While many smaller questions and issues were addressed immediately on-site, e.g. during one of the “merge parties” or in work sessions resulting from and building upon previous discussions, many sessions ended with clear lists of planned actions for the coming weeks and months.

To showcase the kinds of actions we decided on, and to have something to measure our success by at the next developer workshop, we're sharing here our full list of action items. In the fall, approximately at the half-way-point between the last and the next workshop, the community will meet at a themed Telecon to evaluate this list and see what the status on these plans is.

In the following you can find a list of all sessions and the actions decided upon by the participants. Some of the plans mentioned were already set into action - a huge thank you goes to the motivated community members working on all these actions!

Community engagement and handling pull requests

document this process in invenio-docs (curator role, responsibilities, templates, schedule, reviewer guide)
make PR checklist more extensive (criticality, bug/feature, packages impacted, target release (old, current, new brach))
merge party this afternoon and on a regular basis
dream parties (form pitch to architecture proposal, mockups)
demo parties
public documentation of people to talk to for packages and features (e.g. list of module maintainers)
dedicated role for community contribution management
identify people involved from the community and their involvement (e.g. searchable expertise board)

Translations / i18n, i10n

test if yaml-files can have four letter codes
group tests Münster approach way of overwriting python strings, java-script
Zack and David try to implement the Münster approach into the official cookiecutter so its easier to use and has a global storage of frontend translations

Large file management

Multipart uploader
- Issue 2612
File linking API
- Issue 2613
File metadata
- Issue 2614

Invenio RDM Version 12

fix the blockers until end of April
document features
translate in May
going forward and release v12 before open repositories

Invenio CRIS

further call for interest in InvenioRDM visio conferences (presentation)
Created a GitHub Discussion to gather more use cases and interest from the community inveniosoftware/product-rdm#153
decide between specific in communities and more ambitious in a specific module based on traction
If invenio-communities is the way to go, try to put specific fields in community data model in a bucketed way
deepen code review

Customising Schema

Write a feature request for overlay config for required fields per instance / community.
Improve docs on Custom Fields - checklist on when they're useful, community examples.
Case studies / knowledge sharing of the 'hard way' customisations that have been achieved.

Deployment

do a Session on small one VM deployments
do a Session on Helm charts
update deployment documentation

Digital Preservation

Brush up Invenio-SIPStore (Max) Check out Invenio-Archivematica, or look at the AM Python client (Max) Check if an AM plugin for Invenio is needed (Panna?)

Workflow to keep internal roadmap updated

GitHub Discussions implemented for feature request forum
CERN will do quarterly cleaning & updates of roadmap items (Jan, Apr, Jul, and Oct) and will host talks about issues at corresponding Telecons

RDM Curation Workflow

open question is if/how this could go into the core or if it should remain an extra package maintained by TU Graz
ensure possibility to configure aspects in order to meet discussed requirements, use cases, and user stories
upload package to GitHub for future collaboration
after follow-up discussion, one idea is to extend the functionality of the requests in the core and build the curation on top of these changes
- more details
- Split approval and acceptance of requests
- Requests can be extended with requirements/checks. These have to be fulfilled before a request can be accepted.
  - Can be specified per request type
  - Repository wide requirements are applied to all request types

Harvesting vocabularies

Start a coordinated sprint in May (CERN and Uni Münster).

Creating a place to collect Invenio modules etc.

Awesome Invenio - links to the actual GitHub Repository where everyone with a GitHub account can add their or other projects and create a PR for it. It's mainly curated by @egabancho for the time being.

Community workflow improvements

Pitch governance documentation update
Propose group structure (InvenioRDM Interest and Task Groups)
Propose new/additional way of subscribing to the list-serv
Pin and list welcome message and resource list to the welcome channel in Discord

Developing with Devbox

share repository with all used scripts with community

Repeatable installs

always copy package-lock.json from ../var/instance/assets/ to my-site/. Make sure that git ignores the creation/modification file dates if the file content didn't change
have an extra CLI param in invenio-cli that will:
- copy the package-lock.json from my-site to ../var/instance/assets/
- if the my-site/pipfile.lock is newer than my-site/package-lock.json, then overwrite or fail (some python deps might have changed)
- run npm ci instead of npm install
the development workflow with linked editable (watched) modules should be tested, to ensure nothing breaks
when in NODE_ENV=production, the above should always happen. When in NODE_ENV=dev, then I can use a new param invenio-cli install --keep-deps
New issue tracked here: inveniosoftware/product-rdm#179

One VM Deployment

improve documentation with the deployment
- Docu PR work in progress
cookiecutter issue: inveniosoftware/cookiecutter-invenio-rdm#278
- have a finished docker-compose without dev container
- improve secrets by default

Preservation of workshop outcomes

Create a Google form for filling in session information (→ Steve)
Process information from the Google form entries (→ Markus)
Create a repo in the inveniosoftware namespace (→ someone from CERN)

Kubernetes Helm-charts requirements

Issues on the [helm-invenio Github repo](https://github.com/inveniosoftware/helm-invenio/issues)
Share of current community charts:
- University of Münster: helm pull oci://harbor.uni-muenster.de/ulb/invenio --version 0.4.0

Workgroup structure - Governance

please contact Lars if you know a good mailing list software! - in progress

InvenioRDM feature request form

GitHub Discussions feature request forum implemented

Usage statistics

Created discussions
- inveniosoftware/product-rdm#181 inveniosoftware/product-rdm#182

Evaluate proposed strategies for handling PRs

regular merge parties right after Telecons (cut Telecons to 30 minutes) (Sara)
train more people as maintainers with merge rights (in on-site meetings) (CERN)
figure out a clear labeling system (Karolina)

InvenioRDM Partner Meeting Summary, March 2024

2024-04-23T00:00:00Z

The InvenioRDM partner community met in Münster, Germany from March 18 - 22, 2024, for our first in-person annual meeting in over four years. Forty-six attendees from over fifteen institutions spent the five days making connections, planning, and diligently working on all things InvenioRDM.

Organizer Sarah Wiechers, software developer and research data manager for the Service Center for Data Management, ULB Münster, proposed the use of Open Space Technology, a meeting organization methodology in which the agenda and topics discussed are voted upon and implemented by the meeting’s attendees on the meeting date. This methodology was highly effective, allowing all community members the opportunity to pitch ideas and vote on their favorites for discussion.

University of Münster’s InvenioRDM Community GitHub site contains our full output of discussions, decisions, and plans. Key topics addressed were the timely handling of pull requests, planning for the v12 release, improvements to community workflows, translations, large file management, deployment, digital preservation, vocabularies and fixtures, Kubernetes Helm-charts requirements, and much more. To help manage feature requests going forward, we have agreed to implement GitHub Discussions, a tool which will allow for providing context, asking questions, and upvoting most-wanted features. GitHub Discussions is now part of our workflow for keeping the Roadmap updated.

We were thrilled to be able to work with so many old and new friends in beautiful Münster, and a huge thanks goes to our local hosts at University of Münster for their hard work and hospitality. We hope that you will be able to join us at next year’s meeting, with Hamburg currently slated as the host city. In the meantime, please take advantage of our following updated community workflows, including:

Development-focused chats during half of each telecon
A new email list - coming soon
Newly established Interest Groups (long-standing) or Task Groups (deliverable-focused). For the current list of all Groups, see the new Onboarding page

Prism: The New Feinberg Repository for Global Dissemination of Research

2023-11-28T00:00:00Z

The Prism institutional repository has launched at Northwestern University. Prism preserves and makes available articles, conference presentations, preprints, datasets, and other items created by faculty, staff, and students. Prism helps openness, maximizes reproducibility, and enhances research connections within Feinberg School of Medicine and across the globe.

Prism replaces the former DigitalHub and includes many much-anticipated features, such as the ability to create metadata-only records for offsite datasets, set embargo dates for releasing content to the public, create and curate communities of practice, and share private links to view and edit with colleagues. These new features complement existing features such as the ability to assign Digital Object Identifiers that make records citable, indexing by Google to make research widely discoverable, and a responsive staff at Galter Library to answer questions and provide support.

Kristi Holmes, the Director of Galter Health Sciences Library, Associate Dean of Knowledge Management and Strategy, and professor of Preventive Medicine, played a leading role in the development of Prism. According to Holmes, “It is essential to have a robust institutional repository that can keep up with the latest technologies and trends. As models for open access and data sharing continue to evolve, it's clear that institutional repositories will play an increasingly critical role to make research Findable, Accessible, Interoperable, and Reusable (FAIR).” She continues, “What makes this project particularly special is the strong collaborative approach we’ve taken both at Northwestern and also in partnership with the larger open source community. I’m grateful to our team at Galter Library for their incredible work.”

“We are excited to announce the launch of Prism as the institutional repository for Feinberg School of Medicine” says Karen Gutzman, Head of Research Assessment and Communications at Galter Health Sciences Library and Learning Center. Prism builds on a strong research foundation first made possible in the DigitalHub repository. “One of the most exciting features of Prism is the ability to create communities on topics, projects, or events” says Gutzman. Communities include open access research from Feinberg on COVID-19, training presentations from the Biostatistics Collaboration Center, and the NUCATS Grant Repository, a centralized resource for grant writers and investigators internal to Northwestern provided by the NUCATS Institute. “Prism is an excellent home for the NUCATS Grants Repository, allowing us to easily share exemplar grant templates and other resources with investigators across the Northwestern University community,” says Dr. Richard D’Aquila, Director of NUCATS.

Prism is built on the InvenioRDM software, which also forms a strong and sustainable foundation for Zenodo. “With its user-friendly interface and advanced features, InvenioRDM is truly a game-changer in the world of repositories. This platform is designed to make research more accessible and open to the public, promoting innovation and collaboration within the academic community,” says Holmes. Over the past several years, CERN and Northwestern have collaborated as core co-developers of the software, partnering with the global Invenio Open Source Community to develop InvenioRDM as a turnkey, scalable, and top-of-the-class user experience software for repositories. The InvenioRDM software offers a reliable environment for science, empowering preservation, credit, discovery, and sharing while maintaining integrity in its responsiveness to the evolving needs of the research community, including data sharing policy compliance. Northwestern contributions to the open source project are led by Matt Carson, Senior Data Scientist and Head of the Galter Library Digital Systems Department, with developer Guillaume Viger leading technical work and Sara Gonzales, Senior Data Librarian at Galter Library, contributing to a wide range of efforts, including serving as the Community Manager.

Notably, Northwestern and CERN recently expanded this collaboration through an award from the NIH Office of Data Science Strategy/Office of the NIH Director pursuant to OTA-21-009, Generalist Repository Ecosystem Initiative (GREI), to Zenodo to help researchers improve discoverability of their data and lead to greater reproducibility and reuse of data. Through this award and others, the infrastructure reflects Desirable Characteristics of Data Repositories for Federally Funded Research and continues to evolve to meet research needs and support a vibrant data ecosystem.

Conference Spotlight: InvenioRDM Workshop Day at Open Repositories 2023

2023-07-10T00:00:00Z

Last month InvenioRDM project partners convened in Stellenbosch and Cape Town, South Africa for the 18th International Open Repositories Conference (OR2023). We were thrilled to offer our second InvenioRDM workshop on the 12th of June (see our wrap-up of the first workshop: OR2022 workshop). During three eventful hours we spoke with librarians, data managers, administrators, and developers about the basics of joining the InvenioRDM project and standing up an instance. We also shared information on customizations and add-ons for those interested in the platform’s advanced capabilities.

Following Nicola Tarocco’s introduction and demo of key InvenioRDM functions and features, our sessions included:

Maximilian Moser’s presentation on customizations for his local InvenioRDM instance TU Wien Research Data, as well as a CLI command enabling very large file upload
Dan Granville’s presentation on IIIF in InvenioRDM, exemplified through his work with Data Futures GmbH
Guillaume Viger’s presentation on the launch of Northwestern University’s Prism instance, with a particular focus on successful migration of awkward data
Matt Carson’s presentation on InvenioRDM’s support of the FAIR principles, with additional information on the platform’s support for data policy initiatives
Maximilian and Guillaume’s presentation on the details of, and endless possibilities for, deployment of InvenioRDM instances
Zacharodimos Zacharias’ presentation on his and the CERN team’s iterative and productive efforts to achieve fast and efficient data migration as a test for Zenodo’s move to InvenioRDM later this year.

The workshop attendees were engaged, brought great questions, and gave us much food for thought. Our conversations with potential new partners were so extensive that we were not able to include all planned presentations! Be on the lookout for future blog posts containing this bonus content.

But wait, there’s more! InvenioRDM’s OR2023 representation included Nicola’s fantastic lightning talk on behalf of InvenioRDM in the annual Repository Rodeo. In addition, Maximilian Moser and David Eckhard of TU Graz presented their work on a communication protocol to connect machine-actionable DMPs with InvenioRDM, and Zacharodimos Zacharias presented on InvenioRDM’s mature integration with ROR (the Research Organization Registry).

All of the InvenioRDM Workshop Day presentations are now available for download from Zenodo. We hope to host another InvenioRDM workshop at the 19th Annual Open Repositories Conference June 3-6th, 2024 in Gothenburg, Sweden. If we’ve missed you at our previous two workshops, we hope to see you there!

Introducing the InvenioRDM GitHub Archiver (IGA)

2023-06-26T00:00:00Z

The InvenioRDM GitHub Archiver (IGA) is a new software tool created by the Caltech Library. InvenioRDM is the basis for many institutional repositories, such as CaltechDATA, that enable users to preserve software and data sets in a long-term archive. The metadata contained in the record of a deposit is critical to making the record widely discoverable by other people. However, creating detailed records and uploading assets can be a tedious and error-prone process if done manually. This is where our new tool comes in.

IGA creates metadata records and sends releases automatically from GitHub to an InvenioRDM-based repository server. It constructs a metadata record using information it gathers from the software release, the GitHub repository, the GitHub API, and various other APIs as needed. Here are some of IGA’s other notable features:

Automatic metadata extraction from GitHub releases, repositories, and codemeta.json and CITATION.cff files
Thorough coverage of InvenioRDM record metadata using painstaking procedures
Recognition of identifiers that appear in CodeMeta and CFF files, including ORCiD, ROR, DOI, arXiv, and PMCID
Automatic lookup of publication data in DOI.org, PubMed, Google Books, & other sources if needed
Automatic lookup of organization names in ROR (assuming ROR id’s are provided)
Automatic lookup of human names in ORCiD.org if needed (assuming ORCID id’s are provided)
Automatic splitting of human names into family and given names using ML-based methods if necessary
Support for record versioning
Support for InvenioRDM communities
Support for overriding the metadata record IGA creates, for complete control if you need it
Ability to use the GitHub API without a GitHub access token in many cases
Extensive use of logging so you can see what’s going on under the hood

Data and software archived in a repository need to be described thoroughly and richly cross-referenced in order to be widely discoverable by other people. Of particular interest to software developers is that a repository like CaltechDATA offers the means to preserve software projects in a long-term archive managed by their institution. IGA helps make the creation of metadata and InvenioRDM records for software and data managed in GitHub as easy as possible.

More information about the InvenioRDM GitHub Archiver is available at caltechlibrary.github.io/iga/.

How does InvenioRDM support proper resource curation and validation?

2023-05-30T00:00:00Z

With mandates and directives such as the European Union’s Plan S, the recently-introduced National Institutes of Health Policy for Data Management and Sharing in the United States, and the White House Office of Science and Technology Policy Memo requiring free and immediate access to federally funded research in the U.S. by the end of 2025, increased emphasis is being placed on curation, validation, and preservation of research output. In addition, efforts such as the Generalist Repositories Ecosystem Initiative (GREI) aim to bring next-generation generalist repositories into alignment on key features required for discovery and reuse of research outputs. Zenodo has joined this initiative and will migrate to the InvenioRDM platform later this year, which will play a crucial role in helping to support best practices in research data management for this popular service.

From ensuring deposited records abide by FAIR principles, to granular access control, InvenioRDM supports resource curation and validation with a variety of functionalities. Below are some of the key features that make the platform a great option for sharing and preserving your work:

Deposits:

Deposited records check all FAIR principles (Findability, Accessibility, Interoperability, Reusability)
Configurable confirmation text before deposit to ensure agreement with security and privacy policies
Long-running record drafts for pre-publication edits
A wide variety of licensing options
Ability to set an embargo period for records or files with automated embargo removal at a specified date
Contextualization of records via secondary identifiers to publications, project sites, external data, and other associated research outputs
Citations in multiple formats provided for each record
Empowers communities to self-curate domain specific records
Ability to archive software via GitHub integration within records, allowing data to be linked to code [coming soon]
Usage metrics and analytics (COUNTER-compliant statistics gathering for record access and downloads) [coming soon]

Documentation:

Extensive support documentation
Possible addition of informative pages e.g., Terms of Use, Privacy Policy, Deposit Agreement.

Metadata and Discovery:

Supports industry standards for interoperability
Ability to broker metadata to search and discovery services
Hand curation by repository administrators for metadata quality assurance
Versioning of all deposits, with a per-version permanent identifier and a per-deposit (across all versions) concept permanent identifier
Configurable registration with external identifier systems e.g., DOI

Access, Security, and Quality Control:

Authentication configurable with institutional credentials or third-parties (e.g., ORCiD, GitHub)
Checksum for uploaded files ensure file integrity throughout each file's lifetime
Restricted access capability
Can be configured for secure storage
Automated periodic file audits

New version of CaltechDATA launches on InvenioRDM

2022-09-21T00:00:00Z

Caltech Library is pleased to announce that CaltechDATA, our institutional data and software repository, launched a major upgrade on Wednesday September 21, 2022.

CaltechDATA has served as critical research infrastructure for campus since 2017, and it hosts over 20,000 records containing datasets and software for a wide variety of disciplines. With this launch, CaltechDATA now runs on the open-source InvenioRDM platform and brings many new features that Caltech researchers have requested:

Easier record creation with autocomplete for creators, affiliations, subjects, and funders
Automatic record versioning
Private share link for reviewers
Improved record views, with dynamic citations and an expanded file previewer

This version of CaltechDATA also introduces communities, which enable groups at Caltech to create their own record curation and approval processes. Researchers can collect records into a single browse and search interface. A curation pipeline allows records to be submitted by Caltech users, and then approved by a defined set of curators. We’ve pre-seeded a small number of initial communities, and look forward to seeing what researchers create.

InvenioRDM is a customizable open-source repository platform developed by CERN and twenty partner organizations, including Caltech Library. It is built on the twenty-year history of the Invenio repository platform, whose most-successful implementation is the Zenodo generalist repository hosted by CERN. InvenioRDM takes the features of Zenodo and makes them customizable for institutions. InvenioRDM will enable Caltech Library to more rapidly roll out new features and collaborate with other institutions to establish repository best practices.

Conference Spotlight: InvenioRDM Workshop Day at Open Repositories 2022

2022-07-11T00:00:00Z

On June 6, 2022 InvenioRDM partners from CERN, Northwestern University, TU Wien, NYU Libraries, and Cottage Labs assembled to present the InvenioRDM Workshop Day at Open Repositories 2022 (OR2022). At this first in-person meet-up in over two years, the team talked InvenioRDM with over 30 workshop attendees.

In the morning, participants were treated to Introduction and Installation presentations, followed by hands-on assistance with running InvenioRDM on their own machines. Attendees appreciated the one-on-one help and feedback and came away with a greater understanding of how to support the repository at their own institutions.

In the afternoon, participants were treated to show-and-tell sessions on local customizations at both TU Wien and Northwestern, followed by an introductory session on partner benefits and how to join the community. Next followed fascinating in-depth presentations on increasing accessibility in InvenioRDM, building off the robust work in this field undertaken by NYU Libraries and Northwestern. A case study in data migration followed, and the afternoon wrapped with a presentation on local implementation and adoption approaches at NYU libraries, which incorporated feature and sprint-based development of repository policies. In addition, the InvenioRDM-based COVID-19 SSH Data Portal of the European University Institute was demoed by Cottage Labs.

The team was pleased to be able to share information and tips in person with potential adopters, and to experience some much-missed in-person teambuilding. Here’s hoping these meet-up opportunities increase in the future!

And although OR2022 unfortunately did not offer a hybrid option for attendance, the whole InvenioRDM community (and beyond) can experience the workshop day by visiting this Zenodo community to see individual presentations, the day’s itinerary, links to additional resources, and a feedback form.

If you’ll be attending a conference soon, and might be thinking of presenting on InvenioRDM, please let Sara or Lars know!

InvenioRDM Partner Contributions: Virtual Project Meeting, 2021

2021-10-12T00:00:00Z

On September 28-29 the InvenioRDM project team gathered again (though sadly, again, without fondue) for the Virtual Project meeting. Over 50 partners and team members were present and a great deal of helpful feedback was shared as we worked to develop the roadmap for the next stage of the project. Thanks to everyone for your contributions to the discussion, they are much appreciated!

The InvenioRDM project team would not be so robust without the thoughtful contribution of each of its members. Thank you to all of you who were able to commit the requested 1.5 person months to the project for the next year, in some cases more. The work of each partner site results in unique and valuable contributions to the team. We’d like to use this post to highlight, in a summarized format, the contributions that each project partner highlighted for us at the meeting. Here’s the amazing compendium of work that you have contributed over the past year:

BNL: Integrating the keycloak OAuth plugin module a part of the InvenioRDM framework.
Caltech: DataCite 4.3 support and JSON REST API for DOI Registration. Registering very complete metadata records with DataCite.
Data Futures GmbH: IIIF Presentation API record serializer, IIIF powered search view thumbnails, embedded Mirador 3.0 IIIF previewer.
EkoKonnect: InvenioRDM part of Open Science cloud platform.
Geo: Predefined queries for GEO Priority Areas, map preview/reference.
HZDR: Documentation of deployment steps, testing.
INFN: Community survey, requirements gathering, testing functionalities, participating in Metadata Working Group, commitment to REST API testing, writing of guides on how to integrate InvenioRDM APIs with programming languages used by research communities.
JRC: Invenio-records-resources feature to support one of January Release functionalities.
Northwestern: Various core contributions, including implementing OpenID Connect / OAuth at NU (includes SSO and MFA) and sharing local migration process and local implementation rollout.
NYU Contributed usability study findings and presented to Telecon; proof of concept geo previewer / viewer for vector GIS data; working toward extending the concept of index sample metadata records via the command line tool.
NII Testing based on local scenarios; knowledge sharing for InvenioRDM operation on Kubernetes (Monitoring, Logging and Backup strategies, etc.).
TIND Simple k8s demo setup (helm-invenio).
Tubitak Provided Turkish translations to InvenioRDM; will maintain and participate in release testing and Interest Group participation.
TU Graz: Ongoing work: 18n of the python modules; i18n of the react modules; OAI-PMH, core sprint participation.
TU Wien: During development sprints, helped implement Access Control (v1.0), Share-by-Link (v2.0), and Keycloak SSO (v4.0). Would like to contribute to Communities code in future.
Universitat Hamburg: Contribution highlights: S3 plugin, initial SAML integration. Committed to: Help with testing, documentation, dissemination.
Universitat Munster: Development of helm chart, to deploy InvenioRDM inside a Kubernetes infrastructure; implementation/documentation of Kubernetes based architecture (https://github.com/ulbmuenster/inveniordm-2-production); tests of REST API by automated (OAI-PMH based) import of datasets from the WWU document repository miami (https://miami.uni-muenster.de) into InvenioRDM.
Universitat Tubingen: Contributions planned if production status reached: dissemination, acquiring funding for development needs in interdisciplinary research projects, participation in NFDI’s (nationwide infrastructural developments in domain-related RDM).
Universitat Freiburg: Regular testing of new releases and features; automatic deployment & recovery strategies; basic UI customization.
ZHB Luzern: Testing: usability of setup resources (e.g. invenio-cli) & customization and styling; evaluation of metadata standard.

Thanks again, everyone, for your contributions! Hope to see you in person at the next project meeting.

InvenioRDM reaches major milestone - v6.0 released

2021-08-05T00:00:00Z

We're very happy to announce that InvenioRDM v6.0 LTS has been released!!!

Try It

Want to try InvenioRDM? Just head over to our demo site: https://inveniordm.web.cern.ch

If you want to install it, follow the installation instructions on https://inveniordm.docs.cern.ch/install/

Production ready

InvenioRDM v6.0 is the first release to be suitable for production services, and therefore the first to receive the Long-Term Support release label. This marks the achievement a major milestone for the InvenioRDM.

Features

Following is a high-level overview of features currently supported by InvenioRDM. This is just the beginning as we have a packed road map with exciting features ahead of us.

Records

Any resource type InvenioRDM allows you to store publications, datasets, software, images, videos or any other resource type you may have thus can serve as a single repository for all your records.
Any file format/size InvenioRDM accepts any file format in any size given that your underlying infrastructure can support it.
Versioning support Records and files are all versioned with optimized storage for large files.
DOI registration via DataCite InvenioRDM can register DOIs with DataCite for all records, and allows you to write plugins for other identifier schemes.
DataCite-based metadata InvenioRDMs internal metadata is based on the DataCite Metadata Schema which is a simple yet powerful format for describing nearly any research output (paper, data, software, ...).
Strong support for persistent identifiers Authors, affiliations, licenses, related papers/datasets etc can all be identified via persistent identifiers such as ORCID and RORs.
Extended Date Time Format (EDTF) support Publication dates and other dates support the EDTF format for recording imprecise dates and date ranges such as 1939/1945.
Previewers InvenioRDM comes with previewers for common files formats such as PDFs, images, CSV, Markdown, XML and JSON.
Citation formatting. InvenioRDM can generate citations strings for your records using the Citation Style Language with support for more than 800+ journal citation styles.
Record preview. Before you publish your record, you can see a preview of how it looks like.
Metadata-only records Both records with or without associated files are supported.
Identifier detection and validation. InvenioRDM comes with support for automatic detection and validation for a large number of persistent identifier schemes (i.e. less typing and clicking for end-users).

Search

Faceted search. InvenioRDM supports fully customizable faceted search.
Advanced query syntax. InvenioRDM has support for advanced querying via simple term search, phrase search, range search, regular expressions and custom ranking/sorting/
Auto-complete as you type. InvenioRDM exposed advanced APIs for search-as-you-type scenarios.

Auth, permissions and security

Login via institutional account. InvenioRDM makes it easy to integrate your institutional authentication provider such as e.g. Keycloak, OAuth or alternative use e.g. ORCID for login.
Restricted records. InvenioRDM supports restricting access to files only or to the entire record.
Share by link. Restricted records can be shared with peer-reviewers or your colleagues via secret links.
Embargo support Restricted records can be embargoed so that they are automatically made publicly on a specific date so that you can comply with e.g. funders' Open Access mandates.
Logged in devices. InvenioRDM allows users to see a list of currently logged in devices on their account.

Customizations

Styling and theming InvenioRDM can be styled and themed to fit into your institutional visual identity.
Custom vocabularies All vocabularies such as types for resources, dates, roles, relations, affiliations etc can be customized to your local instance.
Subjects InvenioRDM can load external subjects vocabularies used for classifications such as Medial Subject Headings (MeSH) and many others.
Permission system InvenioRDM supports advanced customizations to the permission system for e.g. IP-based access control.

APIs and interoperability

REST API InvenioRDM exposes a strong versioned REST API for all operations on the repository, that allows you to build your own integrations on top of InvenioRDM.
Export formats InvenioRDM supports exporting records metadata in multiple formats such as JSON, Citation Style Language JSON, DataCite JSON/XML, Dublin Core.

Infrastructure

Large file support InvenioRDM supports uploading and handling TB-sized files and can manage from MBs to PBs of data as long as your underlying storage systems supports it.
Multi-storage systems InvenioRDM allows you to integrate backend multiple storage systems in the same instance such as S3, XRootD and more.
Deploy anywhere InvenioRDM is a Python application and you can deploy into your institutional infrastructure wheather it is on bare metal, VMs, containers, Kubernetes or OpenShift.

Partners

The development of InvenioRDM is the result of an Open Source project kicked of 2 years ago with a diverse set of partners from all over academia and research. Most of the development and testing work has been conducted during the pandemic making it extra challenging for the people involved but also in a fully online environment.

Just the start

This release is just the start. Our next major milestone is to bring Zenodo.org on top of InvenioRDM, which means that most larger features have been shipped along the way, and that the system have been fully tested against large-scale heavy production loads.

InvenioRDM Community Spotlight: Summer 2021

2021-06-07T00:00:00Z

Summer 2021 is an exciting time in the development of InvenioRDM, as the team works towards the Long-Term Support (LTS) version. Here are just a few updates we have to share about recent work:

Contributions from the community

InvenioRDM partners are not only local implementers, but frequently contribute their coding expertise to the project. Read this recent news item about the efforts of TU Wien developer Maximilian Moser related to the share-by-link feature and the authentication modules.

Usability testing

One way that the entire community can contribute towards InvenioRDM development is to test the most recent version of the software either through your local implementation or at the CERN sandbox site. Report any bugs you find using this form. Timely and accurate bug reporting gives the entire project a boost!

Getting the word out

InvenioRDM will be featured in two sessions at Open Repositories 2021, June 7-10, “Poster Minute Madness” and the “Repository Rodeo”, both taking place on June 9. If you or someone you know would like an introduction to the software, encourage them to attend!

Are you presenting on InvenioRDM at an upcoming meeting? Please let us know using this form.

InvenioRDM v2.0

2021-03-26T00:00:00Z

InvenioRDM v2.0

We're happy to announce InvenioRDM v2.0!

Try it

Want to try the new features in v2.0 - just head over to our demo site: https://inveniordm.web.cern.ch/

If you want to install it, follow the installation instructions on https://inveniordm.docs.cern.ch/install/

Release notes

See the full release notes and upgrade instructions.

What's new?

Versioning support

InvenioRDM now supports versioning for records. By default, InvenioRDM does not allow you to change the files after you have published a record. With the new versioning support, you can the files can now be changed however in new versions of the record.

Creating new versions

Now, a user can create a new version of the record, add or remove files and publish it by clicking the "New version" button.

One way to make new versions is from the record's landing page directly by clicking on the "New Version" button. Only allowed users can do so:

Another way is while editing a previously published record:

Search

Search results will by default only display the latest published version of a record. However, toggling the flip switch in the versions facet, allows you to view and search all versions.

Landing page

The landing page for previous versions of a record clearly displays that a newer version exists, with a link to it.

Also, in the right-hand column you have a link to view all published versions:

Permissions

We manage permissions at two levels when it comes to versioning.

Each specific version can have its own record/files visibility (e.g. one version can be restricted another not). Embargo dates also apply to specific versions.

Ownership of records, secret links (see below) and sharing of records is applied to all versions. So for instance, if you create a secret link to edit a record, that link will allow the holder to edit all versions of that record.

Get an access link (REST API only)

During this month, we have also added the capability in the REST API to create a shareable link (similar to the Share or Get a link feature on platforms like Google Docs, Dropbox, ownCloud and the like). This link is interchangeably referred as an "access link", a "secret link" or a "shareable link".

You are able to create links with different permission, for instance to allow

Anyone with the link to view the metadata of a restricted record.
Anyone to access the restricted files of a record with public metadata.
Anyone with an account to edit a record.

Note: One of the current limitations (that will be resolved further down the road) is that each secret link can only grant one permission, and they're not hierarchical (e.g. the permission to read files does not entail the permission to read the metadata).

In the future, we plan to also allow sharing only a preview of an unsubmitted record. This is particular useful for instance for peer-review scenarios, where a user can prepare a dataset and provide a journal with a link to the unpublished dataset. The peer-reviewer can then anonymously access the dataset, and the researcher can later update the dataset based on feedback from the peer-reviewer.

We plan to have a first minimal user interface for allowing users to use this new feature in the April release (v3.0). Below you see some of the mockups:

Usage

A secret link can be created by sending a POST request to the record's access/links endpoint, e.g.:

curl -H "authorization: bearer <PERSONAL_TOKEN>" -H "content-type: application/json" -d '{"permission": "read"}' https://<HOSTNAME>/api/records/<RECID>/access/links

If everything went well, the (JSON) response should contain a token. To use the token, it has to be specified in the query string of a URL, e.g.:

https://<HOSTNAME>/records/<RECID>?token=<TOKEN>

This will grant the user the permission with which the secret link was created (even if they don't have an account).

Note: Optionally, an expiration date (UTC timestamp) can be specified when creating a new secret link:

{
  "permission": "read",
  "expires_at": "2021-12-31 23:55:00"
}

Upgrading

The last major new feature is that you are now able to upgrade an InvenioRDM v1.0 instance to v2.0.

Do remember to make a backup of the database and files before performing the upgrade!

Please checkout the full upgrade instructions.

REST API changes

The JSON serializations for records in the REST API have only changed slightly this month, mainly for the new versioning support:

Added new top-level key versions.
Added new top-level key parent.
Moved the top-level key conceptid to parent.id
Moved the access.owned_by to parent.access.owned_by

Minor changes

Database connection pooling: We have improved the disconnect handling of the database connection pool, so that connections are automatically opened/closed every hour.
Documentation: We have slightly reorganised the install/customize part of the documentation.
Documentation: We have slightly reorganised the REST API part of the documentation.

Known issues

Versioning

Deposit form: When you create a new version, the files of the previous version are not transferred to the new version so you have to reupload everything.

The following are all carry-overs from the previous release:

Landing page

These are all carry-overs from the previous release:

Preview: The image previewer is not using IIIF and only previews images up to 500kb.
Preview: If you e.g. upload a .JPG extension file, you can set the default preview for the file, but the list of files will not have a preview button.
Preview: The caption of the preview box does not change when selecting another file for preview.

Deposit form

Validation error messages: The form validation error messages are not displaying correctly. Labels are system names, and sometimes [Object (Object)] is displayed for errors.
Missing files: If files are missing to be uploaded, the publish button is disabled, but there is no feedback to the end-user that files are missing.
Performance: We have seen some slight performance issues when entering information into the form. We're looking into this, to make sure it is not noticeable.
Affiliations vocabulary: We have not initialized the affiliation vocabulary, and thus there is no auto-completion for affiliations (creators and contributors).
Publish button: It is not enabled when reloading a page with a valid record - you first have to save the record for the publish button to be enabled.
Protection: The files restriction in the protection widget only become visible after uploading a file. It should be hidden only for metadata-only records.
Languages: The language vocabulary search results are not ranked properly for the query. For instance searching for "english" will show less relevant results first.
Creators: The list of creators in the deposit form does not display the role nor identifier if entered.

List uploads

The title of a search result item is not clickable even though it looks clickable.

Backend

Embargo records are not automatically published due to the missing background job.
The REST API JSON serialization has timestamps that are not in UTC, nor do they have timezone information.

Installation

Some versions of NPM 7 are known to cause issues -- NPM 6.14.5 is recommended.

Feedback

For this particular release we're especially interested in hearing about your challenges and success about upgrading from v1.0 to v2.0.

Please post a message on the #rdm-general Discord channel.

For other issues, please first check if you've hit a known issue:

Bugs: First check for known issues above, then report them on the Invenio-App-RDM repository on GitHub.
Feedback: Suggestions for improvements, results of user testing, etc. - please reach out to Lars directly (Discord chat or mail).

What's up next?

During the April iteration (deadline April 30th) we're moving most of the CERN teams on top of InvenioRDM. This means, we'll be around 12 senior and junior developers working on InvenioRDM.

This in itself, poses its own challenges in onboarding new team members and reaching high productivity fast.

We'll be running three teams, each with its own focus:

Team Communites: The goal for this team is to integrate the creation of communities in InvenioRDM v3.0 (you will not yet be able to associate records with communities)
Team PIDs: The goal of this team is to ensure that InvenioRDM v3.0 can register DOIs for uploads.
Team QA: The goal for this team is to catch-up on left-overs from previous iterations, perform quality assurance and release management.

Credits

The development work in this release was done by:

CERN (Lars & Zach)
Northwestern University (Guillaume)
TU Wien (Max)

InvenioRDM v1.0 (February release)

2021-02-26T00:00:00Z

We're happy to announce InvenioRDM v1.0!

VERY IMPORTANT, read the full release notes before you install and run the system! The v1.0 label does not mean the system is suitable for production use!

FAQ (do not skip!)

Can I use v1.0 in production?

No. Do not even try, or think it might work out!

When can I use InvenioRDM in production?

The first release of InvenioRDM that you can use in a production system will be the LTS (Long-Term Support) release due in end-July.

Why the v1.0 label?

We follow semantic versioning, and in the coming months we'll be releasing v2.0, v3.0, etc. We use the v1.0 label because we're now in a state where the project partners can start running demo systems to demonstrate the progress of the project to their stakeholders. Furthermore, the project partners will be able to start user testing more widely.

Can I upgrade from v1.0 to v2.0?

Read the upgrade contract below.

Upgrade contract (do not skip!)

The following "contract" is meant to align expectations on how and what you'll be able to upgrade from v1.0 to v2.0.

You MUST expect breaking changes to anything on subsequent releases until the LTS release! REST APIs, programmatic APIs, features, Jinja/React templates, data model, vocabularies, etc.

We will ONLY guarantee that you will be able to upgrade a database created with v1.0 to v2.0. With that, we mean that through a manual, labour intensive and offline process you'll be able to upgrade your database. Basically this boils down to, that we will document the steps you need to apply in order to move your data from v1.0 to v2.0 code. In no way do we promise it will be easy! You MAY need to apply manual changes to records. We of course plan to make this an easy and smooth process for the LTS release, but for now it's not.

What's new?

Now that you've hopefully read all the above and are very scared of installing InvenioRDM in production (you should be!), then let's share a bit of the excitement we have for the release this month.

We have reached a really major milestone for InvenioRDM. We now have a minimal working repository with a minimal acceptable user experience with a very minimal feature set. From now on, we are in a releaseable state! That means that in the coming months we'll be improving the UX, the feature set, the stability and production readiness, and thus we're in a very good position for accomplishing our production-ready LTS release in end-July.

Below you'll find the specific areas we've been working on this month.

Access control

The major new feature this month is the support for minimal access control.

The deposit form now has a new protection widget that allows you to deposit records in the following categories:

Public records with public files
Public records with restricted files (with or without embargo)
Restricted records with restricted files (with or without embargo)
Public metadata-only records
Restricted metadata-only records (with or without embargo).

Below are some examples of how this now looks in the deposit form:

Restricted records and files

Restricted records and files are only visible for the owner (the user who deposited the material). This means that e.g. a restricted record will not show up in the search results of other users.

User without access (no files and no edit button visible):

User with access (files and edit button visible):

Note, that the message "Sorry, the files are restricted!" is a temporary message, until we have a solution for the access badge (see next section).

Limitation: Access status

We have temporarily removed the access status badge (see before/after screenshot below) until we have found a suitable solution for the badge. The current badge was not easily transferable to the new access control, and is mixing indications about if you can access a record, and/or if files are available.

Before:

After:

Limitation: Embargo feature

You can apply an embargo period to restricted files or records. Once an embargo period is over the record/files are supposed to be made automatically publicly accessible. In v1.0 however, we were not able to include the background job responsible for doing this automatically.

We'll ship this task in the March v2.0 release.

Limitation: Manage permissions

Currently, only the owner is able to see a restricted record/files. In future releases, we'll of course add the support for managing permissions, so that restricted records and files can be shared with colleagues, groups and entire institutions.

Access backend

Underneath the hood, we have prepared the entire access control system to be able to share access with other users, communities, IP networks, etc. Thus, the backend part for allowing management of permissions is almost done, while we are primarily missing the frontend that allows an end-user to interact with the permission system.

Minimal UX

Disabled unfinished features

We have disabled all unfinished features from the user interface. We have for instance disabled the communities module, the version flip-switch in search results, a button to the advanced search and so on.

The features will be added back, once they have reached sufficient quality to be running in a demo system.

Deposit form fields

We have also removed all deposit form fields from the user interface that were not ready. Each of these fields will be added back progressively as they are finished. Most notably we e.g. had to remove the subjects field for this release.

Field help text

We have added help text to as many fields as possible. Please provide feedback, if they are understandable.

Styling issues

We have fixed a large number of small styling and consistentcy issues. Still some remain, and you're feedback is very much appreciated on anything you find in this respect.

List uploads

The list uploads page now has a nice greeting for first time users:

Also, the overall styling of the page was improved:

Search guide

We have added a guide (taken from Zenodo) on how to perform advanced queries. See it on https://inveniordm.web.cern.ch/help/search.

Release v1.0

We have gone through a lot of quality checking to ensure that we'll be able to upgrade data created with the v1.0 release to the coming v2.0 release. This work has focused on ensuring, for example, proper validation and integrity of the data that makes it into the database.

We have also ensured that all database tables have proper upgrade recipes for our automatic schema upgrade system (Alembic).

Known issues

Landing page

Preview: The image previewer is not using IIIF and only previews images up to 500kb.
Preview: If you e.g. upload a .JPG extension file, you can set the default preview for the file, but the list of files will not have a preview button.
Preview: The caption of the preview box does not change when selecting another file for preview.

Deposit form

Validation error messages: The form validation error messages are not displaying correctly. Labels are system names, and sometimes [Object (Object)] is displayed for errors.
Missing files: If files are missing to be uploaded, the publish button is disabled, but there no feedback to the end-user that files are missing.
Performance: We have seen some slight performance issues when entering information into the form. We're looking into this, to make sure it is not noticeable.
Affiliations vocabulary: We have not initialized the affiliation vocabulary, and thus there is no auto-completion for affiliations (creators and contributors).
Publish button: It is not enabled when reloading a page with a valid record - you first have to save the record for the publish button to be enabled.
Protection: The files restriction in the protection widget only become visible after uploading a file. It should be hidden only for metadata-only records.
Languages: The language vocabulary search results are not ranked properly for the query. For instance searching for "english" will show less relevant results first.
Creators: The list of creators in the deposit form does not display the role nor identifier if entered.

Search results

HTML entities (e.g.  ) in the description fields are not removed from the descriptions displayed in search results, causing them to be printed verbatim.

List uploads

Facet: The search facet distinguishing between published and drafts on the list uploads page is using system labels instead of human readable labels.
The title of a search result item is not clickable even though it looks clickable.

Backend

Embargo records are not automatically published due to the missing background job.
The REST API JSON serialization has timestamps that are not in UTC, nor do they have timezone information.

Installation

Some versions of NPM 7 are known to cause issues -- NPM 6.14.5 is recommended.

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli check-requirements
invenio-cli init rdm
cd my-site
invenio-cli containers start --lock --build --setup

This month's version of invenio-cli is v0.22.0.

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli containers stop

To destroy the Python virtualenv, and remove the docker containers, you can run:

cd my-site
invenio-cli destroy

Feedback

First, check if you've hit a known issue.

Bugs: First check for known issues above, then report them on the Invenio-App-RDM repository on GitHub.
Feedback: Suggestions for improvements, results of user testing, etc. - please reach out to Lars directly (chat or mail).

Credits

The development work in this release was done by:

CERN (Lars & Zach)
Northwestern University (Guillaume)
TU Graz (Mojib)
TU Wien (Max)

A special thanks goes to Max and Mojib as well as their institutions TU Graz and TU Wien for joining full-time during the February iteration. This made it possible to reach this month's milestone for the regular team from CERN & Northwestern University!

Take care and stay safe!

InvenioRDM Onboarding Resource Roundup

2021-02-10T00:00:00Z

Congratulations! Your team has just joined the InvenioRDM open source project and is eager to contribute as much as possible. With a mixture of dedicated developers, resource curators, subject matter experts, and various leadership stakeholders, your team simply needs to know about the available resources and how to get started. Read on!

Developers

Whether you would like to install and customize InvenioRDM, contribute on the main project sprints, or take on development of a module locally to contribute back to the project, there are a few important points of entry to participation:

Roadmap: Familiarize yourself with the overall project timeline and goals.
Development Roadmap: Get to know the specific milestones
Sprintboard: See the planning board for the current sprint
Documentation: Read the official project documentation
Discord: Ask questions in the project Discord (chat/video platform) and introduce yourself in either the general or newcomers-help channels.
Discourse: Want to provide feedback, but in a less time-sensitive manner than a chat? Either join or start a thread in the Discourse discussion forum
Local Instance Setup: Install a local InvenioRDM instance for development
Contribute code: Learn about the development process and how to contribute code to the project

Community Members

All stakeholders at your institution are welcome to contribute to InvenioRDM, whether their contribution is code-based or not. To create a tool to best serve the community, outlining stakeholder requirements will be vital:

Demo site: Access the latest sandbox version of InvenioRDM here. Create an account to test record creation
Discourse: Provide feedback on testing here. Use the Feedback category or start a new thread wherever you like
Metadata Interest Group: This is the first community-focused group organized to help support the InvenioRDM development effort. This group meets about once/month and discusses issues related to the InvenioRDM Metadata Model. To join, either make a post on the Discourse thread or in this dedicated Discord channel
Conference Calendar: Are you presenting on InvenioRDM at an upcoming conference? Make a note here so other interested members of the community can attend the session, or perhaps join your poster or presentation effort
Welcome, Help and General Questions: Reach out to Community Manager Sara Gonzales via Discord for general questions or to join the Metadata Interest Group
Schedule a Demo: Is your site, or a site you know, considering implementing InvenioRDM? Reach out to Community Manager Sara Gonzales via Discord to schedule a one-hour demo and information session about the repository

Everyone

Project list-serv: Lars will have shared this email address with you upon joining the team. This is the frontline communication tool for the InvenioRDM community. Make this email address a safe sender to receive the latest updates on teleconferences (a.k.a. telecons) and community-based information such as invitations to the Metadata Interest Group meetings. Polls, forms, and additional community tools will be sent to this list-serv
Telecons (see the list-serv for connection information): All partners of the InvenioRDM community are invited to bi-weekly telecons throughout the year. One telecon per month will focus on development updates, and the other will focus on community-based topics
InvenioRDM Telecon Schedule
Blog: You are reading it! The blog will also alternate between development-based and community-based topics. Would you like to contribute a blog post? Sign up here or contact Sara to get on the schedule
Code of Conduct
Governance

Welcome to the team, and happy collaborating!

InvenioRDM Alpha 15 (January Release)

2021-02-01T00:00:00Z

2020 didn't stop us from shipping new releases. 2021 won't either. We’re excited to announce the InvenioRDM Alpha 15 (January release)! Thank you to our team members for their efforts on this release.

What’s new?

This month was primarily dedicated to improving the user experience of the deposit form.

We have made significant improvements to the overall user experience of the deposit form. That said, we still have many smaller UX improvements to make. This work will continue throughout February.

Creators/Contributors

The creators/contributors fields have gotten a complete makeover that significantly reduces the clutter. We have optimized the new creators field for adding basic information (name/affiliation) as well as linking the affiliation field to vocabulary records (e.g. ROR). We postponed the more advanced use case of allowing the user to edit all fields such as identifier scheme and identifiers for affiliations, as this created an overly complex form for the majority of use cases.

The creator field we imagine can be extended in the future with support for importing authors via predefined files (e.g. BibTex, etc.).

One particular feature to highlight is that the name identifiers field now does automatic detection of the identifier scheme. You can now paste an ORCID (e.g. an ORCID URL), and the backend will automatically detect it as an ORCID and normalise the stored value.

Coming Soon

We have decided to hide certain fields that we know need further work. You'll see these as "Coming soon". Also, we have temporarily removed the language field from titles and descriptions as we had last minute issues with these fields.

Minor UX Improvements

Sticky side bar: The side bar with the save/publish buttons, is now sticky so it will follow you when you scroll down the form.
Resource type: The drop-down was reduced from two drop-downs to a single drop-down to reduce the number of needed clicks to select a resource type.
Delete/discard button: There's now a delete button that you can use to delete your draft or discard edits to a published record.
Button consistency: We have applied consistent styling, wording and and naming to most buttons.
Errors for inline fields are now consistently displayed.
Brand color: We have now made it easier to customize the styling of the field groups via a common brand color. Previously, if your header was purple, the field groups would remain blue.
File warning: The warning displayed under the files section was updated to match Semantic UI styling.
Field help text: We've added an example of field help text to the Dates field, which will expand to all fields needing it.
Array fields styling: dates, related identifiers and titles are now properly using the entire form grid from side to side.

Consistent naming of related work.
Title: The deposit form now has a title.

Other Changes

Vocabularies: The vocabularies REST API received some significant backend changes, which means that we can now properly show translated text in auto-completion fields such as languages, license, affiliations, and subjects.
Extended Date Time Format indexing: We now have support for searching EDTF date ranges properly (contributed by TU Wien).
Geospatial indexing: We are now able to properly index almost any GeoJSON data type (points, boxes, polygons, etc.) so that they can be searched properly at a later point. They are indexed via their centroid which is computed using the Shapely library (contributed by Cottage Labs).
Core API methods needed for reindexing of vocabularies was added as well (contributed by EU JRC).
Prettyprint: If you explore the REST API in a browser, you can now add ?prettyprint=1 to the URL, and the JSON will be nicely formatted for you.

Known Issues

Discarding a change to an edited record doesn't work.
Licenses cannot be saved.
Languages cannot be saved and only Danish and English are currently available.
License data is missing.
Error messages at the top of the form are not displayed correctly.

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

This month's version of invenio-cli is 0.21.0 .

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli stop

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

InvenioRDM Alpha 14 (December Release)

2021-01-11T00:00:00Z

We’re excited to announce the InvenioRDM Alpha 14 (December release)! Thank you to our team members for their efforts on this release.

What’s new?

Vocabularies support

The focus of the December release was to add support for linking bibliographic records with authority records such as subjects, grants, licenses, languages and more.

This includes:

Auto-complete on deposit fields
Facet searches based on vocabulary values
Support for internationalization on deposit, on facets, etc.
REST APIs for vocabularies
Linking of records in the backend.

Deposit form autocompletion

The most visible way to see the vocabularies in action is via the deposit form fields.

The below components were added in the deposit form and they support vocabulary autocompletion:

Languages

You can see the autocomplete search help:

And an example of i18n support, when the instance is in Turkish:

Licenses (preview)

We have added support for both standard and custom licenses. Note that we were not able to fully complete this work, and there are known bugs in the license widget. Most notably you'll see:

Search field does not allow spaces
Link not working
UX issues such as over-long page
Descriptions for licenses not available.

Subjects (preview)

The subjects field autocompletes from a subjects vocabulary that can host multiple different vocabularies at the same time.

The current version is not yet connected with the API, thus it has some demonstration values instead. Only the values you see in the image below can be selected. Don't save a record with subjects yet either.

Invenio v3.4

Thanks to the hard work of the December Invenio sprint, Invenio v3.4 was released (see more here). The release of Invenio v3.4 was an important step for InvenioRDM. We now depend on stable releases of Semantic UI.

Partial save and error reporting

This release includes support for partial validation on draft creation. It also displays partial saves in the uploads page, for example with "No title" if the title was not included in the draft.

For example, saving an empty draft will tell you precisely what is missing at the top, along with pointers per field (see title at the bottom of the image).

The former error space will update to good (green) once you save a draft that is publication-ready. The publish button will then be enabled.

Install

$ invenio-cli init rdm
$ cd my-site
$ invenio-cli packages lock --pre
$ invenio-cli install --pre
$ invenio-cli services setup
$ pipenv run invenio vocabularies import languages licenses
$ invenio-cli run

Feedback

Here is the template to give feedback if it’s not automated:

## Bugs

(note, that there's known issues in this release for license and subjects fields)

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

Invenio v3.4.0 released

2020-12-17T00:00:00Z

We are proud to announce the release of Invenio v3.4.0.

Head over to our Getting started to see it in action.

Read the full release notes.

What's new in Invenio v3.4?

New UI Framework: Semantic UI

The largest new feature in Invenio v3.4 is the release of Semantic UI as the new UI framework. This works has been under way since we released version 3.0 with a deprecation notice on AngularJS 2.5 years ago.

Semantic UI is now the new default UI framework for Invenio, replacing the existing Bootstrap v3 framework. Most importantly, we have added support in Invenio v3.4 for supporting multiple UI frameworks concurrently. This means that Bootstrap 3 is still supported in v3.4 together with Semantic UI, and that we in the future can support new frameworks.

UI Frameworks evolve very rapidly and we expect that Invenio eventually will have to go through multiple UI frameworks over it’s life-time. Allowing support for multiple frameworks, isolate future changes, and even allow third-parties to develop custom themes for Invenio without impacting the business logic Invenio.

Technically the support for multiple UI frameworks works by having specialised Jinja template loading and themable webassets build systems.

Read more about the new system in User interface styling.

Reorganised documentation

We have reorganised the Invenio documentation, to make it easier for new developers to get started with Invenio.

In particular we now have a documentation overview page on https://inveniosoftware.org/documentation/ that gathers both Framework, RDM and ILS documentation under one umbrella.

We have also reorganised the main Invenio Framework documentation on https://invenio.readthedocs.io/en/latest/

Records improvements

Invenio v3.4 marks the start of some larger improvements we want to make to the overall data flow for records in Invenio. With Invenio v3.4 we are releasing new features for Invenio-Records. These improvements are not being used by other Invenio modules yet, but allows us to ensure the backward compatibility of the changes.

Overall, the changes made to Invenio-Records focuses on having records stored in separate database tables/Elasticsearch indexes and making a simplified programmatic API that is less error prone to use.

The following only describes the new featuers at a high-level.

Dumping/Loading support

Records now supports dumping and loading of records. This will eventually replace the using signals for customising the JSON being indexed, as well as standardising the record being loaded from Elasticsearch, database or other sources.

Type support

Currently, a record only supports native JSON data types. In particular, date/time data types are not supported. We have now added the possiblity to support that certain fields inside your JSON have custom data types (such as datetime).

Extensions

We have added support record extensions as a mechanism for replacing signals. Overall the signals were causing issues, because you often needed signals per type of record you were dealing with. With the extensions, you can now more easily hook into

System fields

A typical issue you would deal with in Invenio is to create a record and a persistent identifier. System fields allows you to easily compose a record type with new features, so that e.g. Record.create(...) will also automatically create a persistent identifer.

Backward compatibility

All changes are backward compatible thus existing source code and stored records will continue to work with the latest release.

Testing improvements

GitHub Actions instead of Travis CI

We have moved almost all our 100+ repositories from using Travis CI as the continues integration system to GitHub Actions. This was due to Travis CI imposing a migration from .org to .com and lowering the total number of resources availble for open source projects. Overall, this meant very long waiting times for pull-request builds as well as PyPI releases. We have therefore migration to GitHub Actions. We would like to thank Travis CI for the service they have provided through out the years to the open source community.

Reprodicible tests

We have developed a new tool Docker-Services-CLI, that helps bring up/down required services such as databases, Elasticsearch, etc for testing. This makes it easier to investigate specific test failures locally, as well as making it easier to migrate between continuous integration systems.

Centralised test dependencies and Pytest-Invenio

With the release of Pytest-Invenio v1.4 we have centralised test dependencies to a single module, and can more easily control breaking changes from third-party packages. Previously new version of isort and pytest have been causing wide-spread test failures due to minor changes.

Minor changes in v3.4

Invenio-Accounts:
- Password length validation during login was removed.
- "Log in" instead of "Sign in" is now being used consistently throughout the application. Previously both version could be found.
Invenio-App:
- The /ping endpoint (used by e.g. load balancers for aliveness checks) now also supports HEAD and OPTIONS HTTP verbs as recommended by e.g. HAProxy documentation.
Invenio-Celery
- Only Celery to 4.4.x to 5.0.x releases are supported. Note, Celery have declared Celery 4.4.x as a Long Term Support release. Note, that Celery changed command line arguments from celery worker -A <app> to celery -A <app> worker between version 4 and 5.
Invenio-DB
- Now support SQLAlchemy <1.4 which include e.g. PostgreSQL 12 support.
- Hides the database password from CLI output when running db init or db create.
Invenio-Indexer
- Elasticsearch delete requests now uses optimistic concurrency control by providing the the version and version_type in delete requests. The previous behavior can restored by calling RecordIndexer().delete(record, version=None, version_type=None) instead. This fixes an issue where deleting and recreating a document right after would fail.
Invenio-JSONSchemas
- Fixes an issue related to nested allOf being ignored.
Invenio-OAuthClient
- Added support for CERN OpenID contrib.
Invenio-Records-REST
- The header Cache-Control: 'no-cache' is now sent on successful HTTP 200 responses to ensure that browsers will not cache responses client side and ETag will be evaluated.
Invenio-REST
- Fixes a bug with CSRF checking when the endpoint did not exist.

Deprecations in v3.4

Bootstrap 3

Bootstrap v3 has reached end-of-life and will no longer receive further updates from their maintainers. Migrating to Bootstrap v4 or v5 is as difficult as upgrading to Semantic UI and there are no easy migration paths.

We will continue to support the Bootstrap v3 integration throughout the maintenance period of Invenio v3.4. Invenio v3.5 may move all Bootstrap 3 templates to a separate Invenio package, that you could install, however v3.5 will not add new Bootstrap 3 templates.

You should plan allocating time for a migration from Bootstrap 3 to Semantic UI during 2021

Features removed in v3.4

Elasticsearch v2 and v5 support

We have removed support for Elasticsearch v2 and v5 as announced in v3.3. Elasticsearch v2 and v5 have reached end-of-life and is no longer maintained with security fixes, thus you should strongly consider upgrading in case you have not already done so.

AMD/RequireJS

The old AMD/RequireJS assets build system, that was deprecated and replaced with Webpack in Invenio v3.1 has now been completely removed from the code base.

Maintenance policy

Invenio v3.4 will be supported with bug and security fixes until the release of Invenio v3.6 and minimum until 2021-12-17.

See our Maintenance Policy.

Metadata: Schema Reference Document and Future Work

2020-12-15T00:00:00Z

On 17 and 24 November 2020 the InvenioRDM project partners met in two telecons to discuss the Metadata Reference document.

These conversations were a follow-up to the community's review of the document. Feedback and questions were collected in a Google Doc prior to the conversations. The conversations helped to confirm many answers to questions about the data model that had been posed in the Doc, such as: whether ‘role’ is required for Creators (no - only for Contributors), are Resource Types customizable (yes), and which file is shown in the previewer when a record has multiple files (the first, unless otherwise specified). See the Doc for responses to additional questions.

Use Cases Requested

For some fields, further feedback is needed in order to make them as useful for the community as possible. In particular, your use cases are requested for the following scenarios:

Adding a Language tag to Titles
Records with more than one Publisher (making Publisher a repeatable field)
Having a built-in “Unknown” option for Contributor type and/or role
Having more than one Identifier per Affiliation

Please post your use cases back to the top of the original Google Doc or to the new Metadata Interest Group topic at: https://invenio-talk.web.cern.ch (more on this below).

Subjects

In addition, the topic of Subjects was brought up for further discussion. The Subjects example in the Metadata Reference document shows the incorporation of one controlled vocabulary, the Medical Subject Headings, into the schema.

However, there are many controlled vocabularies that we can potentially map to the Subjects field. Which will be most useful to incorporate for the wider community, and what are potential use cases?

This is one of the first issues we would like to discuss in the Metadata Interest Group, which is set to launch in January 2021. This group welcomes the contributions and insights of developers, users, administrators, decision-makers, and anyone else at participating organizations with metadata experience, interest, and insights. Through the end of 2020 we will work to publicize the group and recruit members. If you’re interested in joining, please either sign up here: https://invenio-talk.web.cern.ch/t/metadata-interest-group/144 or DM me on Discord (see the Welcome channel).

Community Outreach

We are committed to serving the entire InvenioRDM community as our family of adopters continues to grow and thrive. This is the first, we hope, of many blog posts that will address topics of interest to the community at large, including developers, administrators, curators, and decision-makers. If there is a topic you would like us to explore, or perhaps even contribute a post on yourself, we’d like to know. If the topic would be best covered at a community-focused telecon (also on the horizon for the coming months!), we’d like to know that as well. Please contribute any and all ideas at my contact listed above.

Take care and stay safe!

InvenioRDM November Release

2020-11-29T00:00:00Z

We’re excited to announce the InvenioRDM Alpha 13 (November release)!

What’s new?

Files support

It is now possible to upload files on the draft form and publish them.

File uploader

When creating a new draft, you can now also upload files that will be included in the published record. The file uploader supports:

Drag-n-dropping files to be uploaded
Concurrent uploads (by default up to 4)
Setting the default preview file
Completely disabling files to allow for metadata-only records

New REST API

Big part of our efforts went into designing a new files REST API that will allow InvenioRDM to easily support upload workflows depending on third-party storage systems (e.g S3).

You can check out the documentation of these endpoints in our newly added REST API reference.

Files listing and preview

Uploaded files are also listed and can be previewed on the record's landing page, using a variety of extensible previewers. We currently support preview of PDFs, images, ZIP files, CSV tables, and some text-based formats like XML, JSON, and Markdown.

By default, files are listed in alphabetical order and the first previewable file is shown in the preview box. You can specifically set which file you want to be by default previewed in the record's draft form.

Reorganized CLI commands

Thanks to the community's feedback, in this release we have also updated the existing CLI commands, with the goal of improving the development and operational workflows of an InvenioRDM instance. Our newly added CLI reference documentation lists and describes in detail the command groups, their sub-commands, and the options they support.

Make sure to go through our documentation, since many old commands have now new options and different default behavior (e.g. service setup will now also create demo records).

Feedback

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

InvenioRDM October Release

2020-11-11T00:00:00Z

We're happy to announce the InvenioRDM October release. Thank you to our team members for their efforts on this release.

In early October we held a virtual project workshop. As part of the workshop we have updated the roadmap. The roadmap is now publicly available in two versions, one simplified and one for project tracking. See below:

1- Simplified

2- GitHub for project tracking

What's new?

Metadata Schema

The entire metadata schema for bibliographic records has been updated, and is now ready for a thorough review by all project partners. The update includes changes to the JSONSchema, Elasticsearch mappings, the REST API data validation layer as well as addition of many fields to the deposit form.

See (https://inveniordm.docs.cern.ch/reference/metadata/) for a full reference of the new metadata schema.

User Experience

We have done a lot of work on improving the UX of the primary pages such as the frontpage, search results and record landing page. Below you can see some screenshots of before/after:

Search results

The search results have been tightened up as well. Also, we’ve added support for nested facets, so you can expand broader categories into subcategories (e.g., Publication and Image).

Landing page

The landing page you’ll notice now has Edit and New version buttons (only the Edit button currently works). The right column has been tightened up, and most of the new metadata fields are now properly displayed on the landing page.

Human readable labels

In the facets you’ll notice we now have human readable labels. Instead of 'publication' it will say 'Publication'; instead of 'open' it will say 'Open Access'.

Similarly to search results and landing pages, you’ll now see the correct resource type as well as icons on Open Access.

Localization

Dates, like the publication date which supports Extended Date Time Format is now properly localized as well using the Unicode Common Locale Data Repository. For instance here an English and Turkish localization:

Affiliations

The affiliation display was redone:

Manage section

There’s a new manage section on the record landing page, which allows you to edit a record.

CLI Improvements

We have made it easier to get started with developing InvenioRDM. Partially to make our own developers' life easier, but also to make it easier to customize InvenioRDM.

Shells

You can now easily activate the Python virtualenv shell as well start a Python terminal from your instance using the following commands:

invenio-cli shell
invenio-cli pyshell

Watching assets

We have now simplified how you can change styling via automatic watching of file changes on assets. Previously you had to manually rebuild the assets. Now instead you can simply execute the following commands, and the styling will automatically rebuild once the file changes:

invenio-cli assets --force --development
invenio-cli assets watch

Develop an Invenio module

Developers often need to install the latest development versions of Invenio modules to work on them. This can now easily be done with:

invenio-cli ext module-install ~/src/invenio-app-rdm ~/src/invenio-rdm-records
invenio-cli assets --force --development

Develop a React module

The above works only for Python packages. If instead, you are working on one of our React libraries, you can now easily install and watch the module for changes as well:

invenio-cli assets watch-module --link ~/src/react-invenio-deposit

Install

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To stop the instance without destroying the records that were created, you can just run:

cd my-site
invenio-cli stop

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

Feedback

Here is the template to give feedback if it’s not automated:


## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

InvenioRDM September Release

2020-10-12T00:00:00Z

We are glad to announce InvenioRDM Alpha 11 (September release)!

What's new?

We now have support for drafts, meaning you can now start a deposit, save it, and come back many days later to publish it.

This simple change marks a very big milestone for the backend of InvenioRDM. During the past months we have been revamping the entire data and control flow of the backend. The flow is what is supporting the core of InvenioRDM - submitting, editing and searching for records - and it determines what is possible and what's not possible in InvenioRDM. These changes enable the customizations needed by partners including customizations to permissions, publishing workflows, metadata fields, indexing and serializations just to name a few.

A high-level overview of the new flow is shown below:

This new flow is implemented in the following Invenio modules:

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers, you can now just run:

cd my-site
invenio-cli destroy

It's the first feature by Rodrigo Almeida, the newest addition to our team!

InvenioRDM August Release

2020-09-01T00:00:00Z

We are happy to announce InvenioRDM Alpha 10 (August release)! Thank you to our team members for their efforts on this release.

What's new?

We made some limited changes this month. However, three key areas were tackled.

The concurrent Invenio Sprint brought the final Semantic-UI updates across all modules! If something does not show up right in your default RDM instance, then it's a legitimate bug now.
These UI updates complement the updates to Search. The record search page now uses the new API endpoint from the last release. Customization of the search results is back (with documentation forthcoming). Different search pages are easier to set up.
More library improvements across the board were merged.

Update invenio-cli to version 0.16.0 and follow the updated documentation to get started.

Semantic UI transition wrap-up

One of the many pages that were transitioned to Semantic-UI

The transition to Semantic-UI is for all intents and purposes complete. Future additions will use this framework and have a consistent look with the rest of InvenioRDM. Close to 15 styling issues were closed in the process.

Search customization and transition to new API

The search page now uses the new API (/api/rdm-records endpoint) we introduced in the last release. The new endpoint supports pagination and sorting, but aggregations were disabled to focus on the transitioning only.

Aggregations will be re-enabled soon.

Custom styling of search results is possible again: documentation to explain the new way is pending however.

Library Improvements

Finally, various improvements and fixes were made across modules including:

an overhaul to how links are generated in API responses,
a way to hook callbacks into action endpoints,
the convergence of validation for drafts and records, and
the introduction of record versioning behind the scenes (which will be made more visible in coming releases).

Some of these changes are visible in the API responses:

{
    "links": {
        "self": "https://127.0.0.1/api/rdm-records/jnmmp-51n47",
        "self_html": "https://127.0.0.1/records/jnmmp-51n47",
        "files": "https://127.0.0.1/api/rdm-records/jnmmp-51n47/files",
        "edit": "https://127.0.0.1/api/rdm-records/jnmmp-51n47/draft"
    },
    "metadata": {
         "conceptrecid": "5fk5g-mq814",
         ...
    },
    ...
}

What do you need to do?

You can install the latest invenio-cli v0.16.0, create a new instance and see what it looks like. Make sure to follow the updated documentation - there are a couple of caveats to be made aware of.

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. When you provide feedback on Discourse your message should be pre-populated with the classic template (bugs, what worked well, what didn't work well, wishes for documentation).

Here is the template to give feedback if it's not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe! The next release will be big.

InvenioRDM July Release

2020-08-04T00:00:00Z

We are happy to announce InvenioRDM Alpha 9 (July release)! Thank you to our team members for their efforts on this release.

What's new?

The July release adds translation support and starts integrating the major backend development of last release into InvenioRDM. In particular, draft functionality has been added at the API level and is used under the hood for deposits. To achieve this integration, the hard problems like responsibility separation, error handling, pagination, linking, and internal PID management were solved. Other improvements and more details follow.

Update invenio-cli to version 0.15.0 and follow the documentation to get started.

Module translation

Thanks to the efforts of TÜBITAK, InvenioRDM was set up for internationalization: all text has been registered for translation and the Transifex service has been enabled. The team is currently at work on a VSCode extension to help future translators. Turkish was added as a first alternative language! How to go about translating and using the various tools will be laid out next month once we have more experience under our belt!

Library improvements

Error handling, default MIME type headers, and uniform body + querystring deserialization have been added to Flask-Resources, our Invenio-agnostic Flask REST library.

Note that invenio-resources is now invenio-records-resources (it includes the former invenio-records-agent) and invenio-drafts-resources is the new package that houses the draft functionality.

Behind the scenes, persistent identifiers have also seen some work.

New API integration (including draft functionality)

The deposit page now uses the new API to create records by creating drafts and publishing them immediately. In the next release, we hope you will be able to save and publish records separately. That being said, you can already do so on the command-line! Make sure to follow the updated documentation to see creation, publication, retrieval, and search in action there.

In addition to these tent-pole features, Python 3.7 support and general containerization was further improved thanks to Cottage Labs through an updated base image.

Community domains vocabulary

The list of available domains/disciplines that can be added to a community is now based on a pre-defined (and configurable) vocabulary.

What do you need to do?

Follow the documentation site: https://inveniordm.docs.cern.ch/ and install the latest invenio-cli v0.15.0, create a new instance and see what it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

Here is the template to give feedback if it's not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

InvenioRDM June Release

2020-07-01T00:00:00Z

We are happy to announce InvenioRDM Alpha 8 (June release). Thank you to our team members for their efforts on this release.

What's new?

The June release is a major backend development release. We developed three new modules that lay the foundation for the new core data flow of Invenio(RDM). Our work goes far to provide a better developer experience. You will be able to work on and customize Invenio(RDM) in an easier and cleaner way.

The primary focus for the release was:

Continue the migration from Bootstrap to SemanticUI
Bug fixes and improvements on the frontend (mainly on deposit)
Develop new core modules for Invenio(RDM) backend

UI Customization

You are able to customize your templates again. The JSX customizations are not functional yet though, which means the search page results are not customizable.

Visual appearance

We have migrated the file previewer, the OAuth login, and the settings pages to SemanticUI.

You might still notice some issues related to visual appearance (e.g. the name previewed file is not in a panel as before). We will be working on making all this perfect in the next sprint on SemanticUI (end of August). In the meantime, it is, at least, better.

Creators, Contributors and Affiliations Identifiers

The Deposit page now accepts ORCID iDs for people (creators and contributors) and ROR identifiers for organizations (creators, contributors and affiliations). An icon with a link to the respective entity on the record landing page reflects this connection.

Contributors are now optional on the deposit page as well.

Python versions

There are now Invenio images for Python 3.7 and Python 3.8! This means that you can use Python 3.6, 3.7 and 3.8 for local development and the full containerized mode.

Backend and REST APIs

We have put a significant amount of work in refactoring some of the core API flow of Invenio. This addresses many of the issues that Invenio was facing in this domain. As a result, three modules were created:

https://github.com/inveniosoftware/flask-resources: A generic Flask module to implement REST APIs and make their customization easy.
https://github.com/inveniosoftware/invenio-resources: A module customizing Flask-Resources for Invenio(RDM) APIs.
https://github.com/ppanero/invenio-records-agent: An experimental module to bridge the representational and data access worlds in Invenio. This module hosts the business logic and might be merged back into invenio-resources.

What do you need to do?

Follow the documentation site: https://inveniordm.docs.cern.ch/ and install the latest invenio-cli v0.14.2+, create a new instance and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. We are experimenting with Discourse's built in post template. This means when you provide feedback on Discourse your message should be pre-populated with the classic template:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Thank you for your patience and support. We are working hard all the time to improve things. Take care and stay safe!

InvenioRDM May Release

2020-06-05T00:00:00Z

We are happy to announce InvenioRDM Alpha 7 (May release). Thank you to our team members for their efforts on this significant release.

What's new?

The May release is a major integration release, with a few rough edges that still need polishing. We have integrated a lot of code that was developed in separate branches into the main InvenioRDM code base. We have done this now, to ensure we have enough time to iron out integration issues.

The primary focus for the release was:

Migrate from Bootstrap to SemanticUI
Deposit form frontend (JavaScript)
Integrate new communities prototype.
Upgrade to Invenio v3.3

Known issues (please read!)

As a rule of thumb, many features will be broken and not work. Thank you for your continued patience and feedback as we work to make improvements.

A lot of code has changed in this release, and many features have not yet undergone quality control.

UI Customizations not working

The existing method for customizing the UI will change slightly, We hope to provide documentation for this in the next release.

Visual appearance

You'll notice issues related to the visual appearance, that we have not yet had the time to refine. This is with regards to alignment, colors, behaviours, font-sizes, and user experience.

Data model and deposit form

The data model and deposit form is far from complete. We have not yet focused on implementing the right data model with the right fields. For instance the access right displayed in the deposit form will change, as will many of the other fields displayed.

Communities

Communities is in a very initial state and many things will change.

Previewer, OAuth login, Settings pages

The file previewer and OAuth login still need to be migrated to SemanticUI, and thus the modules are likely not to work properly in this release.

What do you need to do?

You can install the latest invenio-cli v0.14.x, create a new instance from the beginning and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image:

docker pull inveniosoftware/centos7-python:3.6

To install:

pip install invenio-cli --upgrade
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Communities

To see the communities click "Communities" menu, then "New community". You'll be prompted to log in. Simply create a new account, and log in with it.

Deposit form

To see the deposit click "Uploads" then "New upload"

Want to get involved?

Follow the documentation site: https://inveniordm.docs.cern.ch/ for an up-to-date step-by-step install and usage of InvenioRDM.

Give us feedback on Discourse: https://invenio-talk.web.cern.ch/t/inveniordm-alpha-7-may-release/109 for this release.

Thank you for your patience. We are working hard all the time to improve things. Take care and stay safe!

Invenio v3.3 released

2020-05-20T00:00:00Z

We are proud to announce the release of Invenio v3.3.0. With this release, Invenio v3.1.x reaches end of life, and will no longer be maintained.

Python compatibility

Invenio v3.3 supports Python versions 3.6 and 3.7. Python 2 support in Invenio ended on January 1st, 2020 with the official end of life for Python 2 on the same date.

Getting started

See our quick start guide.

Upgrading

See our upgrade guide.

Release notes

Please see the full release notes for details about minor changes, deprecations etc.

Highlights for Invenio v3.3

Python 3.7

Python 3.7 is now supported by Invenio!

Improved support for Single Page Applications

Invenio v3.3 improves the support for Single Page Applications (SPA) by adding REST APIs for account management operations such as login, logout, user registration, password change, email confirmation and more. The integrated OAuth client also adds a new REST API so that you can login via your OAuth providers such as GitHub, ORCID and Globus.

CSRF Protection

Invenio-REST adds a CSRF middleware, called CSRFProtectMiddlewate to protect API views against CSRF attacks. The CSRF checks can be skipped in REST API calls when using a personal OAuth API token.

Dependency management

We have revamped management of third-party dependencies. New releases of some of the many third-party packages Invenio depends on could often cause dependency conflicts. Invenio v3.3 introduces the concept of coordinator packages, who are responsible for properly specifying third-party dependencies.

The release notes contain a table with all the current coordinator modules.

Helm chart (BETA)

Helm-Invenio provides a helm chart to deploy an Invenio application along with all its required services: Redis, RabbitMQ, HAProxy, Nginx, Elasticsearch, Logstash and PostgreSQL.

This chart is currently in beta version. Its templates are currently specific for OpenShift. However, work is on-going to support bare Kubernetes deployments.

Maintenance policy

Invenio v3.3 will be supported with bug and security fixes until the release of Invenio v3.5 and minimum until 2021-05-18.

See our maintenance policy.

What's next?

In Invenio v3.4 we are planning to release two major new features:

Theming support and a new Semantic UI theme.
A statistics bundle that adds support for collecting COUNTER Research Data Usage Metrics compliant statistics.
Index migration support for migrating between Elasticsearch clusters (aka zero down-time reindexing and index migration).

InvenioRDM April Release

2020-05-05T00:00:00Z

We are pleased to announce the InvenioRDM April release. The Deposit page has been our focus and that meant designing the frontend and the backend based on lessons learned from the beta invenio-deposit python module and from the invenio-records-js and invenio-files-js previous angularjs modules. We will be implementing most of these designs in May.

Thank you all for testing the last release and for your many contributions.

Some Highlights include:

Requests for comments (RFCs) from the numerous conversations. These are living and changing documents.
- Deposit frontend: https://github.com/inveniosoftware/rfcs/issues/27
- Deposit backend: https://github.com/inveniosoftware/rfcs/issues/22
Two new modules have been created to improve the REST API and deal with the deposit backend
- https://github.com/inveniosoftware/flask-resources
- https://github.com/inveniosoftware/invenio-resources
On the frontend we have added a proof of concept deposit page with 2 fields (title and resource type).
These are in part powered by an improved controlled vocabulary backend and an all-new React frontend.
This page is accessible through the Upload link of the search page.
Invenio-S3 v1.0.3 has been released with better big file upload support
Various dependency fixes have been applied and the cookiecutter slugification made more solid
Auto-reloading of webpack has been made possible allowing us to work much more effectively with React

Want to get involved?

Although in this release, we haven’t made significant visual changes, you are always welcomed to try new releases - invenio-cli v0.12.X, this time around- by following the documentation site: https://inveniordm.docs.cern.ch/ . Give us feedback on Discourse: https://invenio-talk.web.cern.ch/t/inveniordm-alpha-6-april-release/104 for this release.

As mentioned, apart from the /deposits/new proof of concept page you won’t see much other differences. Take it easy this time. We will be asking a lot of feedback from you next month!

Thank you for your interest, take care and stay safe!

Invenio v3.2 released

2019-12-20T00:00:00Z

We are proud to announce the release of Invenio v3.2.0. With this release, Invenio v3.0.x reaches end of life, and will no longer be maintained.

Python compatibility

Invenio v3.2 supports Python 2.7 (until 2019-12-31), Python 3.5 and Python 3.6.

Getting started

See our quick start guide.

Upgrading

See our upgrade guide.

Release notes

Please see the full release notes for details about minor changes, deprecations etc.

Highlights for Invenio v3.2

Files bundle

We have released four new modules as part of the new Files bundle:

Invenio-Files-REST: Object store REST API for managing files in Invenio.
Invenio-Records-Files: Integration layer between records and files.
Invenio-Previewer: File previewers support.
Invenio-IIIF: IIIF Image API support for thumbnail and zooming on images.

To understand more about how to use the new Files bundle see our integration guide on https://invenio.readthedocs.io/en/latest/tutorials/handling-files.html.

Elasticsearch v7 support

Invenio now supports Elasticsearch v7.

Marshmallow 3 compatibility

Invenio now support both Marshmallow v2 or v3. The support is done via a compatibility layer, that allows Invenio to work with either schemas from v2 or v3. This should allow users to upgrade to Invenio v3.2 without being forced to upgrade their Marshmallow schemas immediately to v3.

We advice all users to start planning an upgrade of their Marshmallow schemas from v2 to v3 as the upgrade is non-trivial and needs proper testing due to significant differences between Marshmallow v2 and v3.

Invenio will continue support for both Marshmallow v2 and v3 for a transition period to allow users to upgrade at their own pace. After the transition period Marshmallow v2 support will be deprecated and removed from Invenio.

For information about how to upgrade see https://invenio.readthedocs.io/en/latest/tutorials/upgrade-marshmallow.html.

Search index prefixing

Elasticsearch does not support the concept of virtual hosts and thus with previous versions of Invenio it was not possible to share an Elasticsearch cluster between multiple Invenio instances.

We have now added support for index/alias/template prefixing, so that all names can be prefixed with a string and thus allows multiple Invenio instances to share an Elasticsearch cluster. Note, that this is only name prefixing, thus technically two Invenio instances will can read each other indexes and thus must trust each other.

Read more about the new feature on https://invenio-search.readthedocs.io/en/latest/configuration.html#index-prefixing.

Maintenance policy

Invenio v3.2 will be supported with bug and security fixes until the release of Invenio v3.4 and minimum until 2020-12-20.

See our maintenance policy.

What's next?

In Invenio v3.3 we are planning to release the Statistics bundle including:

invenio-stats
- Invenio module for statistical data processing and querying with support for collecting COUNTER Research Data Usage Metrics compliant statistics.
counter-robots
- Library for COUNTER-compliant detection of machines and robots.

In addition to the Statistics bundle, we will also release final versions of the following two modules:

invenio-index-migrator
- Elasticsearch index migrator for Invenio (aka zero down-time reindexing and index migration).

Introducing RDM, ILS and Framework

2019-10-15T00:00:00Z

We're happy to announce a major overhaul of inveniosoftware.org. Some of the highlights of the new website include:

Rebranding of Invenio into three products: InvenioRDM, InvenioILS and Invenio Framework
A new forum
People in the community
Logo downloads

Framework, RDM and ILS

The primary reason for the website overhaul is a rebranding of Invenio into three different products:

InvenioRDM - a turn-key research data management repository.
InvenioILS - an Integrated Library System.
Invenio Framework - a core framework for large-scale digital repositories.

Both, InvenioRDM and InvenioILS are applications built on top of the Invenio Framework. On each product page, you'll find a lot more information about the product as well as their current roadmaps.

Talk - Discourse forum

We are also launching a new forum for both users, administrators and developers, which replaces our current troubleshooting repository on GitHub and compliments the current chatrooms.

People and institutions in the community

We have also added a new people section to better showcase the persons who are making Invenio into a real community. Don't hesitate to email us if you'd like to be displayed on the list.

New logos

Last but not least, we've made some very minor modifications to the Invenio logo, as well as made a dedicated download page where you can get SVG versions of all logos.

team_colour:

Towards Invenio v3.2 and Elasticsearch 7 support

2019-08-07T00:00:00Z

Two CERN sprint teams with a total of 14 developers have just each finished a two-week sprint:

Sprint team 1: Focused on the Invenio v3.2 release which will include the new Files bundle.
Sprint team 2: Focused on adding Elasticsearch 7 support as well prepare the Invenio v3.3 release which will include the new index migration utilities.

Highlights

A total of 23 new module releases where made during the sprint. The highlights from the two sprints include:

Elasticsearch 7 support.
Elasticsearch index prefixing and suffixing support for shared clusters (this is needed for the upcoming index migrator utilities).
Marshmallow v2 and v3 compatibility. Invenio is now able to accept both Marshmallow v2 and v3 schemas. In your Invenio instance you will need to pin the Marshmallow version that matches your schemas, and follow the upgrade guide provided by Marshmallow to upgrade your schemas.
Sentry support is now using the Sentry-Python library instead of Raven library (you can still switch back to Raven by setting SENTRY_SDK = False in your configuration).
Rate limiting now differentiates between guests and authenticated users, and allows for external modules to provide per user rate limits.
Improved HTML sanitisation support in several modules.
Improved support for client-side infinite scroll in the REST API.
Housekeeping: we have fixed a significant number of build failures as well as deprecation warnings from other libraries.

All of above highlights will be released together with Invenio v3.2. Most of the individual modules have already been released, however you are still on your own if you decide to go head with them prior to the Invenio v3.2 release (if you encounter problems, we are of course very interested in hearing about it, so that we can solve them before the v3.2 release).

Future plans

Invenio v3.2 and Files Bundle

The primary focus is still to release Invenio v3.2. The pending issues are limited to final testing and documentation.

Elasticsearch v2 and v5 deprecation

Invenio v3.3 will add index migration utilities that will allow Invenio users to upgrade their Elasticsearch clusters to supported Elasticsearch versions. In Invenio v3.4 we plan to then remove support for Elasticsearch v2 and v5.

Python v2 support ends January 1st 2020.

Python v2.7 will reach end of life on January 1st, 2020. Invenio will only support Python 2.7 until that date. From January 1st, 2020 we will remove Python 2.7 from our test matrixes, and thus new module releases after January 1st 2020 will very likely no longer work on Python 2.7.

We are already seeing a large number of our dependent Python libraries that have removed Python 2 support, and thus we will not be able to continue Python 2 support beyond January 1st, 2020.

Invenio v3.3 - Index migration and usage statistics

Invenio v3.3 is planned for release in late 2019 or early 2020. The primary focus for Invenio v3.3 will be adding support for Elasticsearch index migration as well as releasing the Statistics Bundle (COUNTER-compliant usage statistics). The Statistics Bundle includes the following modules:

Invenio-Stats
Invenio-Queues
COUNTER-Robots

Invenio v3.3 may also see the release of a new module, Invenio-Records-Permissions, which will significantly simplify the defining and managing access control for records.

Releases overview

invenio-access: v1.2.0
- Removed DynamicPermission from Invenio-Access (deprecated since v1.0.0)
invenio-app: v1.2.0
- Fixed issue with instance_path and static_folder being globals evaluated once which caused problems with fixtures in pytest-invenio.
- Improved the rate limiting to differentiate between guests and authenticated users.
- Added possibility for external modules to provide per user rate limits via the Flask g global request object.
- Fixed deprecation warnings from Werkzeug.
invenio-assets: v1.1.3
- Changed module to hide webpack warnings (primarily needed for the cookiecutter-invenio-instance to reduce output clutter).
invenio-base: v1.1.0
- Added support for allowing instance_path and static_folder to be callables which are evaluated before being passed to the Flask application class (related to invenip-app fix ).
invenio-celery: v1.1.0
- Fixed missing release on PyPI.
invenio-config: v1.0.2
- Added ALLOWED_HTML_TAGS and ALLOWED_HTML_ATTRS default configuration for bleach HTML sanitisation library (values are used by Invenio-Records-REST, Invenio-Formatter and Invenio-Previewer).
invenio-db: v1.0.4
- Added PostgreSQL v10 into the test matrix to ensure future compatibility.
invenio-formatter: v1.0.2
- Added a new Jinja filter sanitize_html that uses the bleach library to sanitise data and to be used in combination with the safe template filter to prevent Cross-Side Scripting (XSS) vulnerabilities.
invenio-indexer: v1.1.0
- Added Elasticsearch 7 support.
- Added before_record_index.dynamic_connect() signal utility for connecting index receivers directly to specific indexes.
- Fixed Elasticsearch index prefixing support.
invenio-logging: v1.2.0
- Changed Sentry integration to use the sentry-python module instead of raven library. Raven library is still supported for backward compatibility.
invenio-oaiserver: v1.1.1
- Added Elasticsearch 7 support.
- Added Marshmallow v3 support (with v2 backward compatibility).
- Improved documentation on how to configure Elasticsearch (https://invenio-search.readthedocs.io/en/latest/configuration.html)
invenio-oauthclient: v1.1.3
- Fixed deprecation warnings from Flask-OAuthlib
- Fixed issue with the ?next parameter not supporting a query string.
invenio-records: v1.3.0
- Removed the CLI (deprecated since v1.1.0)
invenio-records-rest: v1.5.0
- Added Elasticsearch 7 support
- Added CSV serialiser (for allowing CSV exports)
- Added Marshmallow v3 support
- Added “from” and “aggs” query parameters for better supporting client-side infinite scroll use cases.
- Changed SanitizedHTML marshmallow field to use central configuration from Invenio-Config.
- Fixed a deprecation warning.
- Fixed Elasticsearch index prefixing support.
- Fixed bug with browsers not respecting the content type when caching the REST API responses. (PENDING merge)
invenio-rest: v1.1.1
- Added compatibility layer for marshmallow v2 and v3
invenio-search: v1.2.1
- Added Elasticsearch v7 support
- Fixed bug with Elasticsearch index prefixing support.
- Added index suffixing and write aliases.
- Deprecated Elasticsearch v5 support.
- Changed default library used for making request to Elasticsearch from requests to urllib3 (default recommended library).
invenio-theme: v1.1.4
- Added an error handler for HTTP 429 (rate limiting error).
xrootdpyfs v0.1.6
- Fixed bug preventing previewing large ZIP files (2GB+).

Files Bundle

The Files Bundle also saw releases of the following modules:

invenio-files-rest: v1.0.1
invenio-records-files: v1.1.1
invenio-previewer: v1.0.1
invenio-iiif: v1.0.1
pytest-invenio v1.2.0

We don’t recommend upgrading to these versions until Invenio v3.2 have been released. In particular, we have made breaking changes to Invenio-Records-Files from the v1.0.0a11 to v1.0.0, that are likely to impact you if you depended on the unsupported alpha releases.

Stay tuned for the Invenio 3.2 release!

Invenio security releases - XSS and Host header injection

2019-07-15T00:00:00Z

Two vulnerabilities have been identified in supported Invenio modules.

Invenio-Records (security advisory): A Cross-Site Scripting (XSS) vulnerability has been identified in Invenio-Records in the administration interface.
Invenio-App (security advisory): A Host header injection vulnerability has been identified in Invenio-App.

In addition, two XSS vulnerabilities have been discovered in unsupported Invenio modules:

Invenio-Previewer (security advisory): An XSS vulnerability affecting the JSON, Markdown and iPython Notebook previewers.
Invenio-Communities (security advisory): An XSS vulnerability affecting the Jinja templates.

The vulnerabilities were found after an XSS vulnerability was reported to Zenodo by Ciro Santilli. As a standard measure and after patching Zenodo, we reviewed the Invenio source code for potential similar issues to those identified in the Zenodo source code. This led to the discovery of three additional XSS vulnerabilities. The host header injection vulnerability was discovered after a standard vulnerability scan of another service running at CERN.

Releases

We have issued two new Invenio releases fixing these issues:

Invenio v3.0.2 and v3.1.1

The following individual modules fixing the vulnerabilities have been released:

Invenio-Records v1.0.2, v1.1.1 and v1.2.2
Invenio-App v1.0.6 and v1.1.1
Invenio-Previewer v1.0.0a12 (unsupported)
Invenio-Communities v1.0.0a20 (unsupported)

New security policy

We have taken the chance during handling of these vulnerabilities to also clearly define and document Invenio's security policy. Please have a look and let us know what you think.

Previously, we have sometimes privately notified potentially affected services about a security vulnerability. We have however decided to discontinue this practice, and instead, send out an advance notification to everyone about an upcoming security release including only the severity level of the issue. This allows everyone to plan ahead for the upcoming release and ensure they have staff available to handle the release. This partially smoothens the communication process but also ensures that everyone receives the same information in a scalable approach.

GitHub security advisors

As a new thing, we have also evaluated the new GitHub maintainer security advisories to handle the vulnerabilities. These advisories are reviewed by GitHub and should allow a security alert to be sent to affected repositories.

For more information

If you have any questions or comments about this security release:

Email us at info@inveniosoftware.org

Keeping up with Elasticsearch

2019-05-24T00:00:00Z

In our latest Invenio sprint from May 13-24 we've been working on making Elasticsearch upgrades, migration and reindexing simpler and with near zero downtime. Read on, to learn more about how you keep up with Elasticsearch rapid release and deprecation cycles with Invenio and for a preview of what will come of new features in Invenio v3.2.

End of life

Once you start running a production Invenio instance, you'll quickly notice that your Elasticsearch cluster will become outdated pretty fast, due to Elasticsearch's rapid release and deprecation cycle. Invenio v3 development was started in 2016 against the newly released Elasticsearch v2. When Invenio v3 was finally released in June 2018, Elasticsearch has had two major releases (v5 and v6) and v2 had already reached end of life. As can be seen from Elastic Product End of Life Dates, you can expect a major Elasticsearch version to be supported for a maximum of 2-3 years. Thus, you should plan to regularly upgrade your Elasticsearch cluster in order to not run on unmaintained software.

Index upgrades with downtime

Elasticsearch allows an index created in v5 to be read in v6 which combined with rolling cluster upgrades allows you to upgrade a cluster relatively easy with no downtime. However, you are out of luck as soon as you want to upgrade again from v6 to v7 since v7 cannot read an index created in v5.

The solution is for every major Elasticsearch upgrade to create a new index and reindex content from your old index. This requires you to take offline your Invenio instance to avoid that new content is indexed during the reindexing process. For large-scale production systems with many millions of records, reindexing can take a long time, and thus might require several hours of downtime, which can quickly be a big issue these systems.

Fixing Elasticsearch mappings

A related problem you may hit when running an Invenio instance is that you would like to tweak the Elasticsearch mappings in order to improve search performance and precision. In Elasticsearch you cannot modify the mapping of an existing index, thus similar to above, you need to create a new index and reindex content from the old index.

Use cases

All in all, you'll find yourself in need of tools to manage indexes and performing reindexing in order to:

Upgrade an Elasticsearch cluster between major versions.
Migrate between two Elasticsearch clusters.
Change Elasticsearch mappings of existing indexes.

Near-zero downtime upgrade, migration and reindexing

To solve the above use cases, we've built a new Invenio module named Invenio-Index-Migrator. The module allows you to synchronize two Elasticsearch indexes (in-cluster or cross-cluster) over an extended time period, and when you are ready, you can roll over the new index. Depending on how you have deployed Invenio, the rollover can happen with either zero downtime or near zero downtime (<5 minutes compared to hours)

The new module supports migrating records but also any other index you may have in Elasticsearch such as event logging from usage statistics. The new module works by defining migration recipes and managing them via a CLI

Demo

Disclaimer: APIs etc may change slightly before the final release.

First, you initialize a migration recipe (here a recipe named records) which will create new indexes to hold the new data:

$ invenio index migration init "records"

Next, you kick-off the index migration:

$ invenio index migration run "records"

Index migration happens in two stages:

Bulk migration: First a snapshot of the current index is reindexing into the new index. Invenio-Index-Migrator is flexible an supports several methods. By default, we use the Elasticsearch Reindex API which in most cases are orders of magnitudes faster than any other method. Other methods allow you to use Invenio's default record indexing or simply implement your own.
Synchronization: Once the first snapshot has been fully indexed, a second job is started to keep the two indexes in sync. Essentially this works by indexing any modifications to the old index in the new one.

At any point during the migration, you can check the status:

$ invenio index migration status "records"

Once you're ready, you can roll over the new index:

$ invenio index migration rollover "records"

In an in-cluster migration, the rollover works by shuffling around index aliases (since Elasticsearch does not support renaming indexes). In a cross-cluster migration you update the Invenio configuration to point to your new Elasticsearch cluster.

In case something goes wrong during the index migration, you can of course also cancel the job:

$ invenio index migration cancel "records"

Bonus

During the sprint a number of other issues were addressed that will also be part of Invenio v3.2:

Elasticsearch v7 support was added to Invenio-Search, Invenio-Indexer and Invenio-Records-REST. Only Invenio-OAIServer needs to be upgraded to support Elasticsearch 7.
Search index prefixing allowing multiple Invenio instances to use the same Elasticsearch cluster. Elasticsearch by default does not support the concept of virtual hosts. With search indexing prefixing, it will be possible to use a single Elasticsearch cluster for multiple Invenio instances, given that the Invenio instances either trust each other, or you have a protection-layer like Elastic X-Pack or ReadonlyREST in front of Elasticsearch.

Next steps

The Invenio-Index-Migrator is only the first building blocks in order to make Elasticsearch upgrades, migration and reindexing simple. We hope to add assistant-like features that will make it even easier to keep with the latest Elasticsearch releases.

The Invenio-Index-Migrator will be finally released with Invenio v3.2 which is planned for July-September.

Invenio User Group Workshop 2019, June 10th

2019-05-03T00:00:00Z

We would like to announce the 5th Invenio User Group Workshop, IUGW2019, to be held as part of Open Repositories 2019 (OR2019) on Monday, June 10th from 09:00-17:00 at the University of Hamburg (main building).

Registration (mandatory)

First register to Open Repositories 2019 conference.
Next, register to the Invenio User Group Workshop 2019.

Call for proposals

Invenio User Group Workshop (IUGW) is a biennial workshop where the Invenio repository community meet among current and future users and developers from around the world. The workshop consists of a series of tour de table service presentations and talks from attendees related to the Invenio digital repository framework.

Submit abstract

Deadline: May 22nd at 23:59 CEST

Acceptance: Notification on May 27th.

The Invenio User Group Workshop 2019 will address a wide range of topics related to the overall theme for Open Repositories 2019:

All the user needs

1. Understanding user needs and user experience

User research and engagement
User experience design for repository services
Better user experience through data and workflow integration
Improving repository user interfaces

2. Discovery, use and impact

Increasing content visibility in search engines and discovery systems
Open access discovery, research data discovery
Tools for researchers, interfaces for machines
The role of aggregation services
Measuring use and impact

3. Repositories – evolution or revolution?

Beyond the repository: using repository platforms for purposes not originally intended
Convergence with other types of systems (e.g. current research information systems, digital asset management systems, journal publishing platforms, library service platforms)
Interoperability vs integration: will repositories survive as stand-alone systems?
The developing role of repositories in the scholarly communications and research information systems ecosystem (e.g. the Next Generation Repositories vision)
New models for scholarly sharing (e.g. blockchain)
Data mining, artificial intelligence and machine learning

4. Supporting open scholarship and cultural heritage

Providing access to different types of materials (e.g. research data, scholarly articles, pre prints and overlay journals, open access monographs, theses and dissertations, educational resources, archival and cultural heritage materials, audiovisual materials, software, interactive publications and emerging formats)
Workflows and support services for the repository users
Training, communication and outreach
Long-term access and preservation
Repositories as digital humanities and open science platforms
Working with large and complex data sets

5. Open and sustainable

Service and business models that meet user needs
Local systems vs repository as a service
The expanding role of service providers in the repository landscape, pros and cons?
Sustainability of the open source community model
Securing long-term funding for open infrastructures
Open business models and open governance for open infrastructures

6. Policies, licensing and the law

Impact of GDPR (General Data Protection Regulation) and copyright laws
Publisher policies, embargoes and rights retention
Licenses, use and re-use of content
‘Closed’ material in ‘open’ repositories
Compliance and impact of funder policies (e.g. Plan S) on repositories

7. How can metadata and standards help our users?

Development and standardisation of repository metadata
Data models and entities
Linked open data and repositories
Persistent identifiers (e.g. DOI, Handle, URN, ORCID, ISNI)
Open citations
International Image Interoperability Framework (IIIF)

8. Repositories and global knowledge

Integration with other open knowledge resources (e.g. Wikimedia and Wikidata)
National vs global solutions
Repository systems and language barriers
Repositories in the global south
User needs in developing countries

InvenioRDM: a turn-key open source research data management platform

2019-04-29T00:00:00Z

CERN has partnered with 10 multidisciplinary institutions and companies to build a turn-key open source research data management platform called InvenioRDM, and grow a diverse community to sustain the platform.

The InvenioRDM project is funded by the CERN Knowledge Transfer Fund, as well as all the participating partners, including:

Brookhaven National Laboratory (US)
Caltech Library (US)
CTSA Program National Center for Data to Health (US)
Data Futures (UK)
Graz University of Technology (AT)
Helmholtz Zentrum Dresden-Rossendorf (DE)
Northwestern University (US)
OpenAIRE (GR)
TIND (NO)
Tubitak (TK)
University of Hamburg (DE)
University of Münster (DE)

The project has an ambitious one year schedule in which it will deliver:

InvenioRDM - A research data management platform based on Zenodo and Invenio v3 Framework.
A community of public and private institutions to sustain InvenioRDM.
Minimum two existing repositories migrated to InvenioRDM, with Zenodo being one of them.

The key to successfully achieving the ambitious schedule is that InvenioRDM will be based on Zenodo that have already been successfully validated over the past 5 years.

Our vision in the next five-years, is to make InvenioRDM a world-leading extensible research data management platform used by research institutions all around the world and with businesses providing services, support and customizations on top of InvenioRDM.

What is a RDM platform?

An research data management (RDM) platform allows researchers to share and preserve scientific results. Researchers can share anything from publications, posters, presentations to datasets and software. Once a researcher have shared a result, they get a DOI (Digital Object Identifier), that allow them to properly cite their result.

There are three primary purposes of RDM is to

Disseminate and archive
Enable reproducibility
Enable reuse

Most importantly, research funding agencies all over the world have realised the huge potential economic and social benefits of RDM to society and are now demanding solutions.

Zenodo

CERN in partnership with OpenAIRE has built one such RDM service with European Union funding called Zenodo. Zenodo has been highly successful and has in its 5 years of existence become a world-leading general purpose research data repositories.

In fact, other institutions have already taken Zenodo source code (which is open source), and started building their own local RDM solutions on it. The goal of this project is to join our efforts to build a common RDM-platform from which both CERN, other institutions and private businesses can profit.

Multidisciplinary partners

The real strength of the project is that it brings together a suite of partners from multidisciplinary domains, that each bring unique knowledge and know-how from their specific domains that will be critical to the success of project:

Institutional partners:
- University of Hamburg
- University of Münster
- Caltech Library
Health and medical science partners:
- Northwestern University
Physics partners:
- Helmholz Zentrum Dreseden Rossendorf
- Brookhaven National Laboratory
- TUBITAK - The Scientific and Technological Research Council of Turkey
Digital humanities partners:
- Data Futures
Business partners:
- TIND Technologies
Community partners:
- OpenAIRE

Building InvenioRDM

InvenioRDM will include most of the features that Zenodo alreday include today such as e.g. DOI miniting capabilities, versioning support and COUNTER compliant usage statistics to name some few.

The work in transforming Zenodo into a general purpose RDM-platform will involve three key areas:

Core repository

The repository platform will at its core include an extensible metadata model based DataCite metadata schema with support for handling millions of records and peta bytes of data. The repository will further be aligned with the Next Generation Repositories (NGR) standard.

Packaging and distribution

Key for easy adoption of InvenioRDM is to ensure that it is a real turn-key solution requiring minimal experience in installing, operating and administering the platform, or in short getting users started in no time. Thus, a significant part of the efforts will go into simplifing the installation, improving the packaging and distribution, as well as providing excellent end-user documentation.

Customization and extendability

A key requirement for InvenioRDM is that it can easily be extended and customized just enough to adapt to each particular institution. This includes for instance defining authentication mechanisms (SAML/LDAP/OAuth), integrating with mutliple storage backend system, and most important of all to make these customizations easy.

Contact

For more information about the InvenioRDM project, please contact:

Lars Holm Nielsen

Invenio Product Manager

CERN IT Department

Email: info@inveniosoftware.org

Invenio Training Bootcamp 2019

2019-03-25T00:00:00Z

The Invenio v3 Bootcamp was held from the 19th to the 21st of March, 2019 at CERN. The event reached the maximum number of 30 participants with attendees from all over the world (Czech Republic, Danmark, Germany, Finland, France, Japan, Norway, Spain, Switzerland, UK and USA). The bootcamp was targeting developers wanting to learn more about Invenio and to understand how to create or customize an Invenio repository.

Format

The three days of the bootcamp was organised with a succession of talks and practical hands-on sessions, presented by 6 Invenio experts. Participants were able to develop each of the functionality on their laptop with a constant assistant in case of need.

The objective was to discover the Invenio framework and acquire knowledge on how to build a new repository by progressively introducing new concepts.

The main topics included:

Getting started with Invenio
Tour of functionalities and infrastructure
Customizations of data models and look & feel
Deposit of new records and references between data models
Access control
Security
Deployment and application architecture

Try it yourself

We've made all the material publicly available for all those who couldn't participate in the event, so you can try out the same exercises at home:

If you run into troubles, we're happy to answer questions you may have.

Takeaways

Participants had the opportunity to experience how to work with Invenio thanks to the hands-on sessions and to understand how to apply each of the introduced functionalities to use cases of their organization.

The bootcamp attracted quite a lot of interest for Invenio and participants were very involved and curious: we were very glad to answer to all of the interesting questions that were asked.

We would like to thank everyone for their participation and help to have contributed to the success of the event.

Invenio User Group Workshop 2019 @ Open Repositories 2019, June 10th

Our next event is the Invenio User Group Workshop (IUGW) at the Open Repositories 2019 conference in Hamburg. The workshop will be held on June 10th with presentations from Invenio users from around the world.

The call for proposals will be announced early April, so stay tuned.

See you in Hamburg!

Invenio v3.1.0 released

2019-03-11T00:00:00Z

We are proud to announce the release of Invenio v3.1.0.

Head over to our Getting started to see it in action.

Python compatibility

Invenio v3.1 supports Python 2.7 (until 2019-12-31), Python 3.5 and Python 3.6. We expect to add support for Python 3.7 in the near-term future once Celery v4.3 has been released.

What's new in Invenio v3.1?

Webpack build system

Invenio v3.1 comes with a new assets build system based on Webpack for building and packaging your JavaScript applications, stylesheets and much more. The system replaces the previous AMD/RequireJS based system which was deprecated in v3.0.

The old build system is still available to allow users to upgrade to Invenio v3.1 without first migrating to Webpack. The old build system will be removed in Invenio v3.3

For more information about the new build system, please see:

Simplified scaffolding

We have simplified the scaffolding of new Invenio instance by merging the data model template into the main Cookiecutter-Invenio-Instance.

The previous approach of two separate packages -- one for the application and one for the data model -- caused friction and confusion for new users and we, therefore, decided to merge both.

This also fits with our long-term goal, where we want to provide standard data models (such as DataCite, Dublin Core, MARC21) so that users don't have write their own data model.

Docker base image

We have released a new Docker image that can serve as base image for your Invenio instances. The image is based on CentOS 7 and comes with Python 3.6, Node.JS, NPM and some standard libraries often needed by Invenio.

See inveniosoftware/centos7-python on DockerHub.

Pipenv

In order to manage Python dependencies more reliable and securely for your Invenio instance we have moved to use Pipenv which also handles the virtualenv creation. This has all been integrated with the Getting started guide.

Documentation

New sections where added to the documentation specifically on:

Bundles
Requirements
Build a repository
Managing access
Secure your instance
Infrastructure architecture

See https://invenio.readthedocs.io.

Request tracing

Invenio v3.1 has added new features for improved request tracing to allow for better troubleshooting and auditing of problems. The feature allows logging a request id, session id and user id across multiple services such as Nginx and Invenio error logs. This enables e.g. system administrators to identify exactly which Nginx access log line caused a specific error logged by Invenio.

If combined with e.g. centralised log aggregation, this can be used for e.g. viewing requests by a user in real-time, request performance statistics and many other metrics. Please note that in order to be compliant with EU General Data Protection Regulation (GDPR), you must ensure that these logs are automatically deleted after 3 months (the same is the case if you only log an IP address).

Cookiecutter-Invenio-Instance:
- Nginx configuration has been updated to automatically generate a random request id and add is as X-Request-ID header.
- Nginx log format has been updated to log timing information, request id, session id and user id if provided by the application server in the X-Session-ID and X-User-ID HTTP headers. Nginx will remove both headers prior to sending the response to the client.
Invenio-App:
- Extracts the X-Request-ID header (max 200 chars) if set in the HTTP request and makes it available on the Flask g object as g.request_id.
Invenio-Logging:
- The request id is made available to all log handlers.
- The Sentry log handler will add the request ID as a tag if available.
Invenio-Accounts
- The X-Session-ID and X-User-ID HTTP headers will be added to the HTTP repsponse if the configuration variable ACCOUNTS_USERINFO_HEADERS is set to True. This makes the session and user id available to upstream servers like Nginx.

Minor changes in v3.1

Token expiration

The token expiration was changed from 5 days to 30 minutes for the password reset token and email confirmation token. Using the tokens will as a side-effect login in the user, which means that if the link is leaked (e.g. forwarded by the users themselves), then another person can use the link to access the account. Flask-Security v3.1.0 addresses this issue, but has not yet been released.

Globus.org OAuth Login

Invenio v3.1 now comes with support for login with your Globus.org account. The feature was contributed by University of Chicago.

See Invenio-OAuthClient for details.

Health-check view

A /ping view that can be enabled via the APP_HEALTH_BLUEPRINT_ENABLED configuration variable has been added to support load balancers like HAProxy to check if the application server is responsive.

Backwards incompatible changes

Pytest-Invenio: The celery_config fixture has been renamed to celery_config_ext due to naming conflict with fixture provided by Celery.

Deprecations in v3.1

Following list of features have been deprecated and will be removed in either Invenio v3.2 or Invenio v3.3:

Elasticsearch v2 support

Elasticsearch v2 support will be removed in Invenio v3.2. Elasticsearch v2 has reached end of life and no longer receives any bug or security fixes.

Both the support in Invenio-Search for creating indexes for v2 as well as any v2 Elasticsearch mappings in other Invenio modules will be removed.

AMD/RequireJS

Invenio's assets build system based on AMD/RequireJS will be removed in Invenio v3.3.

This involves e.g. the two CLI commands:

$ invenio npm
$ invenio assets build

Several Python modules in Invenio-Assets will be removed, including (but not limited to):

invenio_assets.npm
invenio_assets.filters
invenio_assets.glob
invenio_assets.proxies

Also, bundle definitions in other Invenio modules will be removed. These are usually located in bundles.py files, e.g.:

invenio_theme.bundles

Also, some static files will be removed from bundles, e.g.:

invenio_theme/static/js/*
invenio_theme/static/scss/*

DynamicPermission class

The invenio_access.DynamicPermission class will be removed in Invenio v3.2. It has been superseded by the invenio_access.Permission class. The Permission class by default deny an action in case no user/role is assigned. The DynamicPermission instead allowed an action if no user/role was assigned.

Records CLI

The following CLI commands will be removed in Invenio v3.2:

$ invenio records create
$ invenio records delete
$ invenio records patch

Please use the REST API instead to create, patch and delete records.

AngularJS (reminder from v3.0)

In Invenio v3.0 we deprecated the AngularJS 1.4 application Invenio-Search-JS as AngularJS by that time was already outdated. We have selected React and SemanticUI as the replacement framework for AngularJS.

The new Webpack build system released in Invenio v3.1 is part of the strategy to move from AngularJS to React (note however that you can use Webpack with your favorite framework, including AngularJS).

We have started the rewrite of Invenio-Search-JS and have already released the first version of React-SearckKit which eventually will replace Invenio-Search-JS.

Features removed in v3.1

These following already deprecated features have been removed in Invenio v3.1.

invenio_records.tasks was removed from the Invenio-Records module.

Maintenance policy

Invenio v3.1 will be supported with bug and security fixes until the release of Invenio v3.3 and minimum until 2020-03-31.

What's next?

We originally planned to release the Files and Statistics bundle in Invenio v3.1. We however decided it was more urgent to release the new Webpack build system in order to avoid too much code being written against the old build system.

In Invenio v3.2 we are planning to release the Files bundle including:

invenio-files-rest
- Object storage REST API for Invenio with many supported backend storage protocols and file integrity checking.
invenio-iiif
- International Image Interoperability Framework (IIIF) server for making thumbnails and zooming images.
invenio-previewer
- Previewer for Markdown, JSON/XML, CSV, PDF, JPEG, PNG, TIFF, GIF and ZIP files.
invenio-records-files
- Integration layer between object storage and records.

Invenio v3 Training Bootcamp

2018-12-21T00:00:00Z

We are pleased to announce the first Invenio v3 Bootcamp taking place at CERN, 19-21 March 2019. The Bootcamp is intended as an introduction to developing digital repositories with the Invenio v3 framework.

Website

https://indico.cern.ch/e/invenio-bootcamp/

Planned topics

Creating your first Invenio instance.
Customizing the look and feel.
Working with data models.
Managing access to records.
Managing files.
Creating a new module from scratch.
Depositing records.
Securing your Invenio instance.
Deploying Invenio.

The topics will be covered through practical tutorials and presentations. Note, that the final programme is subject to change, based on inputs we receive from you in the registration form.

Who can register

All registrations are subject to our approval (i.e. don't book flights until we confirm your registration).
Intended audience:
- Software developers with some prior Python experience.
Limited capacity:
- We have limited capacity so we prioritise to have as many different institutions/companies represented as well as people with concrete projects for which they plan to use Invenio v3.

How to register

https://indico.cern.ch/event/773969/registrations/46902/

Cost

The Bootcamp itself is free of charge, but you will need to cover your own expenses during the Bootcamp including (but not limited) to lunches, dinners, coffee, accommodation, transport and social events.

Location

CERN, Geneva, Switzerland.

team:

Invenio

Invenio v3.0.0 Released

2018-06-07T00:00:00Z

Welcome to Invenio 3!

We are proud to announce the release of Invenio v3.0.0. Invenio has been completely rewritten from scratch with a radically improved architecture and technical implementation. Invenio 3 is now a framework, like a Swiss Army knife, complete with battle-tested, safe and secure modules providing all the features you need to build and run a trusted digital repository.

Whilst Invenio 3 is officially released to the world today, in reality it has already been relied upon in large-scale production systems for more than 1.5 years on sites such as:

Also other sites are already in process of being built on Invenio 3:

INSPIRE HEP - an aggregator for High-Energy Physics.
WEKO3 - repository platform for 500+ Japanese universities.

What's new

Invenio functionality is being released in bundles of modules. Invenio v3 contains the following bundles totaling more than 27 individual Invenio modules:

Base: the core application framework with e.g. distributed task queue support.
Auth: accounts management, role-based access control, OAuth 2.0 client and provider, user profiles management.
Metadata: record and persistent identifier management including indexing, querying and OAI-PMH server.

The following bundles are being prepared for release in v3.1:

Files: advanced file management with multi-backend support as well as IIIF Image API support.
Statistics: COUNTER-compliant statistics.

See our roadmap for further details.

Getting started

In order to get started developing with Invenio v3 follow our getting started guide.

Next, head over https://invenio.readthedocs.io to understand how to develop with Invenio.

In addition, each Invenio module also has extensive documentation:

Base bundle

Auth bundle

Metadata bundle

Compatibilities

Python compatibility

Invenio v3.0 supports Python 2.7, 3.5, 3.6. We highly recommend only using the latest official release in each series.

Python 2.7 end-of-life is scheduled for April 2020. Invenio will only support Python 2.7 until that date. We highly recommend that all new projects are started on the latest available Python 3 version.

Elasticsearch compatibility

Invenio v3.0 supports Elasticsearch 2, 5 and 6.

Elasticsearch v2 has reached end-of-life (February 2018) and Invenio v3.0 is the last release to support Elasticsearch v2.

PostgreSQL compatibility

Invenio v3.0 supports PostgreSQL 9.4, 9.5 and 9.6. We have not yet tested Invenio v3.0 with PostgreSQL 10.

MySQL compatibility

Invenio v3.0 supports MySQL 5.6+.

Deprecations

AMD/RequireJS

Invenio v3.0's current static assets management system is based on e.g. RequireJS will be replaced with Webpack. We expect this work to be ready for Invenio v3.1, and thus we are already deprecating the current support. Specifically this means that Invenio-Assets and Invenio-Theme will change significantly in Invenio v3.1. We would have liked to already have this ready for this v3.0 release, but unfortunately it was time-wise not possible.

AngularJS

Invenio v3.0 comes with one AngularJS 1.4 application (Invenio-Search-JS). AngularJS is by now already outdated, and we are planning a rewrite of the application in another JavaScript framework that is currently in process of being selected. Essentially this means that you should not extend Invenio-Search-JS at this point, since it will change significantly.

Maintenance policy

Invenio v3.0 will be supported with bug and security fixes until the release of Invenio v3.2 and minimum one year.

We aim at one Invenio release with new features every 6 months. We expect upgrades between minor versions (e.g. v3.1 to v3.2) to be fairly straight-forward as in most cases only new features are added.

Sprint: v3.0.0RC2 + updated website

2018-05-28T00:00:00Z

This sprint (May 14-25) was focused on preparing Invenio for the final v3.0.0 release so it is ready for launch at the Open Repositories 2018 Conference, June 4th-7th in Bozeman, Montana.

Overview

Result of the sprint (45 developer days, 107 commits, 8.5k lines touched):

inveniosoftware.org now has shiny new blog and roadmap.
inveniosoftware.org was moved to Lektor static CMS and is now also hosted on GitHub Pages.
Cookiecutter templates for Invenio instances and Invenio data models where finalised and made part of the "Getting started" guide.
Invenio v3.0.0 Release Candidate 2 was released.
Flask v1.0 support

Releases

invenio-access v1.0.1
invenio-accounts v1.0.1
invenio-base v1.0.1
invenio-db v1.0.1
invenio-formatter v1.0.1
invenio-oauth2server v1.0.1
invenio-records-rest v1.1.0
invenio-userprofiles v1.0.1
pytest-invenio v1.0.2

List of all changes

cookiecutter-invenio-datamodel:
- Fixing, testing, documenting and cleaning.
cookiecutter-invenio-instance:
- Fixing, testing, documenting and cleaning.
- The development server is now by default running SSL thanks to the recently release Flask v1.0.
invenio-access:
- Fixed annoying warning message.
invenio-accounts:
- Fixed build failure due to Flask-CeleryExt version.
invenio-base:
- Added support for blueprint factory functions in the invenio_base.blueprints/invenio_base.api_blueprints entry point groups. In addition to specifying an import path to an already created blueprint, you can now specify an import path of a blueprint factory function with the signature create_blueprint(app), that will create and return a blueprint. This allows moving dynamic blueprint creation from the extension initialization phase to the blueprint registration phase.
invenio-formatter:
- Fixed issue where badges would be cached wrongly.
invenio-jsonschemas:
- Fixed build error related to Flask v1.0
invenio-oaiserver:
- Fixed build error related to Flask v1.0
invenio-oauth2server:
- Fixed build error related to Flask v1.0
invenio-oauth2server:
- Fixed build error related to Flask v1.0
invenio-oauthclient:
- Fixed build error related to Flask v1.0
invenio-records-rest:
- Added marshmallow fields used for record deserialization.
- Added support for other Invenio extensions to auto-register REST API record endpoints.
- Fixed build error related to Flask v1.0
invenio-rest:
- Fixed build error related to Flask v1.0
invenio-userprofiles:
- Fixed build error related to Flask v1.0
invenio:
- Documentation updates.
inveniosoftware.org:
- Rewritten using Lektor Static CMS system.
- Retargeted website to pass the message that Invenio is now a framework.
- Added a blog.
- Added a roadmap
- Moved all information about legacy releases to separate page.
- Many fixes to design and style.
- Reworked most text sections.
- Moved hosting to GitHub Pages.
pytest-invenio:
- Fixed problem with the celery_config fixture.
- Fixed issue with elasticsearch import.

Sprint: v3.0.0RC1, Elasticsearch v6 and MIT license

2018-03-23T00:00:00Z

This sprint was focused on releasing the Invenio v3.0.0 Release Candidate 1.

As a result of 95 developer days, 485 commits and 88k lines touched (33k additions and 55k deletions):

Invenio v3.0.0 Release Candidate 1 was released.
License has been changed from GPL to MIT for ~30 repositories (part of v3RC1).
Documentation for API authentication in Invenio was added (in OAuth2Server).
Elasticsearch v6 support was added.

Releases

invenio: v3.0.0rc1
invenio-access: v1.0.0
invenio-accounts: v1.0.0
invenio-admin: v1.0.0
invenio-app: v1.0.0
invenio-assets: v1.0.0
invenio-base: v1.0.0
invenio-cache: v1.0.0
invenio-celery: v1.0.0
invenio-config: v1.0.0
invenio-db: v1.0.0
invenio-formatter: v1.0.0
invenio-i18n: v1.0.0
invenio-indexer: v1.0.0
invenio-jsonschemas: v1.0.0
invenio-logging: v1.0.0
invenio-mail: v1.0.0
invenio-oaiserver: v1.0.0
invenio-oauth2server: v1.0.0
invenio-oauthclient: v1.0.0
invenio-pidstore: v1.0.0
invenio-records: v1.0.0
invenio-records-rest: v1.0.0
invenio-records-ui: v1.0.0
invenio-rest: v1.0.0
invenio-search: v1.0v.0
invenio-search-ui: v1.0.0
invenio-theme: v1.0.0
invenio-userprofiles: v1.0.0
dcxml: v0.1.1
citeproc-py-styles: v0.1.1
pytest-invenio: v1.0.0
invenio-marc21: v1.0.0a8

Backward incompatible changes

invenio-oauthclient:
- Added creation/modification timestamps to database models. Requires upgrade of database using the included Alembic recipe. The recipe will set the creation/modification timestamp to the current date + time (same value for all rows).
invenio-oaiserver:
- XSL stylesheet that allow nice rendering of the OAI-PMH feed have been removed as it was GPL licensed. Documentation describes how to add it back.

List of all changes:

All packages:
- License change and general repository clean up.
cookiecutter-invenio-datamodel:
- New template for bootstrapping data models in Invenio.
cookiecutter-invenio-instance:
- New template for bootstrapping new instances.
invenio-accounts:
- Removed Invenio-Admin dependency.
invenio-base:
- Documentation for CLI commands.
invenio-config:
- Documentation for how to set complex values in environment variables.
invenio-db:
- Changed dependency psycopg2 to psycopg2-binary due to warning message.
invenio-indexer:
- Elasticsearch v6 support
invenio-jsonschemas:
- Fixed functools dependency for Python 2
invenio-marc21:
- Elasticsearch v6 support
invenio-oaiserver:
- Elasticsearch v6 support
- XSL stylesheet removal and documentation.
- Removal of invenio-query-parser dependency
invenio-oauth2server:
- Usage documentation
- Fixed missing field length validation in form.
invenio-oauthclient:
- Added timestamps to all models
- Hide part of access token when displayed.
invenio-pidstore:
- Allow modifying "control_number" field for recid minter via config.
invenio-records-rest:
- Usage documentation
- Elasticsearch v6 support
- Removed invenio-query-parser dependency.
invenio-rest:
- Always install Flask-CORS dependency. Please update your install from invenio-rest[cors] to just invenio-rest.
- Add sentry event ID in error response.
- Change from dicttoxml to xmltodict (to avoid GPL dependency).
invenio-search-js:
- Upgrade jquery to v3.3.1
- Fixed facet selection with no results
invenio-search-ui:
- Fixed facet selection with no results
invenio-search:
- Elasticsearch v6 support
- Removed invenio-query-parser dependency.
invenio:
- Removed provisioning scripts and docker configuration (in favour of cookiecutter-invenio-instance).
- Bump all dependencies to v1.0.0
pytest-invenio:
- Fixed click logging output with latest Flask development version.

Sprint: Elasticsearch v5 support

2017-12-01T00:00:00Z

This sprint was focused on:

Elasticsearch v5 support
Preparing to release the metadata bundle.
Removing remaining Invenio-DB warnings.

During the sprint 60 developer days were spent, 52 commits were created and 4.3k lines were touched (3.1k additions and 1.2k deletions).

List of changes:

Cookiecutter-Invenio-Module:
- Minor various fixes for build issues.
IDUtils (v1.0.0):
- global: fix DOI Unicode issues
Invenio-Base (v1.0.0b1)
- Cookiecutter template removal (to be replaced by upcoming cookiecutter-invenio-instance).
- Release checklist, docs build errors, 100% test coverage.
Invenio-DB (v1.0.0b9)
- Remove annoying warning.
Invenio-I18N (v1.0.0b4)
- Example app rendering on ReadTheDocs
Invenio-Indexer (v1.0.0b1):
- New BulkRecordIndexer class with RecordIndexer-compatible API (to be used by Invenio-Records-REST).
- Elasticsearch v5 support.
- Release checklist.
Invenio-JSONSchemas (v1.0.0a7)
- Documentation improvements.
- Release checklist.
Invenio-Mail (v1.0.0b1)
- Broken docs build fix (related to Celery problem).
Invenio-OAIServer (v1.0.0b1)
- Description support in Identify verb (eprints, friends etc).
- Elasticsearch v5 support.
- Release checklist + documentation improvements.
- License change PR
Invenio-OAuth2Server (v1.0.0b3)
- Unpinned oauthlib.
Invenio-PIDStore (v1.0.0b2)
- Release checklist.
Invenio-Records-REST (v1.0.0b5)
- Index after create, update and delete record creation (this will later impact Invenio-Deposit).
- Serializers refactored to be more easily composable.
- Improved tombstone handling for the REST API (e.g. include removal reason).
- Elasticsearch v5 support.
- Improved documentation.
- Dynamic aggregations.
- Release checklist.
Invenio-Records-UI (v1.0.0b2)
- Release checklist + example app docs fix for RTD.
Invenio-Search (v1.0.0b4)
- Elasticsearch v5 support (via version-specific mappings).
- Support for creating only specific indexes.
- CLI for listing all indexes and aliases.
Invenio-Search-UI (v1.0.0a9)
- Release checklist.
Invenio-MARC21 (v1.0.0a6)
- Elasticsearch v5 support.

Elasticsearch 5 support

Search, Indexer, Records-REST, OAIServer and MARC21 have all been upgraded to support Elasticsearch v2 and v5. Elasticsearch v6 is not yet support due to the elasticsearch-dsl package not yet supporting v6 (support already merged in master branch but not yet released).

Other Invenio packages have not yet been upgraded to Elasticsearch v5. E.g. Records-Files, OpenAIRE, Stats, Collections, OpenDefinition, Query-Parser have not yet been tested with Elasticsearch v5.

Choosing version

You will need to know at install-time which version of Elasticsearch you'd like to use. For instance to use Elasticsearch 5 you need to install Invenio-Search like this:

$ pip install invenio-search[elasticsearch5]

Mappings

Due to the differences between Elasticsearch versions, we have opted for version-specific mappings. This means that Invenio modules must provide a mapping per Elasticsearch version they wish to support. E.g. today, the mapping is placed in a directory like e.g.:

mappings/records/record-v1.0.0.json

To support Elasticsearch v2 and v5 you now need two mappings:

mappings/v2/records/record-v1.0.0.json
mappings/v5/records/record-v1.0.0.json

Note that mappings for Elasticsearch v2 may use either the mappings/v2 directory or the mappings/ directory like previously (for backward compatibility).

Adding Elasticsearch v5 support to a module

In case you have site-specific modules and would like to add Elasticsearch v5 support here's a rough guide:

Update travis.yml to test both v2 and v5 (example)
Update setup.py by moving Invenio-Search dependency to extra_require (example).
Move existing mappings and add new mappings for v5 (example)
- Most common change from v2 to v5 is the change from string type to either text or keyword type.
Update docs/requirements.txt by adding elasticsearch5 as an extra requirement to ensure ReadTheDocs builds will be fine.
Fix any API specific calls (see below).

Canonical way of checking for ES version.

from elasticsearch import VERSION as ES_VERSION

if ES_VERSION[0] == 2:
    # ...

Completion Suggesters

The Completion Suggesters have changed from v2 to v5. In v2, suggesters supported an index-time payloads option, which was used to store and return metadata with suggestions. In v5, completions are now returned with their associated document in the _source field.

If you have completion suggesters for v2 you will need to make them compatible with v5. This involves:

API clients should read metadata from _source instead of payload. For v2 the payload is copied to _source by Records-REST, which allows you to already now upgrade API clients to use the new _source field.
On indexing, you need to add the payload only for v2.

Elasticsearch v2 end-of-life

Elasticsearch v2 reaches end of life in February 2018. Elasticsearch v2 support in Invenio will be removed in Invenio v3.1, thus Invenio v3.0 will be released with Elasticsearch v2 support and be maintained until v3.0 end of life (currently TBD).

Sprint: JSONB, bugs & bouncing search

2017-10-20T00:00:00Z

This sprint was focused on:

Metadata bundle: Finalize Invenio-Records, Invenio-Search, Invenio-JSONSchemas.
Fixing new bugs in Base/Auth bundles.

As a result of 40 developer days, 37 commits and 2.8k lines touched (2k additions and 0.8k deletions), the following improvements were implemented:

Fixed bouncing of search results (sorting of the same query could change depending on which Elasticsearch node your query would be answered by).
JSONB now being used for record storage (Thanks to Javier for the PR).
Rendering of JSONSchemas, meaning allOf and $refs can now be resolved on-the-fly to generate an self-contained schema for e.g. deposit forms (Thanks to Pamfilos for the PR).
Boring fixes that makes sure the Base and Auth bundles are stable.

List of changes:

Invenio-Records (v1.0.0b4):
- Beta release (release checklist).
- Changed data storage from JSON to JSONB (requires data migration and PostgreSQL v9.4+) Thanks to Javier for PR.
- Changed signals receiver signatures (backward-incompatible!).
- Fixed invalid MARC in demo records causing export errors.
- Fixed CLI for deletion of records.
- Fixed fractional seconds problem in MySQL causing tests to fail.
Invenio-Search (v1.0.0b1):
- Beta release (release checklist, documentation)
- Fix bouncing search results.
- Bumped Travis PostgreSQL to v9.4 to support JSONB.
Invenio-JSONScheams (v1.0.0a7):
- Fixed double registration of schema endpoint on both UI and API app.
- Added support for resolving allOf and JSONRefs in JSONSchemas (thanks to Pamfilos for PR).
- Fixed some documentation issues.
Invenio-Records-REST (v1.0.0b3):
- Fix bouncing search results.
- Bumped Travis PostgreSQL to v9.4 to support JSONB.
Invenio-OAuth2Server (v1.0.0b2):
- Improved authorization template design (text alignment, cover page usage and display number of users).
- Broken tests fix due to cryptography package changes.
Invenio-DB (v1.0.0b8):
- Alembic documentation refactored and integrated in docs.
Invenio-Records-UI:
- Bumped Travis PostgreSQL to v9.4 to support JSONB.
Invenio-OAIServer (v1.0.0a14):
- Invenio-Records signals signature compatibility.
Invenio-App-ILS:
- Test for ensuring all records export formats are working.

Sprint: Security and annoying warnings

2017-08-11T00:00:00Z

This sprint was focused on:

Data model issues (in Invenio-Access and Invenio-OAIServer).
Security issues (permanent sessions, "remember me", content security policy).
Working demo site (remove annoying warnings, fixed docs, SSL problems, bugs, admin interfaces, ...).

As a result of 89 developer days, 126 commits and 6.6k lines touched (4.7k additions and 1.8k deletions) auth bundle has been stabilized and released in beta version (Accounts, Access, Profiles, OAuthClient and OAuth2Server).

List of changes:

DoJSON (v1.3.2):
- Remove 'Undo' is experimental warning
Flask-Menu (v0.6.0):
- Python 3 warnings fixes
Invenio
- Fixed login problem
Invenio-Access (v1.0.0b1):
- Reviewed module and fixed data model issues.
- Deprecated DynamicPermission in favor of Permission (aligning with Flask-Principal's deny by default behavior)
- Added system roles with support for any user and authenticated user (could be extended to support IP-based access control). ActionUsers was previously used for similar feature by setting user_id to None but this is no longer possible.
- Updated administration interface.
- Added usage documentation (https://invenio-access.readthedocs.org).
- Fixed superuser issues.
Invenio-Accounts (v1.0.0b8):
- Fixed Content Security Policy issues
- Removed remember me login support in favor of using permanent sessions (remember me support could be used to circumvent a revoked session).
- Removed support for login via headers (enabled by Flask-Security by default).
- Fixed Content Security Policy problems in templates.
- Upgraded to Flask-Security v3 (thanks to @jacquire).
Invenio-Admin (v1.0.0b3):
- Disabled Content Security Policy on admin interface.
Invenio-App (v1.0.0b1):
- Adds Jinja byte code caching support.
- iPython is now the default shell.
Invenio-App-ILS (v1.0.0a3):
- Added initial Selenium integration tests.
- Fixed email sending when in debug mode.
- Bumped all packages to latest versions.
- Fixed Celery 4 configuration warnings.
- Clarified force HTTPS behaviour and adapted the user guide.
Invenio-Celery (v1.0.0b3):
- Fixed Celery 4 configuration warnings.
Invenio-DB (v1.0.0b8):
- Disabled SQL statement printing when in debug mode (has to be enabled manually now).
Invenio-I18N (v1.0.0b4):
- Fixed Content Security Policy issues in templates.
Invenio-OAIServer (v1.0.0a13):
- Fixed selective harvesting by timestamp caused by Marshmallow field parsing bug.
- Removed updated timestamp _oai.updated from record in favor of using the record models updated date (fixes issue with selective datetime harvesting).
- Add support for searching by spec in admin interface.
Invenio-OAuth2Server (v1.0.0b1):
- Added feature to show scopes related to an authorized application.
- Added client example application to enable easier testing.
- Added new scope user:email which when granted will return the user's email address in the access token.
- Updated "authortize this application" template.
- Fixed security issue that allowed obtaining a session cookie via an access token and thus bypassing scope protection.
- Fixed Content Security Policy issues in templates.
- Fixed issue when strings where not strictly URL encoded (better error message).
- Fixed template rendering issues when no scopes where given and with example URLs.
Invenio-OAuthClient (v1.0.0b2):
- Added admin interface for UserIdentity.
- Fixed Flask-WTF v0.14/v0.13 CSRF validation issues.
- Reorganized documentation to new structure.
- Removed support for remember me feature.
- Remove "Linked accounts" menu item when no providers where defined.
- Fixed Content Security Policy issues in templates.
- Fixed issue with always redirecting to "Linked accounts" after a login.
Invenio-PIDStore (v1.0.0b2):
- New release.
Invenio-Search-JS (v1.2.0):
- Fixed strict URL encoding of query strings.
- Fixed Content Security Policy issues in CSS.
Invenio-Search-UI (v1.0.0a7):
- Fixed template issue.
Invenio-Theme (v1.0.0b4)
- Fixed Content Security Policy issues in templates.
Invenio-Cache (v1.0.0b1):
- New module which provided Redis/Memcahed caching support.

What's next?

Next Invenio Sprint will focus on:

Metadata bundle:
- JSONSchemas, PIDStore, Records, Indexer, Records-UI, Search, OAIServer, Records-REST
General documentation
Framework launch (process, branches, maintenance plan, user experience)