The National Data Library: building the foundations of the UK’s data infrastructure

The ambition of the National Data Library is to lay the foundations of the UK’s future data infrastructure. To do this, the NDL needs to consolidate the current fragmented ecosystems and to put forward a clear case of its benefits for the UK economy and society.

5 min read 9 February 2025 By Adam Brocklesby, Partner and expert in Government & Roberto Pitea and Steve Shelton, experts in Data and AI

The ambition behind the National Data Library

Although we do not know enough yet about the features of the National Data Library (NDL), we can expect the government to want to extend access to public sector data beyond researchers. If the NDL went for its most ambitious applications, it would need to make 'AI-ready' data available to both public and private sector organisations, including for some commercial uses.

What we have learned from previous efforts

A recent evaluation of an existing data platform has shown that investing in data linkage across public sources can deliver £5 of benefits for every £1 of costs.¹ At the same time, the level of investment in data platforms across the public sector has also been substantial. Baringa’s own analysis shows that the government has spent at least £320m during the past 5 years across its two flagship data platforms², in addition to over £180m spent on health platforms such as the NHS Secure Data Environments and OpenSafely³, and £330m set aside for the NHS’s Federated Data Platform over seven years.⁴ While these are wise investments that can translate into real benefits for the UK, there needs to be a more cohesive approach to tracking and communicating the benefits of this investment at a time of pressure on public spending.

Broadening the NDL use cases beyond research is likely to yield similar, if not higher, return on investment. Yet past attempts to broaden the application beyond academic research have met with a few obstacles:

The Government’s own assessment of past efforts concluded that research and operational use cases will need multiple platforms and products⁵, going beyond the current model of pre-linked, anonymised data assets accessed in a secure research environment.
The Digital Economy Act (DEA) has enabled more than one hundred data sharing pilots across over eighty public sector bodies, creating over £135m worth of savings to taxpayers.⁶ Yet, risk aversion and different maturity levels remain significant barriers.
Public acceptability of data use by government remains an issue. However, a survey of 29,000 UK residents found widespread support to share personal data for research, and even some support for sharing data with profit-making companies involved in developing medical treatments.⁷

How to make the NDL a success

If done right, the NDL can lay the foundation of the UK’s data infrastructure for decades to come. The size, complexity, and multi-stakeholder nature of this ambition, coupled with the intangible nature of its benefits, can make such a programme daunting. For this reason, it is important to set up the right construct for the NDL from the onset. Baringa has partnered with various public sector bodies to help implement digital and data transformation, addressing complex programme, technology, and procurement challenges. It is this experience that has helped us distil these critical success factors for the NDL.

Focusing data curation on government missions

Organising data curation around the government’s missions to create deeper political momentum behind this initiative. Baringa’s own analysis shows that health data dominates existing HM Government’s data platforms. Expanding data availability to domains like public safety, access to opportunity, environment, and energy transition should be a priority for future data curation efforts.

Administrative data

Driving cross-departmental data collaboration

Empowering the existing data ecosystem to curate more cross-departmental data assets. Getting government departments to shape the future of the NDL is a key success factor, as domain expertise can accelerate data sharing and the creation of linked data assets. This is because departments who are already collaborating on policy delivery, are more likely to have the level of mutual trust and working relationships to facilitate joint analytical work. The NDL can function as a catalyst for cross-organisational data sharing, through funding initiatives that can replicate the success of Better Outcomes through Linked Data (BOLD)⁸ led by Ministry of Justice.

Rethinking the NDL architecture

Exploring architecture blueprints that go beyond the classic secure research environment. Contributors to the Wellcome Trust’s recent NDL Technical White Paper Challenge⁹ have highlighted the need to move beyond a single monolithic platform, where a central team makes data available to vetted researchers. Federated architecture and distributed architecture, for instance, offer decentralization, security, privacy and scalability advantages, while offering different governance mechanisms around autonomy, coordination, and control.

Embedding flexibility in how it's procured

Embedding flexibility in how the NDL is procured. The recently published State of Digital Government Review¹⁰ calls for a step change in how government approaches procurement of complex digital programmes, with a focus on flexibility and agility to adapt to the changing needs and technological advancements. Some of the use cases for the NDL will inevitably push the boundaries of data platforms in terms of complexity, scale, and public acceptability.

The NDL will also rely on the broader ecosystem of public sector data platforms. For this reason, procurement should reflect these risks by focusing on standardisation and interoperability in technology procurement to ensure that different systems can work together seamlessly. This is crucial for creating a cohesive digital infrastructure across the public sector. At Baringa we have worked on these challenges with large, public sector digital and data platform programmes.¹¹

Making discovery sharp and focussed

Making discovery sharp and focussed. Given the extensive user research that has taken place already, we recommend that the next discovery phase focus on key pain points already identified such as how to uplift data sharing maturity and how to leverage innovative business and technical architecture for data platforms.

Next steps…

Treat the NDL as an opportunity to build sustainable public sector data infrastructure.
Continue engagement with the public: especially communicate more about the benefits to individuals, society, and the economy.
Focus on where the NDL can make a difference – even if it means sacrificing breadth of use cases in favour of impact.

How can we help?

At Baringa we worked with organisations across all sectors, including government, to build scalable and sustainable data platforms and programmes. To find out more about our work in this space, you can get in touch with our experts, or read about how we have supported NHS England to build their data backbone to unlock better services and care for people.

^{1. https://www.adruk.org/news-publications/news-blogs/evaluation-finds-adr-uk-is-a-beneficial-data-linkage-investment-returning-significant-value-to-government-and-researchers/}
^{2. This includes £240m for the Integrated Data Service (IDS) for the period 2020-21 to 2024-25 (source: https://www.ons.gov.uk/aboutus/transparencyandgovernance/freedomofinformationfoi/committedinvestmentandspendfortheintegrateddataprogramme#:~:text=For%20the%20financial%20years%202020,is%20approximately%20%C2%A355%20million) and £84m for Administrative Research UK (ADR – UK) for the period 2021/2022 – FY2024/2025 (source Interim_evaluation_of_ADR_UK_-_Summary_report__Nov_2024.pdf)}
^{3. https://www.hdruk.ac.uk/helping-with-health-data/the-sudlow-review/}
^{4. https://www.england.nhs.uk/digitaltechnology/nhs-federated-data-platform/fdp-faqs/}
^{5. https://www.gov.uk/service-standard-reports/data-marketplace-alpha-assessment#what-the-team-has-done-well}
^{6. https://www.gov.uk/government/news/new-data-sharing-powers-save-taxpayers-137-million-since-introduction}
^{7. https://understandingpatientdata.org.uk/sites/default/files/2024-09/Public%20Attitudes%202021-2024.pdf}
^{8. https://www.gov.uk/government/publications/ministry-of-justice-better-outcomes-through-linked-data-bold/ministry-of-justice-better-outcomes-through-linked-data-bold#data-and-analysis-pilot-projects}
^{9. https://wellcome.org/what-we-do/our-work/uk-data-library}
^{10. https://www.gov.uk/government/publications/state-of-digital-government-review}
^{11. https://www.baringa.com/en/impact/case-studies/nhs-england-technology-procurement/}

Our Experts

Adam Brocklesby

Partner, expert in Government and Public Sector

Contact Adam

Roberto Pitea

Expert in Data, Analytics and AI

Contact Roberto

Steve Shelton

Partner, expert in Data, Analytics and AI

Contact Steve

Related Insights

Catherine Wright

Partner, Government and Public Sector

Team of health professionals in a meeting

Delivering NHS England’s most complex non-clinical technology procurement ever

Supporting NHS England build the data backbone that will unlock better health services and care for people in England for years to come.

Learning from crisis: How to embed mission-led government

Now that the budget has set the ‘what’ of government, we encourage focus to shift to the ‘how’.

New horizons: The case for public sector drone investment

Baringa research shows drones can deliver impressive cost, time, and environmental savings in public services. Download our report to find our more.

Is digital and AI delivering what your business needs?

Digital and AI can solve your toughest challenges and elevate your business performance. But success isn’t always straightforward. Where can you unlock opportunity? And what does it take to set the foundation for lasting success?

Find out more

Previous Insights

Featured

Synchronisation of the Baltic power system with Continental Europe opens exciting opportunities for batteries

Featured

How to unlock the multi-billion dollar e-waste opportunity

Featured

The rise of the COO podcast

The National Data Library: building the foundations of the UK’s data infrastructure

The ambition behind the National Data Library

What we have learned from previous efforts

Next steps…

How can we help?

Our Experts

Related Insights