In this interview, Pádraig Dalton, former Director General of the Central Statistics Office of Ireland, reflects on leading the shift from an organisation producing statistics to one providing insights and data services across the system. Having consulted with the National Institute of Statistics Rwanda (NISR) under the Rwanda Economy Digitalisation Programme, he provides advice as the NISR embarks on a similar path.

With the evolution of big data and AI, how do you see the role of traditional national statistical offices changing?

The evolution started some time ago, before major developments in AI. Originally, statistics institutes were the only ones compiling data, but there is now a competitive data market.

One of the challenges facing decision-makers is that the private data market is largely unregulated. When reviewing data from private providers it can be hard to find details of the underpinning methodology or the extent to which they have accounted for any bias, etc. Furthermore, data may not be comparable over time, it may be a one-off, and it may not be compiled in accordance with international standards and therefore not be comparable across countries.

Many national statistical institutes are increasingly using all available data sources to compile information, going beyond primary data to using more comprehensive administrative, publicly available sources (e.g. via web-scraping), and in some cases private data sources where access is provided.

What are the successes the Central Statistics Office (CSO) of Ireland had, and mistakes made in transitioning from a traditional stats agency to one that embraced broader data use?

The CSO started its journey of a shift towards administrative data as far back as the early 2000s, but it is perhaps only in the past 10 years that real progress became evident. We were compiling data from primary data sources (surveys and census data) but we knew we wouldn’t be able to meet the demand for more frequent, timely and granular data through primary data sources only. It was also clear to us that if we couldn’t meet the needs of decision-makers, they would go elsewhere to source their information needs. In the absence of information, noise tends to fill the vacuum.

We moved from an organisation that produced stats to one providing insights and evidence, ultimately providing data services across the system. A stats agency needs data to be organised and structured in a particular way to support the evolution from compiling statistics to providing evidence and insight. This particular change also required a move from dissemination to strategic communication. Today, the CSO extracts key messages and stories from the data and presents them in different formats tailored for different categories of users. These could be infographics, visualisations, graphics, YouTube segments, press releases or sound bites.

Originally, we were using our language – we used to ask government departments what their data needs were. But this was not an obvious question for them; we should have been asking what their policy challenges were. We could then work out the best source of information to answer those questions.

The shift towards secondary data sources also impacts on structure and skill sets. Beyond evidence and insight, we also shifted to an organisation that was providing data services across the broader civil and public service in Ireland. All of this represented a modernisation or perhaps transformation across the Central Statistics Office in Ireland, and this type of cultural change can often be met with an element of resistance from within the organisation, and that was the case in Ireland.

Identifying the need for change, or the burning platform as it was often referred to in Ireland, was a key first step. We also needed to build a picture of what the future could and would look like for all our stakeholders (e.g. staff of the CSO, our user community, respondents) and create a coalition of the willing.

Initially, we worked with two willing departments. We helped them to improve the quality of their data and implemented a quality management framework, which we supported them to implement. We were able to integrate multiple data sources and link them with our own sources of data and produce new insights. Providing tangible examples and focussing on the “art of the possible” was an essential component in bringing the vision to life.

Being able to create new analysis is one thing, but extracting the insights and presenting them in an accessible and digestible manner to all user groups, which required bespoke products for different user groups, is another thing. We recruited an expert in comms to communicate information rather than just disseminate data.

Once these new insights were published, other departments took notice. They hadn’t realised they could talk to the CSO and that we could be directly helpful to them. Thereafter government departments started coming to us in droves, asking us to help them resolve their policy challenges.

The problem is, if you create an expectation, you have to meet that expectation, and it then becomes a resource challenge. It took four or five years after establishing the vision before we saw real progress. Positive, sustainable change takes time and you need to be resolute and resilient.

What organisational or skills changes were made to enable this transition for the CSO?

We knew we needed to change our organisational structure, people and skills to achieve the shift from collecting and analysing data to producing usable insights, and providing data services. We recruited a Lean Six Sigma expert and reviewed all our internal activities and processes, which helped us understand how data moves through the data life cycle.

Three new teams were created: Methodology, Data Quality and Communications. They provided services not just for the CSO, but services to the whole system. We also created an Administrative Data Centre (ADC), which is, in a sense, the “clearing-house” for all administrative data holdings in the CSO and which handles all data governance and data protection issues. We surrounded the ADC team with a number of analytically focussed divisions that work almost exclusively with government bodies providing evidence and insights around their policy challenges.

To succeed, you cannot just produce documentation, you need to get out there and work with the government departments. We embedded professional staff across 16 departments, essentially outsourcing our expertise through five-year staff deployments. Telling people what to do doesn’t work, you need to work alongside them to build trust.

The CSO also has a “hit team” – a floating team of statisticians for short-term problem resolution.

While providing evidence and insight to policymakers it is necessary to be clear about boundaries and the CSO never comments on policy. Its role is to compile the facts and it is for others to comment on their implications for policy.

The CSO’s commitment to confidentiality is absolute and none of the analysis compiled can be related to an individual person or company. Confidentiality is not just enshrined in international principles such as the United Nations Fundamental Principles of Official Statistics but it is also enshrined in legislation.

The change that occurred in the CSO cannot be attributed to any one thing or approach, but a clear vision and our focus on progress rather than perfection ensured we kept moving forward.

Could you comment on behaviour change? For example, did you experience internal pushback during the transition?

If you’re not getting pushback, you’re not leading. There was pushback from a small cohort of what some might term statistical purists within the CSO. There were those that accused us of “dumbing-down” official statistics, with some people suggesting they didn’t like the infographics and data visualisations and arguing that if people did not understand the data they probably should not have been using it!

Fortunately, most people saw the need and value in the proposed change and were willing to embrace the challenges that lay ahead. We spent a lot of time doing townhall sessions within the CSO to communicate the vision. You need to understand the drivers of change but also be clear about what happens when you don’t change.

Government departments were on occasion reluctant to engage for a variety of reasons. Sometimes change requires a shift in technology or systems and they did not have sufficient budget. There were perhaps also fears around the risk of exposing their data to the CSO. We emphasised that we weren’t expecting perfection and were not there to audit individual departments. We committed to work with them to help improve their data quality and policy decisions, essentially working on a “no fault” basis. We seconded staff to their organisations, which demonstrated our commitment but we did not talk publicly about the problems we found regarding data quality.

Were there specific use cases that demonstrated the value of merged datasets or proved unexpectedly interesting?

There are so many across housing, education, childcare and the fluidity of the labour market (movement from one sector to another), etc. We integrated education data, revenue and social protection data sources and gained a lot of new insights. For example, many doctors and nurses would leave Ireland for a period of time, very often just after they qualified and there was a need for insight around how long they stayed away and what proportion of them returned.

We explored recidivism in the justice sector and the outcomes for people who spend time in prison, where they get jobs, their average income and their ultimate educational attainment level, etc. Once you combine different sources of data, you quickly start seeing the potential insights within the data.

Ireland undertakes a Census of Population every five years. One of the CSO’s new products is called “Irish Population Estimates from Administrative Data Sources (IPEADS)” and in effect, it now produces census-type information annually by linking up to 19 different data sources from across 12 to 14 data holdings in government bodies. This is a high-quality product and the geographic granularity in analysis is improving all of the time.

Were the data owners encouraged or compelled to share data with the CSO?

In Ireland, an act makes provision for mandatory data sharing with the CSO, which has the legal right of access to all administrative data. There are exceptions for medical records (although we do know which prescription drugs people are on) and criminal records. However, because there were some public trust issues regarding police data on crime, the Minister of Justice shared relevant data directly with the CSO to enable independent analysis, and the CSO now compiles the recorded crime statistics.

Any government body that proposes to “introduce, revise or extend any system for the storage and retrieval of information or to make a statistical survey shall consult with the Director General and accept any recommendations that he may reasonably make in relation to the proposal”. The act was passed in Ireland in 1993, before data stewardship was even a thing. The legal framework in Rwanda [regarding the statistics agency and its mandate] may need to be revised to position it to provide the necessary evidence and insight required by decision-makers and the people of Rwanda.

If you were responsible for implementing and sustaining the data sharing policy in Rwanda, where would you start?   

In Rwanda, I sensed an openness to change and a willingness to change. It struck me how devoted people were to the good of the country and its people, and how many of those I met were really dedicated to public service. The data sharing policy is for the system and not just the NISR. If one Government of Rwanda (GoR) entity provides a data file to another GoR body, and that entity is not able to use it, then the potential of data sharing can’t be fully realised.

We need to talk about data sharing, linkage and integration as one piece.

The usage of identifiers and data infrastructure is important, as these are the elements that support linkage and integration, which is the sweet spot. The focus should be on creating an environment where data can be shared because the standards are in place. The systematic implementation of common data standards is required to realise the potential of the available data sources.