Innovating for sustainable data collection: Generating ‘gold standard’ survey data – for 75% less cost

Innovating for sustainable data collection: Generating ‘gold standard’ survey data – for 75% less cost

26 February, 2020    

Traditional face-to-face surveys are regarded as the gold standard for financial inclusion surveys, as they use probability sampling methods and trained interviewers to administer the surveys to people in their homes. However, sending surveyors out into rural and excluded communities is an expensive proposition. A typical face-to-face survey can be as much as 75% more expensive than a mixed-mode methodology that uses more cost-effective data collection modes, such as SMS (text messaging) and computer-assisted telephone interviewing (CATI). The other major limitation of traditional face-to-face surveys is that they are extremely long and result in cognitive fatigue, which reduces the quality of data being collected.

That’s why the insight2impact facility set out to find a more sustainable approach to acquiring limited but high-quality financial inclusion data. We approached that task with the goal of generating insights into people’s behaviour and perceptions that were just as accurate and relevant as the data acquired by traditional face-to-face surveys – and to do so at a much lower cost. (This project is just one element of insight2impact’s work, which aims to positively impact the entire data landscape to improve financial inclusion outcomes.)

Generating accurate financial inclusion data at a lower cost

To reduce data collection costs, we had to consider various technological modes for delivering the survey: We eventually decided on SMS as our primary digital mode, as it allowed us to achieve the widest reach in the populations we were interested in. We had pre-piloted standalone SMS surveys in 2017 and knew that they resulted in skewed estimates, so we needed to fine-tune our approach for our second and third rounds of piloting. In 2018 we decided that a mixed-mode data collection strategy with statistical modeling to correct for sampling and mode bias would be required to produce estimates that were credible.

We set off and started to collect data via SMS and CATI in eight countries. As the face-to-face Financial Inclusion Insights survey datasets were available in all eight countries, we did not have to collect additional face-to-face data: We used these datasets as our reference survey against which we would evaluate our results. In fact, any large-scale face-to-face survey, such as FinMark Trust’s FinScope, could have been used for this comparison – but we decided on the Financial Inclusion Insights surveys, as they also contained the digital financial service indicators that we were interested in. In the end, we had three datasets for each of the eight countries (FTF, SMS and CATI). To make a long story short, we were able to show that by using a small sample of face-to-face data, then combining that with the SMS data and using multilevel regression and post stratification (MRP) to model the results, we were able to calculate estimates for the digital financial service indicators that were comparable to the full indicators in the Financial Inclusion Insights gold standard reference survey. This approach also reduced the cost of collecting these indicators by almost 75% – a major success for us.

Lessons learned for sustainable data collection

The most important thing we learned in this work is a lesson that’s applicable to anyone who’s trying to innovate: Listen to the nay-sayers and then ignore them! We were told many times by fellow researchers that our approach wouldn’t work, due to the inherent inaccuracies the data collection mode would produce – not to mention the difference expected due to mode effects. But our view was that even if we failed, we would still have advanced our thinking around these new methodologies. If we had only tried to incrementally improve on methods that had been proven to work in the past, we’d never generate new knowledge.

The other key lesson we took from the pilots is that it’s essential for innovators to surround themselves with a quality team. Innovation doesn’t happen in a vacuum, with one person dreaming up a creative solution all alone. Our innovations developed organically, as we grappled as a group with the various technical challenges we encountered throughout the three year-long process. We were lucky to have great donor programme managers, as well as a global network of partners with the specialist skills we needed.

An important objective for the programme was to get our new measurement frameworks and data collection innovations adopted, so we would advocate for them in various forums. The most difficult thing we encountered in those efforts was resistance from fellow researchers when we asked them to innovate their measurement frameworks or to consider alternative data collection strategies. The resistance typically centered around a reluctance to lose the ability to make direct comparisons to previously collected data, to remove questions from surveys, or to accept that alternative data sources or shorter surveys might produce better quality data. In addressing this reluctance, we found that if you collaborate actively and practically with researchers from different organizations on an actual survey or research problem, you are more likely to achieve success – as opposed to just standing at a pulpit and preaching about how things should be done differently. Advocacy will only get you so far. It’s true that this practical collaborative approach is time-consuming and resource-intensive, but the result was well worth it for us and the organisations to which we introduced the innovations.

Developing a 360-degree view of financial lives

Our data team’s view has always been that you should match the research problem to the most appropriate data source, whether that’s survey data, supply-side data, transactional data or alternative data sources like social media. We now emphasize the importance of a data ecosystem, where all types of data and research methods have a role to play in building a picture of financial inclusion. To develop a rich picture of how people live their financial lives, we needed a 360-degree view that can only be attained by consulting different data sources and using multiple research methods.

Indeed, we’re not suggesting that large scale face-to-face data collection should be phased out. Our contention is that comprehensive demand-side surveys will remain very important for diagnostic purposes: After all, the focus of this data is on the poor, who often have no digital footprint – so the only way to make them visible is to collect survey data. However, a key learning from this project is that researchers should not be afraid to use new data sources and research methods, even if they are not currently perfect. No data is ever perfect – you just need to define the boundaries of that data to understand to what extent it can be generalised.

We still have quite a bit of work to do: For starters, we need to write up our findings for the eight-country digital financial services mixed-mode pilots in a digestible form. We are also planning to collaborate with partners to produce an academically publishable article on our approach. With this kind of work, it is always valuable to have it pass academic scrutiny. We had a panel of four academics that reviewed our work throughout the process. We even had Andrew Gelman, the modern-day father of MRP, join in our final review session – and to our delight, he was impressed with our work.

We have another round of piloting running now, which explores the use of mixed-mode data collection and MRP to collect gender-specific financial inclusion data in Kenya, Tanzania, Uganda and Pakistan. There is a huge gap in the availability of good gender data, and we believe our low-cost mixed-mode method will help close some of this gap.

Going forward, a key focus for us will be to disseminate the work as widely as possible during the final months of the programme. This process is off to a good start, as we have already had a development partner ask us to design a mixed mode survey to measure headline financial inclusion indicators in several countries. We believe that this promising reception demonstrates the demand for new, innovative data collection methods that generate robust results in a sustainable manner. This advancement could have a substantial impact not only on the gathering of financial inclusion data, but also in collecting Sustainable Development Goal data, and data that speaks to the intersection of financial systems and the real economy.

Grant Robertson is insight2impact’s Head of Survey & GIS Data for Financial Inclusion (Data4FI). 


This article was first published on Next Billion as part of the Openi2i series an initiative of Cenfri and Finmark Trust

Similar Articles
Data training resources for independent learners
Designed for people who want broad exposure to a spectrum of data science and data engineering content, these data training resources are available...
Digitisation of government services in Rwanda: Lessons from the data
Rwanda has set ambitious goals to transition from an agrarian economy to a knowledge-based one by ...
Unlocking growth: 3 key opportunities for remittance service providers
Many remittance service providers (RSPs) face a tough balancing act; they’re trying to innovate and grow, but it feels like they’re driving wit...
Leveraging agricultural data for more effective policymaking
Policymakers are challenged to develop policies that will have a positive social impact. For instance, input subsidies for farmers have been widely...
Supporting responsible financial sector innovation in Jordan
Cenfri, in partnership with the GIZ I-FIN project, is providing technical assistance to the Central Bank of Jordan (CBJ)...
R3Lab Toolkit: Regulating and enabling innovation
Innovation can enable insurance to become a powerful engine of growth and development across the continent, but achieving this requires that market...
Open Finance in South Africa: Promising moves, but challenges ahead
The South African Financial Sector Conduct Authority (FSCA’s) publication of their ...
The Remittance Innovation Toolkit: Guidance on improving access to remittances
There’s no disputing the contribution of migrant remittances to the economic well-being of friends and family back hom...
Rwanda Economy Digitalisation Programme launches second phase
Kigali, 08 May 2024 – The Ministry of ICT and Innovation, the Mastercard Foundation and Cen...
A new approach to tackling deforestation
Over the past three years, the Global Stocktake (GST) embarked on a consultative journey to assess progress towards the ...
Navigating a career in data science
There is demand for skilled professionals in the field of data science as organisations are increasingly adopting a data-driven approach. From unco...
Building the fundamentals of a welfare-enhancing digitalised economy
Digitalisation per se has marginal impact; the crux is the way it is leveraged and implemented. We need a better understanding of how digi...
The Africa Fintech Hub – a platform designed to strengthen fintechs in Africa
Fintechs have the potential to significantly improve the options available to financially underserved individuals. Many fintech start-ups in Africa...
Supporting the development of a digital finance index
Digital financial services are transforming global financial service provision and access. Rapid developments in fintech are disrupting and transfo...
The untold realities of women cross-border traders
A diagnostic study on women cross-border traders between South Africa and Lesotho, Malawi and Mozambique​ ...
Guidance on developing a data and analytics strategy
In today’s data-driven world, many organisations recognise the importance of leveraging data insights to make decisions. However, once they attem...
Unique challenges and opportunities for Earth Observation in Rwanda
When I travelled the windy road from Kigali to the Volcanoes National Park to hike up Mount Bisoke, I spent most of my time looking out the window ...
Case study: Enhancing telecommunications connectivity through data analytics
The Government of Rwanda’s ambitious plan to become a cashless economy by 2024 has seen significant efforts and investments in the cashless a...
Then and now: Is Ethiopia at an inflection point after 15 years of insurance underdevelopment?
Very little has changed in the Ethiopian insurance industry over the last fifteen years but there is a po...
Digitising destinations: a tourist’s journey starts online
COVID-19 turned some of us into virtual voyagers. We spent hours scrolling through posts of p...
Open Finance in Africa: Designing context-appropriate approaches for the financial sector
Open Finance can be defined as the sharing of consumer data between financial service providers (FSPs) and/or third-party providers on the basis of...
State of inclusive instant payment systems in Africa
  Real-time retail payments that enable consumers to send and receive cross-border and domestic transactions digitally are on the rise ...
Reflections on the policy evolution in Rwanda’s ICT sector
In implementing the ...
Remittances, a vital contributor to the SDGs
Remittances are a valuable lifeline for Africans, with over USD 100 billion flowing into the continent annually from a m...
Impact from the Remittance Access Initiative
KYC and CDD requirements can inhibit access to remittances for low-income, r...
Catalysing the use of data and digital technology for tourism in Ghana
Ghana’s tourism industry presents an exciting landscape for growth, ...
Mutual digital infrastructure could drive formalised, inclusive payments in SADC
Interoperable payment systems improve the efficiency and effectiveness of retail and cross-border payments. Cenfri worked with BankservAfrica on a ...
Public sector data frameworks
If properly catalogued, classif...
Developing capacity to boost data-driven decision-making in Rwanda
In recent years, Rwanda has embarked on a transformative journey towards digitalisation to fuel economic growth and innovation. The ...
An assessment of customer due diligence and identity regulatory frameworks
Innovation opportunities for enhanced remittance access...
Understanding CBDC and its application in emerging markets
Central banks have cited several reasons for considering or piloting central bank digital currencies (CBDC). These inclu...
5 lessons for designing fit-for-purpose health microinsurance
Gakere is a 49-year-old mechanic from Nairobi, and like ...
Revealing the consumer experience of digital finance in low- and middle-income countries
Consumers face several frustrations with the potential to affect their experience of digital financial services. These include system glitches, ina...
Insurance regulation and innovation: An assessment of 8 African insurance markets
To support regulatory authorities in sub-Saharan Africa in promoting inclusive insurance market development, there is a ...
Consumer protection in digital finance
The authors of this report would like to acknowledge the support of the International Dialogue on Consumer Protection...
Data insights in achieving retail growth in Rwanda
The retail and trade (merchant) sector is one of the priority sectors for the ...
Enhancing identity verification for refugees in Uganda
In Uganda, refugees and asylum seekers struggle to cash out their remittances due to challenging and time-consuming identity verification processes...
Training improves ability to unlock value from data
Capacity-development is a vital part of the ...
What will it take to build innovative insurance markets in Sub-Saharan Africa?
Despite the best efforts of sub-Saharan African (SSA) insurers, most African citizens ...
Lessons on improving remittance access in Africa
Learnings from the ...
Agriculture an important policy priority for Rwanda
Agriculture plays a major role in the Rwandan economy: close to 75% of the population’s livelihood depends on farming. However, the majority of t...
Cenfri joins the Smart Africa Alliance
Kigali, Monday 29th August 2022  – The ...
Responsive and responsible regulation
The value of non-legally binding guidance Regulation is ...
CBDC for emerging markets: realities and expectations
Cenfri has deepened its work on CBDC amid growing interest from central banks ...
What is data-driven decision making and what does it mean for Rwanda?
Nothing illustrates the incredible power of data-driven decision-making more than the recent COVID-19 pandemic. Lamenting the disparit...
Holistic risk solutions for MSME resilience in Egypt
Micro, small and medium-sized enterprises in Egypt Micro, small and medium-sized ent...
Building a data skills foundation is necessary to achieve meaningful digitalisation
One of the reasons so much emphasis is placed on digitalisation is...
Private sector partnerships for MSME resilience: Learnings from Africa
Opportunities await insurers and tech providers who are willing to work together ...
Insurance Innovation Dialogue
About the Insurance Innovation Dialogue  To support insurance regulatory authorities ...
The role of different partners for effective MSME insurance
Insurers can partner with digital platforms, lenders and insurance aggregators to offer better products that...
Delivering remittances at the last mile
The role of...
Applying a gender lens to trade finance
Small and medium enterprises (SMEs) are considered engines of economic growth and development, especially in developing ...
Enhancing risk assessment processes for improved remittance access
Early insights from IFAD’s Financing Facility for Remittances and Cenfri Our work wi...
The role of women and finance in climate action
A changing climate affects everyone, but it’s the world’s poorest and most vulnerable, predominantly women and girls, who bear the brunt of env...
Opportunities for remittance-linked insurance products in SSA
A demand-side perspective of the risk management and resilience needs of remittance senders from South Africa to G...
Digitalizing farm to table supply chains in Asia
On-demand transport services and insurtech  Whether as a customer or...
Empowering people to solve problems that matter
How Cenfri makes systemic change happen In helping to build...
Africa’s digital future: Policy scenarios
What could Africa look like in 2048?  The digital revolution is gaining ground across...
Credit life insurance in South Africa: the customer’s perspective
Credit life insurance may be the first type of insurance that many low-income consumers encounter. It can offer an opportunity for introducing cons...