SCORE Methodology

Background: The use of remote telecoms surveys to consult with crisis-affected people

Humanitarian actors understand in principle that to better understand and respond to the needs of crisis-affected people, they need to engage with and listen to them. Meaningful accountability to affected populations (AAP) requires more than establishing a complaint hotline or running the occasional focus group; it means consulting with communities before, during, and after any interventions.  Unfortunately, however, most humanitarian actors have yet to develop any systematic, statistically robust means for obtaining the perspectives of those on the receiving end of aid. This gap means that there is a lack of much-needed qualitative information on the quality and reach of humanitarian aid from its primary users. The increasing penetration of mobile telecommunications technology into the developing world over the past two decades has made remote surveying via cell phones a valid and viable tool to use for AAP and humanitarian research alike.

Humanitarian Outcomes first used remote surveys to inform and triangulate our research on aid operations and policy in 2010 (for the State of the Humanitarian System reports), and we have employed this tool many times since, partnering with various international survey providers.

Despite telephone surveys being the principal tool for political polling and other opinion surveys in the developed world, the idea of using them in crisis contexts in the developing world has met with resistance by some in the aid sector more accustomed to in-person survey methodologies, e.g. household surveys. Humanitarian Outcomes has also used in-person surveys, alone and in combination with telecoms surveys. While each modality has its strengths and weaknesses, remote telecoms surveying is preferable in many conflict cases where the humanitarian presence is already constrained because of insecurity or other access obstacles. The ability to reach large numbers of people across vast areas in a fraction of the time required for in-person surveys makes it possible to capture public perceptions more widely and give voice to people who otherwise wouldn’t be heard. Because of the numbers and speed of sampling that the technology allows, it is also easier to achieve a balance in gender and other demographic categories in responses while maintaining randomization.

The SCORE survey series

To gather evidence on humanitarian coverage, operational reach and effectiveness in hard-to-access settings, our surveys asked inhabitants about how they have accessed vital services and which, if any, humanitarian providers had effectively reached them with assistance. See SCORE questionnaire here. Partnering with GeoPoll, the largest mobile surveying provider across the developing world, Humanitarian Outcomes fielded these surveys in Afghanistan, Northeast Nigeria, and Central African Republic in 2019-2020.

GeoPoll can collect random and significant samples of respondents through their partnerships with the major mobile network operators in countries as well as their existing database of over 320 million mobile phone numbers. Respondents are anonymous and their mobile numbers are never shared or attached to their response data (GeoPoll signs a unique ID number instead.)

Remote telecoms surveys can be conducted via:

  • SMS – Short message service (texting). This is the lowest cost option and works well in populations where the literacy rate is high.
  • IVR – Interactive voice response. With IVR, the respondent receives a recorded call prompting him or her to answer questions by pressing buttons on their phone.
  • CATI – Computer-assisted telephone interviewing. CATI surveys use live enumerators who take the respondents through the questionnaire, recording their answers. The additional benefit of this modality is that it can provide greater clarity (e.g. if the respondent does not completely understand the question and allows for some open-ended answers that can lend more detail to the research.

To date, the SCORE project has used CATI for all contexts. With the questionnaire translated into the relevant languages, GeoPoll’s enumerators were trained on the script and conducted the calls from regional call centers.

We targeted the surveys to specific provinces in each country where humanitarian need was high and access constrained. Calls were pushed out to mobile numbers in those areas on a random basis and continued until the sample achieved a satisfactory gender balance. Sample sizes for each country were slightly over the minimum threshold required for statistical significance at a confidence level of 95%.

Respondents were incentivized to participate in the surveys by an offer to receive a small amount in mobile credit.