American author and astute observer of the human condition, Samuel Clemens, once noted that there are “lies, damned lies, and statistics.” He must have had social analyses of North Korea in mind.
Two recent articles by otherwise well-respected journals that cover North Korea require closer examination.
The article What Hostile Policy? (https://beyondparallel.csis.org/hostile-policy-north-korean-views-united-states/ ) published the results of a “micro-survey” commissioned in 2017 by Beyond Parallel in partnership with the Center for Strategic and International Studies (CSIS). It involved 50 respondents living in North Korea regarding the question of whether they saw the United States as North Korea’s enemy.
The authors state that “68% of North Korean respondents do not see the United States as North Korea’s enemy.” The article readily admits – though only toward the end of the piece – that “The sampling method we used was non-probability, convenience sampling as accessibility was a prime consideration.”
This means that the authors knew full well that the subjects chosen did not meet proper survey conditions and that their data were not statistically representative, yet they presented the findings as if they were. Moreover, the authors proclaimed that those findings could even be used in negotiations with Pyongyang as factual evidence.
Perhaps not surprisingly, these 2017 findings are in conflict with the results of a survey conducted by Seoul National University in 2016 from a pool of defectors living in South Korea in which 84% of 138 respondents stated that the United States was indeed the biggest threat to peace on the Koran Peninsula. Although the surveys phrased the question slightly differently, the results are easily generalizable.
Representation, Margin of Error, and Honesty
There are a number of problems with conducting such surveys. First, for any sample to be of value, the subjects must have been randomly selected. In the first case, the subjects selected themselves: they were the ones willing to speak with the interviewer about sensitive topics in North Korea. That requires trust, and one tends to trust only those who are thought to share a common outlook on life.
Second, the sample size is abysmally low. Even when done properly (more on that below), the margin of error for a sample size of only 50 subjects is plus or minus 14%. In other words, with a margin of error that spans 28 percentage points, a finding of, say, 50% of respondents favoring one thing over another could actually range from as low as 36% to as much as 64% – almost twice as much.
In comparison, with a sample size of 138 done properly, the margin of error is roughly ±9%, still not a stellar error range. This is why most professionally administered surveys use stratification and try for a sample size of close to 1,100 participants in order to reduce the margin of error to ±3%.
Third, there is the question of whether the subjects were reciting answers they thought the interviewer wanted – situational demand, as the phenomenon is professionally known – or whether they were truly being candid. When a wrong answer can mean the difference between punishment or reward, the incentive to give questioners what they want is high.
Science versus Wishful Thinking
Another article entitled “History and Identity” (https://sinonk.com/2018/07/02/yanbian-chosunjok-identity/ ) on the Sino-NK website reports on a survey administered in 2016 and 2017 that investigated how ethnic Koreans living in the Yanbian Autonomous Prefecture of northeast China felt towards South Korea and North Korea.
Unfortunately, the same set of problems that were rife in the Beyond Parallel/CSIS survey are also present in the Sino-NK effort. To begin, the findings come from only 43 respondents, fewer still, and with a commensurately higher margin of error.
To be sure, the findings of both studies seem intuitively accurate. However, intuition is not science and one cannot – rather, ought not – base critical decisions on intuition. In this case, one could make the case that such findings are nothing more than wishful thinking.
Admittedly, getting good information about North Korea – particularly on dangerous or sensitive topics – is extremely difficult. Regardless, publishing data of questionable integrity is highly unprofessional. Doing so because it is the only data available is no excuse.
Other problems with these surveys exist. When the general population to be surveyed is known or suspected to have significant subsets that could reasonably be expected to vary in their responses to survey questions, then it is appropriate to identify those subsets and sample them accordingly so that the results of the sampling are reliable across all segments of the population. This is known as stratification of sampling.
Some known population segments in North Korea from which one would expect differing responses to questions include, but are not necessarily limited to:
Pyongyang’s ultra-elites versus lesser elites living outside Pyongyang
Senior party members versus rank-and-file party members
Senior military members versus rank-and-file-military members
Urban factory workers versus rural farmers
The donju, the new merchant class, versus non-entrepreneurs
Some of these segments overlap but with proper stratification, results that are much more representative of each group would be obtained.
It is also worth noting that responses to surveys most likely will vary from region to region. Respondents close to the DMZ, those near special enterprise zones, the so-called hostile class consigned to the far northeastern part of the country, or citizens near the border with China or Russia are likely have divergent perspectives on a number of economic or political issues.
The point here is that, while such surveys may provide some inkling as to the state of affairs in North Korea, using data known to be unprofessionally gathered or otherwise unreliable, is a grave disservice to all North Korea watchers. The results from these efforts are more heuristic than anything else and they should be treated as such.
While we all want more and better information on just about every aspect of North Korea, we cannot pass off questionable science as being reliable. These two reports are a striking case of caveat lector.
Image: Participants wave flowers as they march past a balcony from where North Korea’s leader Kim Jong-un watched a parade marking the 70th anniversary of North Korea’s founding. Photograph: Ed Jones/AFP/Getty Images, September 2018