B. NCD Church Survey

1. Practical procedures

For the NCD Church Survey, 30 people per church are expected to fill in a questionnaire, either online or in paper form. These people have to fulfill the following criteria:

● First, each individual participant must regularly attend the worship service.

● Second, they must have some form of regular task in the church (not necessarily a leadership responsibility).

● Third, they must be part of some form of small group in the church (any subgroup of the worshiping community).

Why 30 people?

The Survey has been designed in such a way that, when strictly sticking to these criteria, it produces the most exact results. However, the number “30 persons” is not a static number. There can be exceptions in two directions:

● First, for churches smaller than 30 people (keeping in mind that in churches with a high nominal membership and a small worshipping congregation the worship attendance should be the point of reference), all members/attendees who fulfill the criteria fill in a questionnaire. If in a church with, say, 15 people, 12 of them fill in the questionnaire (in most cases there are a few that don’t fill in the questionnaire even if being asked), the survey will produce a very exact analysis of the church’s situation.

● Second, there are cases—particularly in larger churches—where the leadership wishes to include more than 30 people. Even if it has to be noted that such a procedure doesn’t produce more exact results, it is possible to enlarge the group beyond 30 people. The important thing is that everyone among that extended group must fulfill the three criteria mentioned. Passing on the survey to members that don’t fulfill the criteria inevitably leads to distorted results.

If in a given church there are more than 30 people people who fulfill the criteria, a random sample (for instance, being drawn by lot or by selecting every second or third person on a list of qualified candidates) should be selected. It is important that, within the group of people who fulfill the criteria, the actual participants in the survey are selected in a randomized way, and definitely not by self-selection (“We need 30 people to fill in the questionnaire. Who is willing to do that?”).

The survey is available both online and as paper version, in order to avoid that one’s access possibility or preference influences the kind of participants chosen.

Objective realities or mere perceptions?

There are two widespread misunderstandings about the items of the questionnaire. Therefore it must explicitly be stated what the items are not targeted at:

● First, they don’t strive to collect “objective facts” about the church.

● Second, they don’t strive to “evaluate the church” (in the sense of, “How would you rate the quality of our relationships (leadership, small groups, etc.) on a scale between 1 and 100?”).

Rather the items deal with the following two areas:

● First, personal and thus, highly subjective perceptions about the church.

● Second, statements about one’s own (rather than “the church’s”) attitudes and actions.

The NCD Survey does not measure any “objective” realities (in analogy to the size of a church building, the age of the pastor, the budget of the church, etc.); if questions of that kind should be asked, they are not part of the formula that calculates the quality index of the church. Rather, the items that are used to determine church quality (or “health/maturity” of an individual) deliberately focus on subjective perceptions of those who fill in the survey.

For example, when accessing the quality characteristic “inspiring worship service,” there are no objective features that would qualify a specific aspect as “inspiring” for all people attending. The very same thing that one person may experience as highly inspiring, may be rather irrelevant for someone else. A “correct” answer is represented by the subjective feelings of the individual who fills in the questionnaire (“Is that inspiring for me?”). This is the reason why the survey doesn’t give any definitions of some of the technical terms used (“What do you mean by “inspiring?”). The answer is, “Inspiring is that what is inspiring for you.”

From that perspective we can say that NCD doesn’t measure any “objective realities,” but it objectivizes different subjective perceptions of many different people and hence provides comparable results.

2. The necessity of an external criterion

It is important to understand that the individual items of the NCD Church Survey have not been chosen according to the criterion of what fits best into a pretermined scheme. Rather, only those items are included in the questionnaire that haven been proven to display a positive correlation with numerical growth that can be verified worldwide. Because of that, it is guaranteed that the eight “scales” (quality characteristics) measured, derived from a grouping of the items in eight categories, and the “quality index” (a score expressing the overall quality of the church) correlate with numerical growth in the sense of increasing worship attendance.

The “external criterion” of numerical growth is a hallmark of the NCD Survey, distinguishing it from other assessment tools that strive to capture “quality” or “health” of the church as well. Why is such an external criterion essential?

Avoiding circular reasoning

Without an external criterion there is the danger of circular reasoning, a phenomenon that many people developing (or working with) such tools don’t even recognize:

● They determine beforehand (maybe on the basis of their own biblical understanding) what quality is or should be and by which features it should be characterized.

● Then they develop a methodology that tests exactly the features that have been worked out beforehand, which is relatively easy to do.

● When using that tool for the assessment of churches, those that are most in line with the predetermined criteria receive high scores (“good churches”), while those that are not in line with these standards receive low scores.

● Finally, the scores of these churches are regarded as a confirmation (maybe even “proof”) that the pretermined criteria developed without any external criterion were the right ones, since all of the “good churches” among the churches researched live in line with them.

Such a procedure presents, of course, purely circular reasoning. In the end, you get out of the research exactly what you have put into it, without any chance for the “research” to break this logical circle. In order to avoid that trap, we opted for the external criterion of numerical growth.

Why numerical growth in worship attendance?

In this context, we didn’t opt for numerical growth because we would deem it the “ultimate goal” of church development. The major reasons for choosing “numerical growth” are the following:

● First, we had to look for a criterion outside of the qualities that we measured. Of course, we could have easily taken one of the quality characteristics (e.g., “loving relationships”) or one of the items on the questionnaire (e.g., “I experience the transforming influences faith has in the different areas of my life”) and could have looked for a positive correlation of all of the other items to those selected. However, by those correlations we would not avoid the trap of circular reasoning (“If the quality of the church is high, it has a high quality”). We had to opt for an external criterion, i.e., something that does not describe the quality/health of the church.

● Second, we had to opt for a criterion that is relatively easy and objective to measure. Theoretically, it could have been interesting to take, for instance, “impact on society” as an external criterion, based on the hypothesis that the internal quality of a church displays itself in the impact that it has on people outside of the church. However, that would have been almost impossible to measure given the fact of extremely different cultures, political settings, and justifiable ministry priorities. As far as worship attendance figures (not just the present ones, but the ones from the past five years in order to measure growth versus non-growth) are concerned, the numbers could be, with all inevitable inaccuracies, relatively easily obtained.

● Third, the external criterion must be comparable among churches in extremely different structures and traditions, and in each case relate to the same reality. That is the reason why “membership figures” did not qualify. There are churches with a huge nominal membership (say, 5,000 in a European state church setting) and 22 people in worship attendance. On the other hand, there may be churches with relatively low membership (say, 120 people) who attract a huge number of people in average worship attendance (say, 2,500 people), as can be studied in some non-Western churches. It is immediately apparent that the membership size in both cases relate to extremely different realities: In the first case it includes many people for whom church life doesn’t mean anything; in the second case membership indicates those that are at the very core of the church that manifests itself in large worship gatherings.

● Fourth, it should be a dynamic criterion (such as growth) rather than a mere static description of the present situation (such as size). Even if a church may have, say, 10,000 people in attendance, it would qualify as a “stagnating” rather than “growing church” if this level had been the same throughout the past three years. However, a church growing from, say, 30 people to 55 within three years, would qualify as a “quickly growing church.”

● Finally, it must be possible to select a comparison group (i.e., churches that don’t fulfill the defined external criterion, or fulfill it to a measurably lower degree). In our case, using the categories of growing churches versus non-growing churches (as comparison group) was a realistic pattern to achieve.

In NCD, numerical growth is not the ultimate goal of church development. We do not recommend churches to work with numerical growth goals (“By the end of 2020, our church will have 250 people in worship attendance”). Rather, we recommend setting precise and measurable goals related to the quality of the church (knowing that this procedure has the strongest imaginable impact on numerical growth). These goals can be either expressed in a description of a given quality level to be achieved (in so far as this description can be measured, i.e., precisely evaluated as to the extent that that goal has been reached at an agreed-upon point in time, such as, “12 more spiritual-gift counselors by the end of this year”), or in scores of the quality index that the NCD Survey reveals (e.g., “From presently 41 to 50 in our gift-based ministry score by the end of this year”).

3. The Quality Index

In NCD, the quality index, i.e., a numerical score summarizing the overall quality of the church, plays a key role. Different segments of church life (for instance, on the level of individual items of the questionnaire, on the level of each of the eight quality characteristics, or on the level of the church’s minimum factor) are all connected to the quality index. With the introduction of the quality index, the discussion on church quality in Christianity worldwide moved to an altogether different level.

The place of “quality” in NCD

Prior to the introduction of the quality index, the term “quality” (as opposed to quantity) was usually one of the most foggy words imaginable. Meanwhile, we have achieved a fundamentally different situation. Today we can state that the “quality index” is easier to accurately measure than attendance figures. Our research indicates that the worship attendance figures communicated to us are considerably less reliable than the quality index obtained by the NCD Church Profile. In other words, throughout the past 25 years many aspects of the “quality/quantity” discussion have been turned to their very head.

It may be helpful to add that the term “quality” in this context must not be confused with that which empirical social science calls “qualitative research.” Qualitative research refers to a different kind of research methodology than quantitative research. In NCD, we apply both quantitative and qualitative methods of research, both targeted at measuring qualitative and quantitative aspects of church life.

Goodness criteria: objectivity, reliability, validity

In all kinds of empirical research, the goodness criteria of objectivity, reliability, and validity play a key role. Without applying these criteria, no scientific assessment tool can be developed. Any research methodology that does not work according to these standards is predestined to produce erroneous and thus misleading results (even if, and especially if, these results may be positively received by those taking such a test).

● Objectivity refers to the independence of the results from the persons involved in the process of processing them—both in the phase of conducting a test and in the phase of evaluating its results. As far as the NCD Church Survey and the various eTests provided by NCD are concerned, a high level of objectivity is already granted by the standardization of the respective tests, particularly in their online versions. However, a situation in which the objectivity could be impaired is a group setting in which church members fill in the questionnaires while being together, and while the leaders tell them “how to do it,” without restricting themselves to purely technical instructions. The same applies when using the Survey with individuals who are illiterate or visually impaired. The process of reading the questions to them while expecting their answers can be done in different ways that have a strong influence on the results.

● Reliability refers to the accuracy of measurements, measuring instruments, and measured results. It is important to note that this criterion exclusively refers to the accuracy of the measurement, regardless of whether the reality that is measures is actually what was supposed to be measured. Reliability can be secured by different methods. One of them is the “test-retest method.” When it cannot be applied, the most frequent procedure is Cronbach’s alpha. This is a measurement of the reproducibility of a given test result. In the case of the NCD Profile, a test-retest method could not be applied, since conducting the test, both in the preparation for it and in evaluating the results, is already part of an intervention that changes measurable reality. For that reason, we opted for Cronbach’s alpha.

● Validity: There can be research data that fulfills both of the criteria mentioned, i.e., it is both objective and reliable, but it doesn’t fulfill the criterion of validity. Validity is concerned with the question: Do we actually measure what is supposed to be measured? In the NCD Church Survey, the external criterion of increasing worship attendance is the primary way of guaranteeing the validity of the results.

Christoph Schalk’s paper Organizational Diagnosis of Churches gives background information about the goodness criteria applied, by illustrating these dynamics with reference to a small number of churches.

Practical consequences

The most important practical consequences of the possibility to assess a church’s quality index are the following:

● First, since any church’s quality can accurately be assessed, it has become possible to talk about quality and health in very precise terms.

● Third, beyond the assessment of the quality index (e.g., 35 or 64), the Church Survey gives insight into the quality scores of many different subcategories, for instance at the level of each of the eight quality characteristics, and on the level of each of the items of the questionnaire.

● Fourth, it has enabled us to talk about the future of Christianity in a much more nuanced way than is possible when the respective prognoses are based on data that don’t include a quality index. Churches with different levels of quality can expect a very different future, a scale that ranges from “vibrant and thriving” to “dying.”

● Fifth and most importantly, the quality of the church can be influenced positively, so that churches can move from low quality (prognosis: dying) to high quality (prognosis: vibrant and thriving). This possibility is usually not included in prognoses on the future of Christianity.

4. Minimum factor theory

The minimum factor strategy—illustrated by a graphic of a barrel filled with water (worship attendance) that is constructed out of eight staves displaying different length (eight quality characteristics)—is a chief feature of NCD. It helps reduce the focus from endless individual elements that are all interrelated and important, to prioritizing one specific area—the quality characteristic displaying the lowest quality at a given time. This possibility has resulted in an enormous complexity reduction, while at the same time helping to focus on measures that are strategically decisive.

Misunderstandings of the minimum factor

It must be stressed that the minimum factor strategy must not be treated as a general pattern of addressing all kinds of topics, but has a clearly limited application area—the work on the eight quality characteristics of growing churches. While each of the eight quality characteristics is equally important, at a given time it makes most sense to focus one’s attention particularly on one of these characteristics—the minimum factor.

In Natural Church Development, there are areas in which we recommend a minimum factor approach (e.g., in the tools developed for loving relationships that focus on eight different aspects of the fruit of the spirit); others in which we recommend a maximum factor approach (e.g., in the tools developed for gift-based ministry); and others, in which we apply a combination of maximum and minimum factor (e.g., the NCD Church Survey, in which we help churches to use their strengths—among others, their maximum factor—to make progress in the area of their present minimum factor.

A minimum factor strategy as a general pattern of action, regardless of the area of application, would be a sure recipe for mediocracy and has never been suggested by NCD. As a rule of thumb, a minimum factor strategy should be applied whenever we are dealing with absolutely essential aspects of health, i.e., every single one must be displayed in order to survive and be healthy. In contrast, a maximum factor approach applies in those areas where the individual aspects evaluated, while being beneficial, are not absolutely essential for every individual or every group.

Empirical verification of minimum factor approach

The minimum factor strategy is not just a mental construct based on the image of the minimum barrel. The graphic provides no way of proving the minimum factor strategy, but is rather meant to communicate dynamics that have been proven before. The two most important empirical verifications of the minimum factor approach are the following:

● First, when doing Multiple Regression Analyses in the phase of the initial NCD research, we tested the correlation that many different expressions of quality have both to numerical growth (for instance, each of the individual quality characteristics, the quality characteristics as a whole, each individual item, different groups of items, the maximum factor, the minimum factor, etc.). From all of these options, the minimum factor of a church turned out to have by far the strongest correlation to numerical growth, and it has the strongest correlation to other expressions of quality.

● Second, when screening the actual implementation of NCD in longitudinal studies, we selected two categories of churches: (a) Those that focused on their respective minimum factor and worked successfully on it, indicated by at least a five point increase in that area when taking a repeat test. (b) Those that either ignored their minimum factor or worked on it inadequately, indicated by the minimum factor score remaining the same as before or even going down. Churches in category (a) experienced an increase in their overall quality between the two profiles of twelve points on average. By contrast, the quality of churches in category (b)—those that, while having a certain interest in church health, set other priorities than working on their minimum factor—dropped by ten points on average.

An alternate way of expressing balance

The minimum factor strategy is another way of addressing the goal of balance (which in NCD categories always implies, balance on a high level). Applying a pure maximum factor strategy to the eight quality characteristics would inevitably lead to increased imbalance. One of the most eye-opening ways of evaluating data is measuring the gap between maximum and minimum factor. The higher that gap (i.e., the higher the imbalance), the less likely is the respective church to grow.

Next