A qualitative and quantitative triangulation approach to validate the measurement of spiritual constructs

Over the span of 10 years a faith-based NGO has tested over 140 unique faith and spirituality questions. This triangulation study was developed to identify which survey questions can be validated as predictive of where a person is in their spiritual journey.

Although capturing spiritual metrics can be quite abstract, the objective of the study is to isolate survey questions that are predictive of whether a person is categorized to be a mature  adherent to the faith by local church elders. Measuring spirituality is inherently complex, involving culture, context, theological leanings, and subjectivity. While it is worthwhile to explore these complexities further, in this study, the judgment of local church leaders was used as ‘true north’ on the maturity level of a respondent for a number of reasons. First, the perspective of local church members puts the process of categorization in the hands of those that would come from the same context as the respondents. Second, the method leverages on elder experience in  interviewing and discerning a person’s spirituality and understanding of the Bible. 

The study uses the triangulation approach, where both quantitative (surveys) and qualitative (interviews) methods are applied. Ordinarily, surveys administered via interviews are the main stream of data for the organization. Instead of attempting to validate the questions through intra-survey methods, a mixed quantitative-qualitative method was chosen. 

In addition, the study utilizes back-to-back interviews to ensure robust qualitative results. Each participant goes through two interviews with two different interviewers on the same day. After their first interview, participants wait until they move on to their next interview with a different elder. The second interviews are video recorded and will go under review and analysis at a later time. Each interviewer is also tasked to categorize each person into: Skeptic, Seeker, New Believer,  or Growing Believer. 

Elders from a local church in the province of study were enlisted to assess where a respondent is on their spiritual journey, similar to baptism interviews. Given the elders’ breadth of knowledge and experience, they were provided free-reign on the questions they asked to study participants. These consecutive interviews shed some light on the complexities of spirituality, not to mention insights on the spirituality of the elders themselves. As such, the percentage of agreement versus disagreement in these elders’ assessments of the participants becomes an important facet of the study. 

The study uses the triangulation approach to validate 82 survey questions against the elders’ assessments of the participants’ spirituality. 200 participants were selected from a mix of communities. All 200 participants were surveyed by contracted enumerators. From these, only 150 participants were chosen for interviews - split among Catholics, Protestants, and a few atheists, agnostics or a religious - based on survey results. 

Once selected, quantitative responses are collected by means of enumerator-administered surveys while the elder assessments were gathered through back-to-back interviews. The survey results were converted into binary indices (correct answer = 1, incorrect = 0) and the spiritual categorizations given by the elders were tabulated. The elder scores for each individual were combined by taking the average, and in the analysis those with an average score under the Skeptic or  Seeker categories were treated as ‘Non-Believer’, and those that were New Believer or Growing Believer were treated as Believer. 

These two sets of data were analyzed together by means of correlation tests, regression analyses, and confusion matrices to identify key survey questions that best predict spirituality as defined by the church elders. Due to complexities in the data, sophisticated regression methods were explored. The following are the supervised machine learning techniques used during the methodology testing: regularization techniques, such as Lasso, Ridge, and Elastic Net regression, as well as Least Angle regression, which combines Lasso regression  and Stepwise regression. 

Preliminary Results

The models identified 9 survey questions that were good predictors for the interview results by at least 2 regression models:

• I read or listen to the Bible on my own. 
• What is your current religion or denomination?
• Do you own a Bible? 
• How active are you in this Church (Sunday Service)?
• It doesn’t matter much what I believe so long as I am good. 
• If I am good enough, God will cleanse me of my sins.
• I show God's love to those around me through telling them about Jesus.
• Eternal salvation is possible only through grace, not works. 
• What type of church (Sunday Service) do you belong to?

The subsequent step after identifying the best predictors involves determining the most effective combination or combinations of questions that accurately identify individuals according to the categorizations given during the interviews. A confusion matrix approach was applied, allowing a specific set of questions to be identified that are linked with a positive and negative predictive value of interview categorization. These findings are currently being refined and will be ready to present at the annual meeting.