What Can Stylometry Tell Us about Book of Mormon Authorship?

Editor’s Note: This KnoWhy is the first in a series which discusses stylometry and its relevance to questions of Book of Mormon authorship. This first article explains what stylometry is and gives readers a short history of stylometric studies performed on the Book of Mormon. Building on this foundation, subsequent KnoWhys will discuss some of the exciting new results from more recent stylometric research.

The Know

Since the Book of Mormon’s publication in 1830, a number of different theories have been proposed concerning its authorship.1 Those who believe that Joseph Smith translated the Book of Mormon by the gift and power of God naturally have accepted that its source texts were written by multiple ancient prophets.2 In contrast, those skeptical of the Book of Mormon’s ancient origin and miraculous translation have generally assumed that Joseph Smith himself authored the entire text, or that it was written by one or more of his 19th century contemporaries.

In order to help shed light on this issue, several studies have relied upon a type of analysis called stylometry.3 This field of research uses various statistical methods to detect linguistic patterns. Stylometry has most notably been used to help answer questions about texts with disputed authorship, such as the Federalist Papers4 and some of Shakespeare’s plays.5 It has even been shown that stylometry can detect an author’s unique writing style even after his or her words have been translated from one language to another.6 The following summaries highlight the results of several notable stylometric studies on the Book of Mormon.7

Larsen Study

In 1980,8 Wayne Larsen, Alvin Rencher, and Tim Layton relied on three different statistical methods9 which used non-contextual words10 to distinguish writing styles among the Book of Mormon’s internally designated authors, including Nephi, Alma, Mormon, and Moroni, as well as several 19th century candidates, including Joseph Smith, Sidney Rigdon, and Solomon Spalding.11 Despite initial concerns from some scholars,12 the validity of using non-contextual words to determine authorship is now widely accepted in the broader field of stylometric analysis.13

Larsen and his associates concluded that the Book of Mormon was written in “distinct authorship styles” and that none of the 19th century candidates that they tested “resemble Book of Mormon authors in style.”14 Not only did the Larsen study pioneer stylometric research on the Book of Mormon, but its results have provided an enduring statistical basis in support of Joseph Smith’s claims.

Holmes Study

In 1985, David Holmes, using measures of vocabulary richness, found no meaningful difference among the Book of Mormon’s claimed prophetic authors.15 He concluded that Joseph Smith himself wrote the text.16 However, in subsequent studies, other researchers discovered that patterns of vocabulary richness are often not reliable enough to distinguish between writing styles.17 Holmes himself recognized the comparative weakness of this method in his reanalysis of the Federalist essays.18 Naturally, these results largely invalidated his earlier conclusions on Book of Mormon authorship.

Hilton Study

In 1990,19 John Hilton and a team of researchers from Berkeley (most of whom were not LDS20) conducted a study using word pattern ratios21 and a new method of differentiation based on what Hilton called rejections.22 This study is especially notable because of its large control samples, which included 26 texts by 9 different control authors and 325 pairwise comparisons.23 Comparisons were made between texts attributed to Nephi and Alma and those from Joseph Smith, Oliver Cowdery, and Solomon Spalding.

The findings of Hilton’s research team largely agreed with the results of the Larsen study, leading them to conclude that “it is statistically indefensible to propose Joseph Smith or Oliver Cowdery or Solomon Spaulding as the author of the 30,000 words from the Book of Mormon attributed to Nephi and Alma” and also that Nephi and Alma “have wordprints unique to themselves and measure statistically independent from each other in the same fashion that other uncontested authors do.” These results indicate that the Book of Mormon was indeed “multiauthored, with authorship consistent to its own internal claims.”24

The Hilton study’s innovative stylometric approach, combined with its thorough statistical controls, make it a landmark study on Book of Mormon authorship. Using a slightly different method, researchers from Utah State University essentially reproduced the results of the Hilton study in 2006.25

Jockers Study

In 2008, Matthew Jockers, Daniela Witten, and Craig Criddle applied two statistical methods—delta and nearest shrunken centroid classification (NSC)—to the question of Book of Mormon authorship.26 They concluded that the Book of Mormon’s literary style most closely matched writing samples from Solomon Spaulding and Sidney Rigdon, two of Joseph Smith’s 19th century contemporaries.27

This study, however, contained at least eight significant errors,28 the most critical being that it used a closed set technique on what is clearly an open set problem.29 This excluded everyone but the study’s selected candidates as potential authors.30 Most notably, the Jockers study did not include Joseph Smith as a candidate author,31 and it made no provision for the text to have possibly been written by its internally claimed authors.32

Moreover, a closed set of NSC values can only measure the sample texts’ relative similarities to the Book of Mormon. This means that NSC analysis will always deliver positive results for one of the candidate authors in the set, even if his or her style happens to be very different from the Book of Mormon.33 In other words, the Jockers study failed to recognize how misleading the results of their analysis could be if the true author of the text was not included in their selected group of candidate authors.

Fields Study

In 2011, Paul Fields, Bruce Schaalje, and Matthew Roper reviewed the Jockers study and introduced an improvement upon the NSC method, which they termed extended nearest shrunken centroid classification (ENSC).34 This allowed for the possibility that an unknown author (or authors) not included in the set of potential candidates could have written the text. The Fields study also included Joseph Smith as a candidate author.

With these adjustments in place and other errors from the Jockers study corrected, Sidney Rigdon and Solomon Spaulding were each assigned a value of 0% for their relative likelihood of authoring the Book of Mormon. Joseph Smith fared little better at 3%. In contrast, the chance of it having been written by one or more authors not included in the set turned out to be 93%.35

While these results cannot identify the unknown author(s) who likely wrote much of the text, they do show that it is highly likely that the author(s) were not any of the 19th century candidates who have been conjectured to have written the Book of Mormon. Thus, the Fields research team offers a third stylometric study on the Book of Mormon which independently contradicts theories of 19th century authorship.

The Why

In light of the above studies, it can be responsibly concluded that the Book of Mormon’s internal claims about its authorship are consistent with the best stylometric evidence currently available. While the Holmes and Jockers studies each reached conclusions inconsistent with the Book of Mormon’s claims of authorship, both were later found to be fundamentally flawed. In contrast the Larsen, Hilton, and Fields studies each produced sound results. Their mutually supporting conclusions should therefore be taken seriously by anyone assessing questions of Book of Mormon authorship.36

Stylometry is not a perfect science, but over the years its methods for distinguishing writing styles have become increasingly refined. In fact, it has been demonstrated that stylometric methods are able to detect an author’s word-use patterns even when he or she attempts to write in a different “voice” or to imitate another text’s style.37 The Book of Mormon’s lengthy texts and complex content would make it especially difficult for its true author(s) to fool the stylometric analysis, whether intentionally or inadvertently. 

It should be understood that stylometry cannot prove that the Book of Mormon was written by multiple ancient American prophets. What it can reliably demonstrate, and what valid data from the above studies collectively argue, is that (1) the Book of Mormon was written in multiple, distinct authorship styles, (2) these distinct styles are consistent with the authors designated within the text itself, and (3) none of the proposed 19th century authors—including Joseph Smith himself—have writing styles that are similar to those found in the Book of Mormon.

Not only do these conclusions strongly refute the most popular alternative theories for 19th century authorship, but they can also strengthen faith that the Book of Mormon is what it claims to be. Firmly embracing the particular words of Nephi, Alma, or Mormon, as authentic statements spoken by true prophets can increase one’s ability to remain firm and steadfast in heeding their personal words and testimonies.

The prophet Jacob declared, “For, for this intent have we written these things, that [future generations] may know that we knew of Christ, and we had a hope of his glory many hundred years before his coming” (Jacob 4:4). While stylometry is capable of detecting stylistic differences among many of the Book of Mormon’s underlying authors, only the Spirit of God can confirm that they truly were ancient holy prophets who were called by God to testify of Jesus Christ.38

• 1. See Daniel C. Peterson, “The Divine Source of the Book of Mormon in the Face of Alternative Theories Advocated by LDS Critics,” FairMormon presentation, 2001, online at archive.bookofmormoncentral.org.
• 2. See Noel B. Reynolds and Charles D. Tate, eds., Book of Mormon Authorship: New Light on Ancient Origins (Provo, UT: FARMS, 1996); Noel B. Reynolds, ed., Book of Mormon Authorship Revisited: The Evidence for Ancient Origins (Provo, UT: FARMS, 1997); Paul Y. Hoskisson, ed., Historicity and the Latter-day Saint Scriptures (Provo, UT: Religious Studies Center, Brigham Young University, 2001); Stephen O. Smoot, “The Imperative for a Historical Book of Mormon,” at The Interpreter Foundation (blog), October 20, 2013, online at mormoninterpreter.com.
• 3. For an overview of this field of study, see Michael P. Oaks, Literary Detective Work on the Computer (Philadelphia, PA: John Benjamins Publishing Company, 2014); for this publication’s discussion of Book of Mormon authorship, see pp. 190–197. See also, Efstathios Stamatatos, “A Survey of Modern Authorship Attribution Methods,” Journal of the American Society for Information Science and Technology 60, no. 3 (2009): 538–556.
• 4. See Frederick Mosteller and David L. Wallace, Inference and Disputed Authorship: “The Federalist” (Reading, MA: Addison-Wesley, 1964); David I. Holmes and R. S. Forsyth, “The Federalist Revisited: New Directions in Authorship Attribution,” Literary and Linguistic Computing 10, no. 2 (1995): 111–127; Antonio Miranda-García and Javier Calle-Martín, “Testing Delta on the Disputed Federalist Papers,” International Journal of English Studies 12, no. 2 (2012): 133–150; Jacques Savoy, “The Federalist Papers revisited: A collaborative attribution scheme,” Proceedings of the American Society for Information, Science, and Technology 50, no. 1 (2013): 1–8.
• 5. See Reginald C. Churchill, Shakespeare and His Betters: A History and a Criticism of the Attempts Which Have Been Made to Prove That Shakespeare’s Works Were Written by Others (Bloomington, IN: Indiana University Press, 1958); James G. McManaway, The Authorship of Shakespeare (Washington, DC: Folger Shakespeare Library, 1962); Hugh Craig and Arthur F. Kinney, Shakespeare, Computers, and the Mystery of Authorship (Cambridge, UK: Cambridge University Press, 2009).
• 6. See John L. Hilton, “On Verifying Wordprint Studies: Book of Mormon Authorship,” BYU Studies Quarterly 30, no. 3 (1990): 97, 108, n. 17; reprinted in Book of Mormon Authorship Revisited, 236–237, 251–252, n. 17; G. Bruce Schaalje, John L. Hilton, and John B. Archer, “Comparative Power of Three Author-Attribution Techniques for Differentiating Authors,” Journal of Book of Mormon Studies 6, no. 1 (1997): 47–63. For the failure of machine translations (automated translations such as those provided by Google Translate and Bing Translator) to obscure or mislead stylometric analysis, see Michael Brennon, Sadia Afroz, and Rachel Greenstadt, “Adversarial Stylometry: Circumventing Authorship Recognition to Preserve Privacy and Anonymity,” ACM Transactions on Information and System Security 15, no. 3 (2012): 12:8–9, 16–19, 21; Aylin Caliskan and Rachel Greenstadt, “Translate Once, Translate Twice, Translate Thrice and Attribute: Identifying Authors and Machine Translation Tools in Translated Text,” Sixth IEEE International Conference on Semantic Computing presentation, September 20, 2012. These findings are important for Book of Mormon authorship studies since it is claimed to be an English translation of a text written in an ancient language.
• 7. For a historical overview of stylometric studies on the Book of Mormon, see Matthew Roper, Paul J. Fields, and G. Bruce Schaalje, “Stylometric Analyses of the Book of Mormon: A Short History,” Journal of the Book of Mormon and Other Restoration Scripture 21, no. 1 (2012): 28–45. See also, John L. Hilton, “Review of Ernest Taves’ Book of Mormon Stylometry,” (FARMS Preliminary Reports, 1986).
• 8. See Wayne A. Larsen, Alvin C. Rencher, and Tim Layton, “Who Wrote the Book of Mormon? An Analysis of Wordprints,” BYU Studies 20, no. 3 (1980): 225–251; reprinted in Book of Mormon Authorship: New Light on Ancient Origins, ed. Noel B. Reynolds (Provo, UT: Religious Studies Center, Brigham Young University, 1982), 157–188.
• 9. See Larsen, Rencher, and Layton, “Who Wrote the Book of Mormon?” Book of Mormon Authorship, 163–177. For a brief overview of these methods, see Paul J. Fields, G. Bruce Schaalje, and Matthew Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification to Investigate Book of Mormon Authorship,” Mormon Studies Review 23, no. 1 (2011): 91: “The Larsen et al. researchers used three statistical techniques—Multivariate Analysis of Variance (MANOVA), Cluster Analysis (CA), and Linear Discriminant Analysis (LDA)—to test for differences in the frequencies of noncontextual words. MANOVA is a method of testing for homogeneity (degree of similarity) within groups of items. CA is a method that can identify which items are closest to each other among all items compared. LDA is a method for determining a set of mathematical functions (discriminant functions) that can be used to classify items into categories based on their characteristics.”
• 10. Rather than conveying an author’s unique ideas, non-contextual words (a, an, the, with, without, etc.) simply provide the grammatical framework in which those ideas are structured. Non-contextual words are ideal for statistical analysis because they show up frequently and most authors have very little conscious awareness of their own unique patterns of using them. As explained by Fields et al., “Studying the function words in a text can indicate an author’s personal manner of expressing his or her ideas since they do not indicate what the author says but the way he or she says it.” Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 91.
• 11. For the 24 Book of Mormon authors included in the analysis, see Larsen, Rencher, and Layton, “Who Wrote the Book of Mormon? An Analysis of Wordprints,” Book of Mormon Authorship, 181. The 19th century writers included Sidney Rigdon, Solomon Spaulding, Joseph Smith, W. W. Phelps, Oliver Cowdery, and Parley P. Pratt. The Lectures on Faith and two sections from the Doctrine and Covenants were also included (p. 163).
• 12. For example, see D. James Croft, “Book of Mormon ‘Wordprints’ Reexamined,” Sunstone (March–April 1981): 15–21. In response to these concerns, see Wayne A. Larsen and Alvin C. Rencher, “Response to Book of Mormon ‘Wordprints’ Reexamined,” Sunstone (March–April 1981): 23–26.
• 13. See Fields, Schaalje, and Roper “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 92–94; Stamatatos, “A Survey of Modern Authorship Attribution Methods,” 540–541; Oaks, Literary Detective Work on the Computer, 1; Antonio Miranda García and Javier Calle Martín, “Function Words in Authorship Attribution Studies,” Literary and Linguistic Computing 22, no. 1 (2007): 50; John Burrows, “Questions of Authorship: Attribution and Beyond,” Computers and the Humanities 37, no. 1 (2003): 7.
• 14. See Larsen, Rencher, and Layton, “Who Wrote the Book of Mormon?” Book of Mormon Authorship, 172. Despite several reviews which questioned the Larsen study’s results, Fields et al. concluded, “On the whole, even after the thoughtful criticism of the Larsen et al. study is accounted for, the results of that early study continue to provide persuasive support for the claim that the Book of Mormon is the work of multiple authors and not the work of any of the likely nineteenth-century candidates.” Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 94.
• 15. See David I. Holmes, “A Stylometric Analysis of Mormon Scripture and Related Texts,” Journal of the Royal Statistical Society, Series A (Statistics in Society) 155, no. 1 (1992): 91–120.
• 16. Holmes, “A Stylometric Analysis of Mormon Scripture,” 118.
• 17. See Schaalje, Hilton, and Archer, “Comparative Power of Three Author-Attribution Techniques for Differentiating Authors,” Journal of Book of Mormon Studies 6, no. 1 (1997): 47–63; Stamatatos, “A Survey of Modern Authorship Attribution Methods,” 540.
• 18. See Holmes and Forsyth, “The Federalist Revisited,” 111–127. David Hoover writes, “Despite the attractiveness of measures of vocabulary richness, and despite the fact that they are sometimes effective in clustering texts by a single author and discriminating those texts from other texts by other authors, such measures cannot provide a consistent, reliable, or satisfactory means of identifying an author or describing a style. … Unfortunately, the long-cherished goal of a measure of vocabulary richness that characterizes authors and their styles appears to be unattainable. The basic assumption that underlies it is false.” David L. Hoover, “Another Perspective on Vocabulary Richness,” Computers and the Humanities 37 (2003): 173.
• 19. See John L. Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 225–253. For the summarized findings of the Hilton study, see John L. Hilton, “Wordprints and the Book of Mormon,” in Reexploring the Book of Mormon: A Decade of New Research, ed. John W. Welch (Salt Lake City and Provo, UT: Deseret Book and FARMS, 1992), 221–226. See also, John L. Hilton, “Some Book of Mormon Wordprint Measurements Using ‘Wraparound’ Block Counting,” (FARMS Preliminary Reports, 1988).
• 20. See Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 233: “As the major LDS contributor in the group, I was little different from my agnostic and Jewish colleagues: each of us seriously questioned whether objective measurement could determine who did or did not write a controversial document like the Book of Mormon.”
• 21. The 65 noncontextual word-pattern ratios which Hilton relied upon in his study were derived from A. Q. Morton, Literary Detection: How to Prove Authorship and Fraud in Literature and Documents (New York, NY: Charles Scribner’s Sons, 1978).  For a comprehensive list of these word patterns, see Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 245.
• 22. See Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 228–229: “If the same word pattern is found to be statistically different between the two texts, we identify the difference as a rejection. The total of the rejections measured when the two texts are tested for a large number of word patterns is identified as the number of rejections. The larger the number of rejections, the more likely the disputed text was not written by the author of the other compared text. Thus, testing a contested document against comparable texts from all possible candidate-authors will identify the most likely writer by eliminating authors whose texts generate high numbers of rejections.” For summarized results of all rejections for texts that were compared to one another, see p. 243.
• 23. See Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 236.
• 24. Hilton, “On Verifying Wordprint Studies,” in Book of Mormon Authorship Revisited, 241.
• 25. This study used a generalized discriminant analysis which is an extension of the linear discriminant analysis used in the Larsen study. See Todd K. Moon, Peg Howland, and Jacob H. Gunther, “Document Author Classification Using Generalized Discriminant Analysis,” in Proceedings of the Fourth Workshop on Text Mining, Sixth SIAM International Conference on Data Mining, April 22, 2006, online at siam.org.
• 26. See Matthew L. Jockers, Daniela M. Witten, and Craig S. Criddle, “Reassessing Authorship of the Book of Mormon Using Delta and Nearest Shrunken Centroid Classification,” Literary and Linguistic Computing 23, no. 4 (2008): 465–491. While delta analysis was already being used in stylometric studies, the application of NSC, which was originally developed for genomic testing, was unique.
• 27. See Jockers, Witten, Criddle, “Reassessing Authorship of the Book of Mormon,” 482. A foundational premise of the Jockers study was that the once prominent, though long discarded, Spaulding-Ridgon theory of Book of Mormon authorship might actually be valid. For several historical-based arguments against this well-known theory, see Matthew Roper and Paul J. Fields, “The Historical Case against Sidney Rigdon’s Authorship of the Book of Mormon,” Mormon Studies Review 23, no. 1 (2011): 113–125; Matthew Roper, “Myth, Memory, and ‘Manuscript Found’,” FARMS Review 21, no. 2 (2009): 179–223; Matthew Roper, “Mythical ‘Manuscript Found,’” FARMS Review 17, no. 2 (2005): 7–140.
• 28. For a summary of these errors, see Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 97. For in-depth analysis of each error, see pp. 97–108. See also, Roper, Fields, and Schaalje, “Stylometric Analyses of the Book of Mormon: A Short History,”37–43.
• 29. See Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 99–101.
• 30. The Jockers study included Solomon Spaulding, Sidney Rigdon, Oliver Cowdery, and Parley Pratt as potential 19th century authors in its closed set. Samples from Henry Wadsworth Longfellow and Joel Barlow were included as control texts, as well as combined samples from the Old Testament books of Isaiah and Malachi.
• 31. For a response to this decision, see Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 108–111.
• 32. The Jockers study’s use of NSC required access to sample texts (other than the Book of Mormon) that are certainly known to have been written by the candidate authors in its closed set. Thus, potential authors like Nephi, Alma, and Mormon couldn’t be included in the study because none of their writings, except those contained in the Book of Mormon itself, have been preserved for comparison. The Jockers study’s fundamental error wasn’t that it didn’t include Book of Mormon authors in its NSC analysis; that would have been impossible. Its error was in assuming from the outset that the authors in its closed set were “selected from among the most likely candidates” and that the results of their analysis “[support] the theory that the Book of Mormon was written by multiple, nineteenth-century authors.” Jockers, Witten, Criddle, “Reassessing Authorship of the Book of Mormon,” 483. By ignoring the possibility of the text being a translation of an ancient document and by unnecessarily excluding Joseph Smith himself as a potential author, the Jockers study “prejudiced their … results from the start,” as explained by Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 100.
• 33. See Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 101: “The logic of Criddle and associates’ approach is no different than asking, ‘Choosing among Boston, New York, and Chicago, which city is closest to Los Angeles?’ and then, upon finding that there is a 99 percent probability that Chicago is the closest, concluding that ‘Chicago is the city in the United States that is closest to Los Angeles.’” Obviously, even though Chicago is the closest among a set of proposed candidate cities, it doesn’t mean that it is truly close to Los Angeles. Likewise, just because a sample text in a closed set is closer to the style of the Book of Mormon than the other texts, it doesn’t mean that it is truly similar in style.
• 34. See G. Bruce Schaalje, Paul J. Fields, Matthew Roper, Gregory L. Snow, “Extended Nearest Shrunken Centroid Classification: A New Method for Open-set Authorship Attribution of Texts of Varying Sizes,” Literary and Linguistic Computing 26, no. 1 (2011): 71–88. The conclusions of these findings were summarized and adapted for LDS audiences in two different articles: Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 87–111; Roper, Fields, and Schaalje, “Stylometric Analyses of the Book of Mormon: A Short History,” 28–45.
• 35. See Fields, Schaalje, and Roper, “Examining a Misapplication of Nearest Shrunken Centroid Classification,” 104–108. In the Fields study’s corrected analysis, Oliver Cowdery received the remaining 4%, “indicating that the writing styles of the candidate authors show very little resemblance to the writing styles in the Book of Mormon” (p. 107).
• 36. In total, four different stylometric methods were used in the Larsen and Hilton studies. Each of these methods independently detected evidence of multiple authorship and ruled out commonly proposed 19th century candidates. The generalized discriminate anaylsis used by the Utah State University researchers adds a fifth method supporting the same conclusions, and the Fields study’s use of ENSC adds a sixth method ruling out proposed 19th century authors. Thus, when viewed collectively, there is plenty of corroborating data to confirm the most fundamental conclusions of these studies.
• 37. See Andrew Queen Morton, Literary Detection: How to Prove Authorship and Fraud in Literature and Documents (New York, NY: Charles Scribner and Sons, 1978).
• 38. See Book of Mormon Central, “How Will God Manifest the Truth of the Book of Mormon? (Moroni 10:4),” KnoWhy 254 (December 16, 2016).