The Ripped Bodice’s Diversity Report: A Critique
--
by Nick and Ari
Summary
We offer a critique of The Ripped Bodice’s State of Racial Diversity in Romance Publishing Report. With its lack of transparency, unethical, and unclear methodology, the diversity report leaves us with more questions than with answers. Though well-meaning, a study like this does a disservice to both publishers and BIPOC authors, while also setting a dangerous precedent of allowing poor ethics and poor data practices to run rampant in the romance community. In the last couple of years, we have seen the damaging effects of allowing misinformation in the media, so why are we still uncritically accepting a report that could be spreading misinformation to be published year after year? We urge The Ripped Bodice to do better and to carefully consider a few of the alternatives presented in this article.
Introduction
Since 2016, The Ripped Bodice, an L.A.-based romance-exclusive bookstore owned by Bea and Leah Koch, has released The State of Racial Diversity in Romance Publishing Report, a yearly data report that seeks to track the publication of books written by authors of color and Indigenous authors. The intent of the report is admirable. However, this study, meant to highlight the self-evident whiteness of romance publishing, is riddled with flaws from the study design, to the ethics, and to the reporting phase. If we want to understand and make impactful commentaries that can sway the state of publishing, it is essential that we responsibly and ethically collect, analyze, and report diversity data to the best of our abilities in order to back our statements with the least biased evidence. We live in a world that is becoming increasingly distrustful of science and lacking in data literacy. There are good reasons why research studies in any academic field are held to rigorous standards. Why then should a study in romance publishing, one that is often cited in major publications such as Publisher’s Weekly and The New York Times, be any different? Why are we accepting at face-value a study generated by two individuals with seemingly no analytical background or desire to accept external expertise?
Ethics Violations
Let’s take this back to basics for a moment: What are ethics?
Ethics are the moral principles and beliefs that govern one’s behavior. From making decisions to personal beliefs, morals help shape us as individuals as we navigate the trials of daily life. Just like in everyday life, ethics also play a crucial role when it comes to research. When one first proposes a theory, one of the first questions asked is: How do I go about proving this theory in an ethical manner? If one is unable to answer this question, then continuing with the study is out of the question. So how do ethics relate to The Ripped Bodice’s diversity report?
With just a quick glance at The Ripped Bodice’s FAQ section, it is glaringly obvious that ethics was an afterthought when this study was being conducted. The methods used to collect the data for this study are completely unethical — dare we say even cross the line of nonconsensual. Again, taken directly from their website, let us go through the following steps in this methods section and break down why some would be considered unethical.
Q: How did we collect this data?
3. Research more than 1000 authors to identify people of color.
Firstly, it is always important to report the exact sample size used in an experiment. Stating “more than 1,000 authors” were included is not helpful as it is too large of a range. Secondly and most importantly, why is The Ripped Bodice determining the racial identities of more than 1,000 authors? This is where the ethical line is crossed. On their FAQ page, The Ripped Bodice explains how they go about determining an author’s racial identity:
Q: How do you determine If someone is a person of color?
A: It is important to remember that race, ethnicity and national origin are all different things. For the purposes of this report, Native American and Indigenous authors are included in the term “people of color.” We understand that there are different schools of thought on this terminology and will continue to do our best to respect the wishes of Native American and Indigenous authors.
We were able to discern, to the best of our ability, members of different racial groups through information like photographs, social media and biographies from author websites. Many authors of color indicate that they identify as such in their bios or on their social media pages. All information we worked from is public. This is not a foolproof system and we readily admit that.
Reflecting on that statement, we must ethically ask, what gives anyone the right to determine another’s racial identity? What gives The Ripped Bodice the right to determine which author is deemed a ‘person of color or Indigenous’ and which is not? Racial identity is much deeper than merely looking at one’s photograph. Ignoring the ethical and bias implications of identifying an author’s racial identity from photographs and social media biographies, what are the exact criteria used for the racial determinations? What happens when authors have no clear photographic or biographical evidence? Are they being excluded from the study or are they classified as white? What about misclassifications? (Please note, that to our knowledge, publishers are rightfully not sharing race/ethnicity data of their authors due to privacy and legal concerns.) So if publishers aren’t providing the racial identities of their authors, it would be safe to assume that authors are not at all aware that their information is being collected for this study.
Informed Consent
It’s one of the most important aspects of a romance book, and yet in this study, informed consent is nowhere to be found.
Nowhere in any of the studies produced within the last five years does it state that surveys were distributed to authors to have them define their racial identity. The Ripped Bodice made these calls themselves. It wouldn’t be a far-fetched idea to assume that The Ripped Bodice has their own form of a race database à la Ancestry.com for authors because if we refer back to their FAQ page, it states that they run the data themselves. We also have to wonder whether this information is being insecurely stored on personal computers without any encryption. It would also be fair to question exactly how much research was done on each author and what other data was collected without informed consent. And lastly, while The Ripped Bodice says it welcomes corrections of misassigned identity, how are authors able to report if they are being mislabeled, if they are not informed that they were included in the study?
For five years, The Ripped Bodice has run these reports, and never have they been questioned on their unethical methodology. In a professional setting, a study like this would have been cast aside immediately. Why have we uncritically accepted an unethical study like this to be produced year after year in our own community?
Transparency, where art thou?
Transparency is key when running any type of study, as any good study should include a reproducible and replicable methodology section. Researchers are constantly testing and retesting theories so it is important to note every step taken as it shows how and why certain results were obtained in a study. When it comes to The State of Racial Diversity in Romance Publishing Report, transparency is severely lacking. Taken directly from their website, let us go through the following steps in this method’s section and discuss why there is a need for more clarity.
Q: How did we collect this data?
1. Identify and contact the leading romance publishers to determine if they will participate (participation consists of providing a list of titles published in a certain year).
2. Collect title data for publishers that choose not to participate from publisher and distributor websites and catalogues.
There are several holes and important unanswered questions here. Firstly, there is no clear definition of what information is being requested from participating publishers. Are participating publishers being asked to provide a catalog of the books published in the year under study? And if that is the case, what makes non-participating publishers different if the books used in these yearly reports are just being pulled from their catalogs, too? Indeed, one wouldn’t technically need a publisher’s participation given that most of these catalogs can be found on a variety of publishing websites. This leads us back to the question of what The Ripped Bodice is actually looking for when they submit a request to publishers. This issue could be easily resolved if The Ripped Bodice had established a clear definition for romance books. In this study, what makes a book a “romance book”? While it may be obvious, it is still important to provide this definition because everyone’s definition of romance is not the same. It is the reason why Nicholas Sparks is considered a romance author by the general public even though most of his books don’t provide a happily-ever-after, which is mandatory in order to label a book as romance. We then must ask whether there are any inclusion and exclusion criteria? Are Young Adult romances included in this study or is it strictly Adult romances? Are Women’s Fiction included? It’s questions like these that highlight the numerous limitations already present right at the start of the study.
If publishers are willingly submitting the requested book data, is there consistency in the definition of what books are included across the publishers? Many publishers also re-issue their books in different formats or repackage them with new covers. As an example, Nora Roberts and Brenda Jackson’s books are routinely re-released. Are these being counted towards the totals? Could there be inherent measurement inconsistencies or errors for data obtained from the publisher vs. the data that The Ripped Bodice had to seek out? All of these questions have an impact on what is being reported, particularly when comparisons are being made across publishers and imprints.
4. Crunch numbers, determine percentages, put report together.
What does The Ripped Bodice mean when they say “crunch numbers’’? Are they referring to any type of statistical tests used to determine and compare percentages and if so, why are these tests not listed in the report? If someone wanted to replicate this study, how would they know which tests to run in order to see if they too would achieve the same outcome? Was there any data cleaning involved? All of these questions, which might significantly change how we interpret the results, are left to the reader to speculate. These are important details because they give readers of the report a clear picture of what exactly is being measured and ultimately reported. How is one to accept any of the data presented if there is so much ambiguity surrounding the information that is collected? These questions continue to pile up as one delves deeper into the analysis of this study, demonstrating that transparency is a bigger issue than we think.
The Math Ain’t Mathin’
Let’s take a closer look at the possible statistical and data reporting issues in this study.
Figure 1: Screenshot of page 3 from 2020 The State of Racial Diversity in Romance Publishing which shows a graph of percentages from 2016–2020 for Avon Romance.
The actual presentation of their final data leaves much to be desired. For one, it’s unclear in each of the graphs (Figure 1), arranged by each publisher or imprint, what exactly is being presented — there are no axes labels nor are any of the titles explicit. Does “WHAT PERCENTAGE OF BOOKS PUBLISHED IN EACH YEAR WERE WRITTEN BY PEOPLE OF COLOR?” refer to the percentage of books by “authors of color and Indigenous authors” per pub/imprint or per total books across all publishers under study here? A good graph should not leave it up to the reader to wonder what is being presented nor should it leave room for misinterpretation.
Figure 2: Screenshot of page 2 from 2020 The State of Racial Diversity in Romance Publishing which shows the different publishers or imprints and the percentage of books published in 2020 written by People of Color.
This also brings up the importance of including total counts — what we call N in statistics — when presenting percentages. 10% of 100 books is a very different number than 10% of 10 books. It’s possible that some of the percentages for some publishers and imprints are inflated as a result of a relatively lower number of total books published in a given year. For example, imprints such as Harlequin or Avon publish more books in a given year than St. Martin’s Press. This can make the numbers presented inappropriate and meaningless to compare across the years and across different publishers without seeing the raw totals. Some of the publishers are broken down by imprints while some are not (Figure 2). Kensington is not broken down into its imprints, but HarperCollins is broken down into Avon, Carina Press, Harlequin Series, HQN, and Mira Books. Once again, the rationale behind these decisions is undisclosed.
Publishing is a fluid industry that fluctuates annually. Several factors may impact the number of books acquired, including imprints facing budget cuts, editors leaving a publishing house, or the reimagining and merging of imprints. If we’re looking at 2020, there’s also the pandemic to consider which possibly pushed back some titles. None of these factors are taken into account nor are they noted anywhere in this report. Instead, the data is presented as evidence that the industry is not sticking to its commitments of publishing more authors of color. Again, this report would be more conclusive if total counts were reported when presenting the results.
So, What’s the Alternative?
If external expertise was sought out, then The Ripped Bodice would know that ideally, a survey is a proper way to conduct a study of this nature. An exemplary diversity report is the Diversity Baseline Survey (DBS) by Lee & Low Books, which aims to survey publishing houses and review journals to establish data and evidence on the racial, gender, sexual orientation, and ability makeup of the publishing workforce. It should be noted that Lee & Low Books collaborated with a team of experts since the inception of the report in 2015 to handle the surveys and data analysis with proper protocols: Toluna administered and deployed the surveys and Dr. Laura M. Jiménez & graduate student Betsy Beckert from the Wheelock College of Education & Human Development at Boston University handled the data analytics of the second iteration of the study. As part of the full report, the DBS clearly outlines the methodology used, presents the exact sample sizes together with the survey response rates, states how the anonymity of participants was maintained throughout the study, and discusses important results of the analysis. These are all excellent research practices that The Ripped Bodice should also be striving for in their report. Further, because the DBS is a survey that is sent out to the participants, informed consent is at the forefront of the study. This also allows potential participants, publishing employees in the case of DBS, to i) opt-out of the study if they wish to do so and ii) identify race, gender, and sexual orientation on their own terms, lowering the probability of misclassification and eliminating the risk of unethical research or research misconduct.
A similar approach would work in the case of a diversity report aimed at assessing diversity in romance publishing. Once the listing of romance books has been appropriately and consistently collected for all publishers and imprints, a survey should be sent out to the authors in question. This will not only allow the researchers to collect accurate race data but also allows the opportunity to expand the report to include other forms of diversity. Most importantly, it allows for the data to be collected ethically with informed consent, allows authors to self-report their race on their own terms, and gives them the option to opt-out if they don’t want to be included in the study. Of course, surveys aren’t without their faults. They won’t give us the exact full picture because response rates tend to be fairly low, but with the appropriate sample size, sampling, and statistical techniques, it can estimate the full picture. Cleaning survey data, which involves fixing or deleting corrupt or poorly formatted or duplicate data, is also often quite challenging, but there are experts out there with the right training.
If a project such as the DBS is too elaborate, The VIDA Count is another good, but a simpler model of a well-managed diversity report. Currently, the study seeks to tally the gender breakdown in major literary publications and book reviews, similar to what The Ripped Bodice is attempting with the racial diversity report. Volunteers in teams are tasked to track the numbers from publicly available information using a standardized rubric provided by team leaders. Each team then meets to compare the numbers to ensure consistency in the data across volunteers. What is exemplary about this report is the overall transparency. The methodologies, the definitions of the measures, the totals at each publication, and the limitations such as publication bias, are clearly laid out in the FAQ or the report. Obviously, neither of these projects is a task for a single person nor is it one that can be completed within a short timeline, especially without the proper training, expertise, and rubric, which is why not everyone is, nor should they be, conducting research studies.
In a 2020 interview with Jason Low of Lee & Low Books, Bea and Leah Koch state that the DBS was a source of inspiration for them. If this is the case, it is baffling as to why they have not taken the initiative to reproduce similar work for The State of Racial Diversity in Romance Publishing Report. Better yet, why have they not sought out external unbiased collaborators with the appropriate expertise and without a stake in romance publishing or even volunteers like in The VIDA Count to help them conduct this study? Despite the flaws being brought to their attention (Nick has personally volunteered to help them to no response), The Ripped Bodice continues to be silent. It’s entirely irresponsible of them to continue to generate this report and go on a publicity tour advertising it year after year and acting like these numbers are hard facts despite being made aware that there are major inconsistencies here.
Some Much Needed Updates
The Ripped Bodice has been publishing their diversity report for five years and yet, there have been no changes or updates made in regards to the methodology nor have they expanded this study to include other forms of diversity. Yes, racial diversity is important, but shouldn’t the goal be to have more inclusive stories all around? While we won’t further discuss this thought, it is something to keep in mind when looking at the bigger picture. But let’s backtrack for a moment and discuss racial diversity. Hypothetically speaking, say an imprint published only books written by authors of color for a certain year, it would be safe to assume that on paper it would look like that imprint is being more inclusive, yes? Because it would definitely reflect that way on the graphs in the diversity report. But what happens if half of those authors of colors chose to only write about white couples? Then would the imprint really be championing diverse romances? It is questions and scenarios like these that are not discussed enough in traditional publishing and these types of conversations need to be added to the diversity agenda.
Let’s talk terminology. For five years, this study reported on romance books written by people of color. However, the term people of color is not as inclusive as some may have previously thought it to be. If we really want to bring this study up to speed with the times, then using the term BIPOC would be a step in the right direction. BIPOC, which stands for Black, Indigenous, and People of Color, is a much more inclusive term as it highlights each racial group’s relationship to whiteness in the United States. The term BIPOC addresses the unique experiences that Black and Indigenous people have faced due to colonialism in the States. This term also addresses anti-Blackness among other racial groups. The fact that The Ripped Bodice has not updated the terminology in this report demonstrates a lack of understanding of racial nuances.
To Hell with Good Intentions
While the intent of the report is important, we cannot dismiss the countless issues found in this study. From the lack of transparency to the unethical and unclear methodology, The Ripped Bodice’s diversity reports leave us with more questions than answers. A study like this does a disservice to both publishers and BIPOC authors. If these figures can be proven to be unreliable and inaccurate, then the media outlets that have referenced this report as a source would then have to retract their stories. For instance, in an article about the 2020 report, Book Riot erroneously claimed that The Ripped Bodice obtained racial data directly from some publishers. This is a result of people making false assumptions on this study based on the scant information provided by The Ripped Bodice.
This report also has the potential to harm prospective BIPOC writers. Aspiring writers have responded to The Ripped Bodice’s 2020 report, expressing their intent to only query publishers who showed relatively higher percentages. Because the percentages are provided without context, the obvious conclusion when looking at the 2020 report is that St. Martin’s Press is leading the charge on publishing diverse books. A closer look at their catalogs will show you that this is not the full picture. St. Martin’s Press released considerably fewer books overall than other publishers included in the study as a result of the closure of several of their imprints.
The report briefly touched upon the statements publishers made with respect to fighting racism and dismantling white supremacy in publishing and alluded to the notion that publishers are not keeping their promises because the findings of this report did not reflect an increase in BIPOC authors being published in 2020. Firstly, we cannot expect romance publishing to become more inclusive overnight because the issues of institutional racism run deep at every level of the industry as noted by the DBS. Secondly, the acquisition process of a manuscript is a lengthy one and once a publisher acquires a book, it does not go into production and on the shelves of your bookstore the very next day. Some books take two to three years before they make it into the hands of readers. If we truly want to see if publishers are keeping their word, we need to be taking a closer look at acquisitions and run appropriate trend analyses using several years of data. As presented in the report, the trends show a meaningless and skewed view of improvement or worsening in the industry as none of the data appears to be normalized over time. Everyone who consumes romance knows that publishing is overwhelmingly white — we don’t need poorly collected and unethical data to tell us that.
Finally, we must address the fact that Romancelandia has often failed to call out their own and hold them accountable — particularly when it comes to well-liked “frontline allies” in positions of power. The diversity report is one of those clear instances where most everyone uncritically accepts the information, no questions asked. There’s no denying that The Ripped Bodice has done important work within the romance community and though the intentions come from a good place, they are seemingly ill-equipped to handle a project as nuanced as a diversity report. Our concerns voiced on social media have yet to be addressed. Thus, they are aware that this report does not serve the propagation of reliable information. Yet they continue to center their voice and promote it as one of expertise when they are not experts (in their FAQ, they state “We’re not statisticians”) and routinely ignore offers of outside experts to correct their inaccuracies. It leaves us to wonder what motivates The Ripped Bodice to produce a report that is unreliable and unethical in its technique and presentation. If the goal is holding publishing accountable in its promise to BIPOC to be more inclusive, one would think The Ripped Bodice would embrace critique and offers to make their report ethical and data-based to best represent how publishing has behaved (for better and for worse) in lieu of that promise; however, they have not.
In a recent article where Bea and Leah Koch discuss their findings in the 2020 diversity report, they state, “We invite you to review the data in this year’s diversity report and draw your own conclusions.” Though it is unclear to us how one is supposed to “review the data” in the diversity report when they have been unresponsive to requests to view the raw data, we have outlined a critique of the various issues in this article. Now it is up to The Ripped Bodice to decide whether or not they want to truly listen and demonstrate their commitment to being actual allies of BIPOC communities, by doing justice to the journey, for better and worse, of BIPOC in romance publishing.
