Systems for Fixing Bias
Bias presents a bigotry-related problem to which there are many more actionable solutions. As we took a first-principles approach to secondary education with the Collins Institute (CollinsInstitute.org), we had to develop a system preventing political bias from creeping into our model. Separately, as consultants, we also helped a firm develop a system preventing too much political bias from polluting its moderation policies. Let’s discuss each system in turn.
Preventing Bias in Education
To block bias from polluting our education system, we teamed up with Metaculus, the world’s leading forecasting company, which essentially runs what can be thought of as large betting marketplaces (without the money part) that speculate about the likelihood of future outcomes (something Metaculus prediction markets achieve with crazy high reliability). We incorporate Metaculus’ prediction markets into our school’s core processes by including prediction market questions in multiple-choice assessments within topics subject to political bias (students are encouraged to make a prediction based on their knowledge related to some relevant outcome that has not yet come to pass).
If a student demonstrates a strong ability to predict outcomes related to a subject while simultaneously performing poorly in general multiple-choice assessments related to that subject, it is a sign that our assessments have become biased and are in need of correction. A person who can’t better predict future events based on current information after education has not been “educated”; they have simply been taught to repeat a particular party’s perspective.”
This system has the added benefit of yielding a metric we can use when evaluating students, with one of the core problems of a single metric for measuring students being that 80% of students will always fall outside the top 20% (which hampers their ability to demonstrate they’re a top performer in job and university applications). In addition to this added benefit, variations of the system also allow us to constantly A/B test different teaching systems.
When you define the “success” of an education system as a real-world measurable accomplishment outside of some state-mandated guess as to what an educated student resembles, you can enjoy the benefits of a constantly updating and improving “self-healing” educational system. Within some topics, we define success as an ability to more accurately and consistently predict future events. In other domains, we use direct authentic assessment (for example, how many five star reviews a student-written fanfiction gets online).
We use AI to constantly generate new potential test questions, which are screened by humans for inclusion in our curriculum. The accuracy with which these questions judge student knowledge is determined by their correlation to students’ performance on authentic assessments.
Our favorite vindication of this model came from an authoritarian old English teacher and failed author who expressed scorn for our system. She pointed out that many popular fan fictions became popular books, like 50 Shades of Grey, which got its start as Twilight fan fiction, and that she would have given E. L. James, the writer of 50 Shades of Grey, bad grades.
There is no more objective picture of the flaws in our education system than an arrogant English teacher who has not one measurable accomplishment in writing to her name fantasizing about how she would lord her arbitrary authority over one of the single most read authors in human history. Yes, E. L. James may not have adhered to all the silly rules the educational orthodoxy uses to determine who is an “upper class” writer and who is a “low class” writer, but her writing objectively served its purpose at an elite level.
Within every subject, there will always be a myriad of little rules used to signal “class” that are not relevant to the functional outcome of a product. Should a student wish to learn these, we can develop authentic assessments specifically tied to mastering these esoteric skills (like having third parties judge the education level of students based on their writing), but we don’t think the pursuit of a now-largely-arbitrary ideal should make up the core of a subject.
Reducing Bias in Content Moderation
At one point, we were asked to help a team developing an unbiased content moderation system based on our school’s design.[[26]](#_ftn26) While it turns out our theories on reducing bias within school systems are largely irrelevant in this domain, theorizing on incentive systems for this team helped us with this book.
The problem with content moderation in tech companies is twofold: First, the staff at these companies is overwhelmingly progressive and individuals at these organizations will actively target and try to eliminate anyone who espouses even moderate political views. We speak here from personal experience; while it may not be a policy of the organization, many individual employees will attempt to purge ideologically different colleagues with a lot of institutional cover given the ideological conformity within these orgs.
For example, 99.6% of Netflix, 98.7% of Twitter, 96% of Google, and 94.5% of Facebook political donations are to Democratic candidates.[[27]](#_ftn27) This is not just a natural consequence of programmers being more likely to be Democrat—as a whopping 26.6% of programmers are Republican—but a manifestation of the rather systemic bias and witch hunts cited above.[[28]](#_ftn28)
Management at these companies must build mechanisms to ensure that the policies developed by their employees don’t reek of extreme political bias. This is a problem when those policies dictate what content to ban and what to keep, as a failure to stay at least plausibly unbiased can cause these companies to face government regulation.
The second problem faced by these companies is that the low-cost, marginally employed individuals who often serve as content moderators usually have a strong socialist bias due to the communities from which they hail. These forces place a heavy progressive bias on these companies’ content moderation practices, which, again, puts their employers at risk of legislation artificially placing a conservative bias on their moderation practices.
Some companies have taken a crack at fixing this is with AI, but if said AI is programmed by ultra-progressive coders, then it is bound to feature progressive bias. How, then, does one resolve this problem?
The solution is fairly simple and not present in current models of content moderation: Create a system that punishes moderators for mistakes and disproportionately punishes them for politically biased mistakes. Specifically: Give users the ability to flag content as being banned for politically motivated reasons and cite the directionality of that bias. Then escalate the contested case to a group of judges with political leanings sympathetic to the banned content (e.g., if the moderator is accused of liberal political bias, the judges will have a conservative bias).
If the judges disagree, the content creator loses their ability to ever appeal a ban again. If, however, the claim is judged as accurate, the moderator accrues a citation. If, at the end of the month, a moderator ever has more than a certain number of citations related to overturned cases—with that threshold changing logarithmically depending on the bias of their moderation—they are punished (perhaps they don’t get a bonus, perhaps they get fired, etc.). For an example of how this logarithmic system would work: A company may permit moderators to accrue 20 citations if 10 were liberal and 10 were conservative but only five citations if all five demonstrated liberal bias.
While this system won’t entirely eliminate bias, it will dramatically reduce it at the cost of allowing many more things to “fall through the cracks” and a slightly larger moderation team (a small price to pay). We expect the actual judging system would rarely be used after a short period of time, with the mere fear of its existence being enough to prevent individuals from knowingly allowing their bias to seep through in their decisions. This system forces moderators to ask themselves: “Would well-meaning people who are politically motivated to support this content also be likely to recognize it as misleading propaganda?”