Wikipedia:Historical archive/Policy/Approval mechanism

Wikipedia approval mechanism, defined

"Wikipedia approval mechanism" means any sort of mechanism whereby Wikipedia articles are individually marked and displayed, somehow, as "approved."

The purpose of an approval mechanism

The purpose of an approval mechanism is, essentially, quality assurance. By presenting particular articles as approved, we (Wikipedians) would be representing those articles as reliable sources of information.

Some basic requirements of an adequate approval mechanism

Among the basic requirements of an approval mechanism would have to fulfill in order to be adequate are:

The approval must be done by experts about the material approved.
There must be clear and reasonably stringent standards that the experts are expected to apply.
The mechanism itself must be genuinely easy for the experts to use or follow. Nupedia's experience seems to show that a convoluted approval procedure, while it might be rigorous, is too slow to be of practical use.
The approval mechanism must not impede the progress of Wikipedia in any way. It must not change the Wikipedia process; it should be an "add-on."
Must not be a bear to program, and it shouldn't require extra software or rely on browser-specific stuff like Java (or Javascript) that some users won't have.
Must provide some way of verifying the expert's credentials as well as a way to verify that he or she approved the article, not an imposter.

Some "desirements":

Makes it possible to broaden or narrow the selection of approvers (e.g., one person might only wish authors who have phd's, another would allow for anyone who has made an effort to approve any articles.)
Allows for extracting topic-oriented sets (e.g., in order to produce an "Encyclopedia of Music"). (The idea is that article approval could contain more information than just the binary "high-quality" bit, e.g. topic area, level of detail, and so forth. Such "approved metadata" would allow easy extraction of user-defined subsets of the full approved article set.)

The advantages of an approval mechanism

The advantages of an approval mechanism of the sort described are clear and numerous:

We will encourage the creation of really good content.
Large, reputable websites and the web in general are more likely to use and/or link to our content if it has been approved by experts.
The addition of an approval mechanism will be attractive to academics who might not participate without it--particularly the academics who might want to be reviewers.
It makes it easier to collect the best articles on Wikipedia and create completed "snapshots" of them that could be printed and distributed, for example.

Generally, Wikipedia will become comparable to nearly any encyclopedia, once enough articles are approved.

I am not sure there are any significant disadvantages of an approval mechanism, but idly, I think there might be one. I think that it's possible that Wikipedia might become more of an "exclusive club" than it is, if people start comparing nascent articles contributed by new contributors to the finished projects. I might not want to contribute two sentences about widgets if I think ten neat paragraphs, with references, is what is expected. Again, I don't know if this is really apt to be a problem.

Another general argument against is that this really doesn't seem necessary. An approval mechanism has been suggested since Day One of Wikipedia, and evidence aside that Wikipedia is working just fine, will probably continue to be suggested 'til kingdom come.

Proposals

Below, we can develop some specific proposals for approval mechanisms.

Sanger's proposal

When I say the approval mechanism must be really easy for people to use, I mean it. I mean it should be extremely easy to use. So what's the easiest-to-use mechanism that we can devise that nevertheless meets the criteria?

The following: on every page on the wiki, create a simple popup approval form that anyone may use. ("If you are a genuine expert on this subject, you can click here to approve this article.") On this form, the would-be article approver (whom I'll call a "reviewer") indicates name, affiliation, relevant degrees, web page (that we can use to check bona fides), and a text statement to the effect of what qualifications the person has to approve of an article. The person fills this out (with the information saved into their preferences) and hits the "approve" button.

When two different reviewers have approved an article, if they are not already official reviewers, the approval goes into moderation.

The approval goes into a moderation queue for the "approved articles" part of Wikipedia. From there, moderators can check over recently-approved articles. They can check that the reviewers actually are qualified (according to some pre-set criteria of qualification) and that they are who they say they are. (Perhaps moderator-viewable e-mail addresses will be used to check that a reviewer isn't impersonating someone.) A moderator can then "approve the approver."

The role of the moderators is not to approve the article, but to make sure that the system isn't being abused by underqualified reviewers. A certain reviewer might be marked as not in need of moderation; if two such reviewers were to approve of an article, the approval would not need to be moderated.

New addition I think it might be a very good idea to list, on an approved article, who the reviewers are who have approved the article.

--Larry Sanger

Bryce's Proposal

From my experience with the Wikipedia_NEWS, it seems that there's a lot that can be done with the wiki software as it exists. The revision control system and its tracking of IP addresses is ok as a simple screen against vandalism. The editing system seems fairly natural and is worth using for managing this; certainly we can expect anyone wishing to be a reviewer ought to have a fair degree of competence with it already.

Second, take note at how people have been making use of the user pages. People write information about themselves, the articles they've created, and even whole essays about opinions or ideas.

What I'd propose is that we encourage people who wish to be reviewers to set up a subpage under their userpage called '/Approved?'. Any page that they added to this page is considered to be acceptable by them. (It is recommended they list the particular revision # they're approving too, but it's up to them whether to include the number or not.) The reviewer is encouraged to provide as much background and contact information about themselves on their main page (or on a subpage such as /Credentials?) as they wish. It is *completely* an opt-in system, and does not impact wikipedia as a whole, nor any of its articles.

Okay, so far it probably sounds pretty useless because it *seems* like it gives zero _control_ over the editors. But if we've learned nothing else from our use of Wiki here, it's that sometimes there is significant power in anarchy. Consider that whomever is going to be putting together the set of approved articles (let's call her the Publisher) is going to be selecting the editors based on some criteria (only those with phds, or whatever). The publisher has (and should have) the control over which reviewers they accept, and can grab their /Approved? lists at the time they wish to publish. Using the contact info provided by the reviewer, they can do as much verification as they wish; those who provide insufficient contact info to do so can be ignored (or asked politely on their userpage.) But the publisher does *not* have the power to control whether or not you or I are *able* to approve articles. Maybe for the "PhD? Reviewers Only" encyclopedia I'd get ruled out, but perhaps someone else decides to do a "master's degree or better" one, and I would fit fine there. Or maybe someone asks only that reviewers provide a telephone number they can call to verify the approved list.

Consider a further twist on this scheme: In addition to /Approved?, people could set up other specific kinds of approval. For instance, some could create /Factchecked? pages where they've only verified any factual statements in the article against some other source; or a /Proofed? page that just lists pages that have been through the spellchecker and grammar proofer; or a /Nonplagerized? page that lists articles that the reviewer can vouch for as being original content and not merely copied from another encyclopedia. The reason I mention this approach is that I imagine there will be reviewers who specialize in checking certain aspects of articles, but not everything (a Russian professor of mathematics might vouch for everything except spelling and grammar, if he felt uncomfortable with his grasp of the English language). Other reviewers can fill in the gaps (the aformentioned professor could ask another to review those articles for spelling and grammar, and they could list them on their own area.

I think this system is very in keeping with wiki philosophy. It is anti-elitist, in the sense that no one can be told, "No, you're not good enough to review articles," yet still allows the publisher to discriminate what to accept based on the reviewer's credentials. It leverages existing wiki functionality and Wikipedia traditions rather than requiring new code and new skills. And it lends itself to programmatic extraction of content. It also puts a check/balance situation between publisher and reviewer: If the publisher is selecting reviewers to include unfairly, someone else can always set up a fairer approach. There is also a check against reviewer bias, because once discovered, ALL of their reviewed articles would be dropped by perhaps all publishers, which gives a strong incentive to the reviewer to demonstrate the quality of their reviewing process and policies.

-- BryceHarrington

Magnus Manske's proposal:

I'll try to approach the whole approval mechanism from a more practical perspective, based on some things that I use in the Wikipedia PHP script. So, to set up an approval mechanism, we need:

- Namespaces to separate different stages of articles
- User rights management to prevent trolls from editing approved articles

From the Sanger proposal, the user hierarchy would have to be:

1. Sysops, just a handful to ensure things are running smoothly. They can do everything, grant and reject user rights, move and delete articles etc.
2. Moderators who can move approved articles to the "stable" namespace
3. Reviewers who can approve articles in the standard namespace (the one we're using right now)
4. Users who do the actual work ;)

Stages 1-3 should have all rights of the "lowerlevels", and should be able to "rise" other users to their level. For the namespaces, I was thinking of the following:

- The blank namespace, of course, which is the one all current wikipedia articles are in; the normal wikipedia
- An approval namespace. When an article from "blank" gets approved by the first reviewer, a copy goes to the "approval" namespace.
- A moderated namespace. Within the "approval" namespace, noone can edit articles, but reviewers can either hit a "reject" or "approve" button. "Reject" deletes the article from the "approval" namespace, "approve" moves it to the "moderated" namespace.
- A stable namespace. Same as for "approval", but only moderators can "reject" or "approve" an article in "moderated" namespace. If approved, it is moved to the "stable" namespace. End of story.

This system has several advantages:

- By having reviewers and moderators not chosen for a single category (e.g., biology), but by someone on a "higher level" trusting the individual not to make strange decisions, we can avoid problems such as having to choose a category for each article and each person prior to approval, checking reviewers for special references etc.
- Reviewers and moderators can have special pages that show just the articles currently in "their" namespace, making it easy to look for topics they are qualified to approve/reject
- Easy handling. No pop-up forms, just two buttons, "approve" and "reject", throughout all levels.
- No version confusion. The initial approval automatically locks that article in the "approval" namespace, and all decisions later on are on this version alone.
- No bother of the normal wikipedia. "Approval" and "moderated" can be blanked out in every-day work, "stable" can be blanked out as an option.
- Easy to code. Basically, I have all parts needed ready, a demo version could be up next week.

Discussion

Basically, I am not sure that we can generate enough interest, yet, on the part of "genuine experts" to act as reviewers for Wikipedia. That is my one big misgiving about this whole project. What do you think? --LMS

Dumb question: why do we need reviewers? So far, quality control seems to be a potential problem that as of yet shows no sign of turning into an actual problem.

See "advantages," above. :-) --LMS

This is not supposed to freeze the article, but, what is approved is a particular revision of the article. Will the approval be to revision n of the article, that the viewer of version n+m can check? --AN

Yes, and yes, or that's my notion of the thing. --LMS

I think this could be useful, but I have some vague misgivings about how well it will work in practice. Will we be able to get enough reviewers, who are actively involved? Who in the world is going to come up with the reviewer validation criteria? The problem here is that some articles are about SF authors, others are cookbook type recipes, and what makes an expert cook does not make an expert on Jamaican Jerk Chicken...

Beyond the logistical questions, I'm a bit worried that this may have some effects on wikipedia productivity. I'm sure one of the reasons that wikipedia thrives is just because it is easy to use. But I also think that there are delicate aspects to the way the community works and is structured which are just as important. If good people with lots of real knowledge feel like they are second class citizens, they will fell less motivated to work on the project. I'm not entirely certain that creating an official hierarchy will have no adverse effects. On the other hand I'm not certain that it will have adverse effects either...MRC

Related to this, is the idea that if I write an article on Jamaican Jerk Chicken, and do a thorough web-search which supports what I write, why can't I be considered an expert for the purposes of review. After all, I may have a more open mind than a lot of cooks out there.

To quote Lee Daniel Crocker from another page on Wikipedia, "Authority is nothing but a useful shortcut used in the world of humans because we haven't had the luxury of technology like this that makes expertise less relevant. What matters is the argument, not the arguers, and this technology supports--indeed enforces--that. Facts are facts, no matter who writes about them"

I know too much intellectual history to trust the experts much. Expertise quite often has more to do with trendiness than with knowledge and understanding. There are examples in almost any field - this generation's experts scoffs at the silly ideas of a previous generation, while destined to be scoffed at themselves by a future generation.

- Tim

Replies to the above:

Will we be able to get enough reviewers, who are actively involved?

That's an excellent question. I just don't know.

Who in the world is going to come up with the reviewer validation criteria?

That obviously would be a matter of some deliberation.

If good people with lots of real knowledge feel like they are second class citizens, they will fell less motivated to work on the project. I'm not entirely certain that creating an official hierarchy will have no adverse effects.

I agree that this is a very, very legitimate concern, and I think we probably shouldn't take any steps until we're pretty sure that the project wouldn't suffer in this way.

Authority is nothing but a useful shortcut used in the world of humans because we haven't had the luxury of technology like this that makes expertise less relevant. What matters is the argument, not the arguers, and this technology supports--indeed enforces--that. ... I know too much intellectual history to trust the experts much. Expertise quite often has more to do with trendiness than with knowledge and understanding.

I think there is some merit to these claims. But I'm wondering what this has to do with the proposal. Is the idea that we cannot trust authorities, or experts, to reliably state what constitutes a clear, accurate statement of human knowledge on the subjects within their expertise? Or is it, perhaps, that fine articles would be given a thumbs-down because they do not toe the party line sufficiently, whatever it is? Well, I don't know about that. Anyway, I'm not sure what the point is here.

Gotta go, the dog is taking me for a walk. --LMS

Another possible problem is that even the experts can be wrong. How can we verify accuracy, then? Even if the Wikipedia was internally consistent, it can still be wrong. Not only that, but an article can change 30 seconds after it has been reviewed. For starters, we'll have to introduce another concept from programming, the code freeze. What we can do is analyze the change frequency on articles (via a program of some kind), and when the changes in an article stabilize so that changes are minor and infrequent, we copy the article, add all the named authors as authors, date it, and make it a static page on the wikipedia. Then you'll have two articles, one is the latest "stable" revision, and the other is open to flux (the rest are archived, available by request or some other equivalent). The tough part is determining "accuracy". We could go democratically and add a voting system, but that has problems since the majority can be wrong about as easily as an individual. Any verification system either requires money to hire people to check on authors' claims of expertise, or the creation of an elite class of authors. The alternative is to foster the sense of community, and work on the trust a wikipedian can earn from fellow wikipedians, but that opens up the door to any mistakes made by a trusted wikipedian being tougher to correct. So I would tend to think that the best argument for the validity of an article is stability in the face of hits. Perhaps do something like this:

A = (n_r/(n_h %Δ)) * (√n_a)/T

where A is the accuracy factor, nr is the number of revisions (since some time), %f¢ is the median (or mean if you must) % change in the article per revision, nh is the total number of hits, T is the technical factor (could be determined by the number of authors involved, etc. it is essentially an attempt at determining the odds that a hit will know enough to revise the article), and na is the number of authors involved (under a radical so that it won't increase linearly). This equation is frightfully arbitrary except in the factors it considers, and a statistician should come up with a better form.

Magnus' implementation of my proposal looks good, but it doesn't quite implement my proposal. The role of a moderator is to approve that a reviewer has the billed (and necessary) qualifications. I don't want anyone standing over the reviewers in the sense of saying, "Yes, you were right to approve this article." In fact, a moderator could very well know nothing about the subject the reviewer addresses, but the moderator can check to see whether the reviewer does have the necessary qualifications (by visiting homepages and matching up e-mail addresses, etc.). In other words, the role of reviewers is quality assurance, whereas the role of moderators would be anti-reviewer-fraud assurance.

Otherwise, the implementation looks pretty good. This advantage is important: "By having reviewers and moderators not chosen for a single category (e.g., biology), but by someone on a "higher level" trusting the individual not to make strange decisions, we can avoid problems such as having to choose a category for each article and each person prior to approval, checking reviewers for special references etc." That's exactly why I wanted it designed this way. Someone could be an ad hoc expert about his pet subject, and a moderator might be able to spot this.

I think ...'s (got to change that nickname, guy! :-) ) proposal really pales beside Bryce's. If we are going to have a "community approval" process, Bryce's is far superior, because it allows us to "approve the approvers." Frankly, I couldn't give rat's patoot whether lots of people would approve of a given article. I want to know whether people who know about the subject (i.e., by definition, the people I'm calling experts) approve of it.

If we did go my route, as opposed to Bryce's, I think we should have an in-depth discussion of criteria for reviewers. Basically, I think we should use criteria similar to those used by Nupedia, but modified to allow for specific expertise on specific subjects--where such expertise might not be codified in degrees, certificates, etc. Nevertheless, I think that the expertise even in those cases must be genuine. If you've read a half-dozen books on a subject of the breadth of, say, World War II, then you know a heck of a lot about WWII, and you can contribute mightily, but you ain't an expert on WWII (probably). Essentially, if we want to adopt a review mechanism in order to achieve the goals of attracting more, well, experts, and in order to have Wikipedia's content used by various reputable online sources, then we must work with the concept of expertise that they use. One rough-and-ready conception of expertise goes like this: you identify people who are experts on a subject on anybody's conception; then you determine who those people consider colleagues worth speaking to seriously and professionally. Those are the experts on that subject.

Frankly, this whole thing is starting to give me a bit of a headache, and I'm not sure we should do anything at all anytime soon. :-) --Larry Sanger

On Bryce's proposal: this is very interesting and I think we should think more about it. Maybe it would end up being a roaring success. In the context of Wikipedia, I can pretty easily imagine how it could be. There is one main problem with it, though, and that is that it isn't going to make the project any more attractive for high-powered academic types to join the club. They would very likely look on such a system as a reflection of a sort of hopeless amateurism that will doom Wikipedia to mediocrity. We know better, of course--but we would like to have the participation of such people. Or I would, anyway. They know stuff. Stuff that we don't know, because they're smart and well-educated. If we can do something that's otherwise non-intrusive to the community to attract them, we should. Another problem, related to this, is that the world isn't going to be as excited about this cool system as we are. They'll want to see stuff that is approved, period--presented by Wikipedia as approved by genuine experts on the subjects. If they can see this, they'll be a lot more apt to use and distribute Wikipedia's content, which in the end is what we really, really want--because it means world domination. :-)

After some more thought, I'm now thinking, "Why can't we just adapt Bryce's proposal for these (elitist) purposes?" It would go something like this. We all have our own locked pages on which we can list articles of which we approve. There is a general rule that we should not approve of articles in areas on which we aren't experts. Then, as Bryce says, people can choose who to listen to and who not to listen to when it comes to approvals. But as for presenting the Wikipedia-approved articles, it would be pretty straightforward: some advisory board of some sort chooses which people are to be "listened to" as regards approvals. This would be determined based on some criteria of expertise and whether the approvals the person renders are in that person's areas of expertise. Then we could present one set of articles as the "Wikipedia-approved" articles. Other people could choose a different set of reviewers and get a different set of approved articles.

Moreover, we could conceivably make Bryce's system attractive to "experts." We could say: "Hey, you join us and start approving articles, and definitely your approval list will definitely be one to help define the canonical set of Wikipedia-approved articles.

This looks very promising to me. Right now, I'd have to say I like it better than Sanger's proposal! --Sanger

First, let me say that there's no technical problem in implementing both the Bryce and Sanger/Manske proposal. Just as different category schemes can coexist peacefully at wikipedia, these could too. We could even use the "expert verification" (no matter how this will work in the end) for both approaches.

For the difference between the Sanger and the Manske proposal about what moderators do, you should think about this: Say I get to become a reviewier because I know biology and a little about computers;) So, a moderator made me reviewer. What's going to stop me from approving a two-line-article about "Fiddle Traditions in General"? If I'd be restricted to biology and computers, we'd have to put all articles into categories (which we don't want), otherwise the moderators would have to check every approved article, which is what I suggested in the first place. Maybe I wasn't clear on this point: I don't want the moderators to check articles that were approved by reviewers for scientific correctness; they should just act as another filter, basically approving every article they get from the reviewers, except for those with obvious errors, or with "unfitting" topics, such as foobar... --Magnus Manske

Actually, Magnus, I later came around to your thinking and neglected to mention it. I.e., I think it would be better to have the moderators always be working to check adequate qualifications.

The other possibility is to have some way to "undo" illicit approvals. This would be an enormous headache, though--anyone whose approval was undone would probably quit. --LMS

I lean toward something like Bryce's suggestion as well where "approved-ness" is just another piece of metadata about an article that can be used to select it, but I'd simplify it even further with a little software support. Let's not forget Wikipedia's strength: it's easy to create and edit stuff. Because of that, we have a lot of stuff that's been created and edited. We need to make it absolutely trivially easy to provide metadata about an article using the same simple Web interface. For example, have an "Approvals" or "Moderation" link which takes the user to a fill-in form where he checks boxes to answer questions like "Is this article factually accurate in your opinion?", "Is this article clear and well-written in your opinion?", "Does this article cover all major aspects of its topic in your opinion?", etc. That information can be stored in the database associated with the appropriate revisions (perhaps the software could even retain article versions for longer if they have a certain level of approval). Storing that info under the page of the Wikipedian who filled in the form as suggested by Bryce (except under program control) is a good way to do it. Then, users can judge for themselves whose opinions they value and whose they don't. This option is available to anyone who is logged in as a specific user (and not to anonymous users), so the software would know to update the "Lee Daniel Crocker/Well written" page when I checked that box.

This software could be very simple--just present the form, and add lines to the appropriate page, which is just an ordinary Wiki page. The "Lee Daniel Crocker/Well written" page, the "Larry Sanger/Copyright status verified" page, the "Magnus Manske/Factually accurate" page, and the "Bryce Harrington/Interesting subject" page are themselves subject to approval by anyone, and their value can be judged by that.

--Lee Daniel Crocker

Implementing an internal equivalent of something like Google's PageRank might be a useful alternative to the approval system. This would effectively rate an article based on how well linked it is and how well ranked the items that link it are. Not only does this scale well (as Google has shown), and work automatically, it is effectively the equivalent to the mental heuristic humans use to establish authority (that is, an expert is one because other people say he is).

In my opinion there should be a separate project that is quality controlled and has approved articles. By having a separate domain and project name, people on wikipedia automatically see the latest revision, while the people at the new site always see the approved revision. With my idea, to make it simpler for newbies, there would be just one affiliated approving project. The staff would consist of administrators, qualifiers, and experts.

To be an expert, you would send in your qualifications such as your degree, job experience, articles written, etc. The necessary qualifications would be fairly small, but your qualifications are assigned a number from 1-5 (which is always displayed in front of your name), and people may filter out articles approved only by barely qualified experts.

The qualifiers contact the university where the expert got their degree, check the archives to see if the expert really wrote the article they claimed to write, etc. The administrator's sole duty is to check the work of the qualifiers and to approve new qualifiers.

Under my idea, any articles that are "stable" (as defined by the formula above) are considered community approved, and as such, are sent to a page on the project (much like the recent changes page) which does not show up in search results and is out of the way, but accessible to anyone.

To actually get approved, the article must be checked for plagiarism by a program, grammar checked by an expert in the language the article is in, and finally, get checked by experts in the subject matter. When the article finally gets approved, all of the registered users who edited the article are invited to add their real or screen name to the list of authors, and the experts get their names and qualifications added in the experts' section at the top of the article.

Even when the article gets approved, people can still interact with it. Any user can submit a complaint about an article if they feel it is plagiarizing or is biased, or they can submit a request to the grammar checkers to fix a typo, and experts may and are encouraged to put their stamp of approval on an already approved article.

An article is rated the same as the highest expert that approves the article. For example, one searching for Brownies might see an article that is titled like this:

4:Recipe for Brownies

This means that at least one expert rated a four approved of the article. A person searching for an article specifies how low they are willing the ratings to go.

-- Techieman