Jump to content

Knowledge Engine (search engine)

From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by Sj (talk | contribs) at 13:02, 14 July 2020 (update and ce /1). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.
A screenshot of the Knowledge Graph surrounding a Wikipedia article.
The search example of the Knowledge Engine states "Ad-free, secure, non-profit: Make Wikipedia your default search".[1]

Knowledge Engine (KE) was a search engine project initiated in 2015 by the Wikimedia Foundation (WMF) to locate and display verifiable and trustworthy information from public-information sources[2] in a way that was less reliant on traditional search engines.[3] It aimed to allow readers to stay on Wikipedia.org and other Wikipedia-related projects when looking for additional information rather than returning to proprietary search engines.[3] It's goal was to protect user privacy, to be open and transparent about how a piece of information originates, and to allow access to related metadata.[4]

The development of the project idea was controversial internally, and was not pursued after 2016. Related ideas were applied to the internal cross-wiki search engine for Wikimedia projects.[1]

History

In 2015, WMF applied for a $250,000 grant from the Knight Foundation to support development of the Knowledge Engine. Its grant proposal noted: "Commercial search engines dominate search-engine use of the Internet, and they're employing proprietary technologies to consolidate channels of access to the Internet's knowledge and information."[5] The project was designed in four stages, each scheduled to take about 18 months.[6]

The project planned to draw information from Wikipedia-related projects and may eventually search other sources of public information such as the U.S. Census Bureau.[5] Leaked internal WMF documents stated the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation."[2] The WMF clarified that it was not intended to be a universal search engine,[5] though it might in time also include academic and open access sources in its search results.[7] Matt Southern in Search Engine Journal attributed media confusion about the Knowledge Engine's scope to the fact that this was "quite a contrast to the original grant application documents".[8]

The project was not discussed publicly with the Wikipedia community while developing the concept,[9] and the prospective project was not mentioned in the WMF's publicly available annual plan.[10] This secrecy was seen as at odds with the goal of transparency.[2] In a text posted on the English Wikipedia's community newsletter, The Signpost, James Heilman, who was dismissed from the WMF's Board of Trustees in December 2015, said he had insisted multiple times that the Knight Foundation grant documentation be made public, without success,[11] and suggested that his push for transparency concerning the grant had been a factor in his dismissal—a suggestion rejected by Jimmy Wales.[2] The Wikipedia community re-elected Heilman to the Board in 2017.[12] Critics say the project's roll out illustrates a disconnect and lack of understanding between a foundation that's increasingly run by people connected with Silicon Valley, and the volunteer community of editors who worry that KE may reflect a change in the WMF's focus from user-generated content to one led by automated data results.[2]

An initial blogpost published by the WMF Executive Director Lila Tretikov and the WMF regarding the project did not address why the original grant application was so much broader than merely developing an internal search engine.[13] Tretikov's employees said the WMF was still not being straightforward with the Wikipedia community.[14] WMF Executive Director Tretikov resigned on February 25, 2016, as a result of the controversy.[13][15] Former Deputy Director of the WMF Erik Möller described the events as "a crisis."[1]

Motivation

A screenshot of Google Knowledge with fast facts from a Wikipedia article.
Since mid-2012,[16] Google Search has included fast facts from Wikipedia articles on its search results pages[2] via the Google Knowledge Graph.[16]

The project proposal asked, "Would users go to Wikipedia if it were an open channel beyond an encyclopedia?"[1] A central source of confusion for the project was the extent to which it would directly compete with traditional search engines as a place to search the Web. According to Vice, "the Wikimedia Foundation, the nonprofit that finances and founded Wikipedia, is interested in creating a search engine that appears squarely aimed at competing with Google."[2] According to The Guardian, "there was considerable doubt over what the tool was actually intended to be: a search engine aimed at halting a decline in Wikipedia traffic sent by Google, or simply a service for searching within Wikipedia?"[13]

Since 2012, Google Search and other search engines had started highlighting brief informational summaries from Wikipedia in knowledge panels alongside search results, reducing traffic to Wikipedia from those search engines.[2] According to Search Engine Watch, this led to a battle for attention,[1] and this project could have recouped some of that traffic.

Although the Wikimedia Foundation (WMF) may not have been attempting to create a general web search engine, the goal of the Knowledge Engine was to let readers and editors be less reliant on proprietary search engines when looking for new information.[3]

Development

Seemingly out of line with WMF's usual "radical transparency", information about the project only became public gradually.[2] As early as May 2015, community members asked about the concentration of staff in a new "Search and Discovery" department, even though public plans made little or no reference to this work.[2][17] The WMF was awarded a $250,000 grant for the project in September 2015 from the Knight Foundation, which was publicly announced in a January 2016 press release.[3] A budget submitted by the WMF included in the grant press release gave $3,421,672 to cover costs, but the WMF acknowledged it had only received $250,000 in financing as of mid-February 2016.[18]

The WMF initially published only portions of the grant documentation,[19] but made the full grant agreement available on February 11, 2016.[11] Further internal documents were leaked shortly after.[2][9] The project comprises four stages, each scheduled to take about 18 months: Discovery, Advisory, Community and Extension.[6] The initial stage of the project is budgeted to cost $2.5 million,[20] the whole potentially running to the tens of millions.[1] After a year, the WMF was to evaluate the development and at the close of the grant, set plans for the project to continue to the second stage, according to the grant document.[6]

Function and purpose

A screenshot of potential sources used by the Knowledge Engine.
Example of federated data sources potentially used by the Knowledge Engine.

The Knowledge Engine was designed to be open and transparent about how a piece of information originates and allow access to metadata.[4] It would have no advertisements, protect user privacy, and emphasize community building and sharing of information.[4] It would draw information from Wikipedia-related projects and may eventually search other sources of public information such as the U.S. Census Bureau,[5] OpenStreetMap,[21] the Digital Public Library of America,[17] and external sources like Fox News.[1] Jimmy Wales and the WMF stated that the project would focus on improving search on Wikipedia and related Wikimedia projects.[2] The grant application stated that it would "create a model for surfacing high quality, public information on the internet."[2] It also advised that "commercial search engines dominate search-engine use of the internet" and states that "Google, Yahoo, or another big commercial search engine could suddenly devote resources to a similar project, which could reduce the success of the project."[2]

Leaked internal documents framed this plan more boldly,[22] stating that the "Knowledge Engine By Wikipedia will democratize the discovery of media, news and information—it will make the Internet's most relevant information more accessible and openly curated, and it will create an open data engine that's completely free of commercial interests. Our new site will be the Internet's first transparent search engine, and the first one that carries the reputation of Wikipedia and the Wikimedia Foundation."[2]

The apparent contradiction between different descriptions of the purpose led to confusion in the media and in the community. In response to speculation, the WMF published a response clarifying its intentions: "We're not building a global crawler search engine ... Despite headlines, we are not trying to compete with other platforms, including Google. As a non-profit we are noncommercial and support open knowledge. Our focus is on the knowledge contributed on the Wikimedia projects. ... We intend to research how Wikimedia users seek, find, and engage with content. This essential information will allow us to make critical improvements to discovery on the Wikimedia projects."[8] Director of Discovery Tomasz Finc added "we are building an internal search engine, and we are not building a broad one.[1] Jimmy Wales stated that suggestions that the WMF is creating a rival to Google are "trolling", "completely and utterly false", and "a total lie".[2][11] while allowing that the Knowledge Engine might in time include academic and open access sources in its search results.[7]

Matt Southern in Search Engine Journal attributed media confusion about the KE's scope to the fact that this was "quite a contrast to the original grant application documents",[8] an assessment echoed by James Vincent in The Verge,[9] Matt McGee in Search Engine Land,[23] and Jason Koebler in Vice.[24]

Controversy

Many in the community were furious that details of such a large project had been withheld by an organization that prides itself on radical transparency. Wikimedia's public story—that it was never working on a search engine—was directly contradicted by a grant proposal made to the Knight Foundation and leaked internal documents.

 —Jason Koebler, Vice[14]

Ruth McCambridge said in Nonprofit Quarterly, "Wikipedia editors have been requesting from December for the grant proposal and grant letter for a project that many surmise is a bid to remain technologically cutting-edge by the Wikimedia Foundation, but which may divert resources and attention from other pressing needs of the community."[25] Generally, large-scale WMF projects such as KE are discussed publicly with the Wikipedia community, but this did not happen with the KE project[9] as Wikipedia community volunteers were initially unaware of the existence of the KE.[2][26] Many Wikipedians expressed outrage at what they perceived to be the secrecy around the KE project and their lack of ability to give input, according to the English Wikipedia's community newsletter, The Signpost.[25] The lack of community involvement raised questions about WMF's commitment to transparency with the Wikipedia community.[9] Wikipedia editors noted that the KE project was not published in the WMF's publicly available annual plan.[10]

Commenting on the WMF's Executive Director Lila Tretikov's reluctance to post the donor documents to the volunteer community, referencing privacy concerns, McCambridge sees "a major difference in culture and values assumptions" compared to previous Wikimedia practice.[25] McCambridge said that "the power of important strategic decisions" now rests "between funders and the top of the organizational hierarchy" and is "not shared with volunteer editors."[25] James Heilman, who was dismissed from the WMF's Board of Trustees in late 2015, said that while on the Board, he had pushed for greater transparency of the Knight Foundation grant documentation and its financing,[27] and suggested that his push for transparency concerning the grant had been a factor in his dismissal – a suggestion Jimmy Wales rejected as "utter fucking bullshit".[2] It appears the Wikipedia community disagreed.[12] The Wikipedia community re-elected Heilman to the Board in 2017.[12] Tretikov eventually released the Knight Foundation grant in February 2016 which disclosed the first stage of the KE project.[25] Tretikov said that she regretted being so late in informing the Wikipedia editing community about the Knight Foundation grant.[17]

Longtime Wikipedia editor and journalist William Beutler told Vice Magazine's Jason Koebler, "Leaving aside whether a search engine is a good idea, let alone feasible, the core issue here is about transparency. The irony is that the Wikimedia Foundation failed to observe one of the movement's own core values ...."[24] A UK Wikipedia editor Ashley van Haeften told Ars Technica via e-mail that "Lila, Jimmy, and the rest chose to keep the project and the Knight Foundation application and grant a secret until the projects were underway for six months, and even then this only came to light because it was leaked."[22] Critics say the project's roll out illustrates a disconnect and lack of understanding between a foundation that's increasingly run by people connected with Silicon Valley, and the volunteer community of editors who worry that KE may reflect a change in the WMF's focus from user-generated content to one led by automated data results.[2]

After the leaked documents, many on the Wikimedia-L mailing list were calling for Tretikov's resignation.[14] The initial blogpost published by Tretikov and the WMF regarding the KE project still did not adequately explain why the original grant application documents appeared so much larger than merely developing an internal search engine.[13] After that blogpost, calls for Tretikov's resignation heightened, because her employees said the WMF still was not being straightforward with the Wikipedia community.[14] "My concern is that we still aren't communicating it clearly enough. This morning's blog post is the truth, but not all of the truth. Namely that we had big plans in the past. It would have been much easier to say that we did have big plans, but they were ditched ... there is clear evidence of something, but we still haven't acknowledged it. We can't deny it", Max Semenik, a Discovery team member explained to Tretikov, according to statements posted of an internal meeting on the WMF's website.[14]

Many WMF staff members have departed as a result of the KE project.[28] After several weeks of crisis,[27] the controversy culminated in WMF Executive Director Tretikov resigning on February 25, 2016.[13][15] Former Deputy Director of the WMF Erik Möller, up to April 2015, portrayed the recent events as "very much out of control" and "a crisis."[1] Wales had previously attempted to create a search engine, but Wikia Search failed in 2009.[18]

References

  1. ^ a b c d e f g h i Sentance, Rebecca (March 3, 2016). "Everything you need to know about Wikimedia's 'Knowledge Engine' so far". Search Engine Watch. Archived from the original on January 13, 2017.
  2. ^ a b c d e f g h i j k l m n o p q r s Koebler, Jason (February 16, 2016). "The Secret Search Engine Tearing Wikipedia Apart". Vice. Archived from the original on February 23, 2016.
  3. ^ a b c d McGee, Matt (February 15, 2016). "Wikimedia Foundation Secures $250,000 Grant For Search Engine Development". Search Engine Land. Archived from the original on May 23, 2016.
  4. ^ a b c Singh, Manish (February 16, 2016). "Wikipedia's Upcoming Search Engine to Rival Google; Offer Full Transparency". Gadgets 360. Archived from the original on February 16, 2016.
  5. ^ a b c d Cuthbertson, Anthony (February 16, 2016). "Wikipedia Takes on Google with New 'Transparent' Search Engine". Newsweek. Archived from the original on February 16, 2016.
  6. ^ a b c Crum, Chris (February 15, 2016). "Wikimedia Works On Search Improvements, Says It's Not Competing with Google [Updated]". WebProNews. Archived from the original on July 3, 2016. Retrieved February 17, 2016.
  7. ^ a b Greis, Friedhelm (February 15, 2016). "Wirbel um angebliche Wikipedia-Konkurrenz zu Google". Golem.de (in German). Archived from the original on February 17, 2016.
  8. ^ a b c Southern, Matt (February 17, 2016). "Wikimedia Clarifies it is Not Building a Global Web Crawler". Search Engine Journal. Archived from the original on February 18, 2016.
  9. ^ a b c d e Vincent, James (February 17, 2016). "Wikimedia says it's not building a search engine to take on Google". The Verge. Archived from the original on September 5, 2017.
  10. ^ a b McCormick, Rich (February 26, 2016). "Wikimedia head resigns after leak exposed search engine plans". The Verge. Archived from the original on April 14, 2017.
  11. ^ a b c Tual, Morgane (February 16, 2016). "Un projet de moteur de recherche sème la discorde chez Wikipedia". Le Monde (in French).
  12. ^ a b c Andreas Kolbe (7 June 2017). "Golden handshakes of almost half a million at Wikimedia Foundation". The Register. Archived from the original on 10 October 2017.
  13. ^ a b c d e Hern, Alex (February 26, 2016). "Head of Wikimedia resigns over search engine plans". The Guardian. Archived from the original on March 28, 2016.
  14. ^ a b c d e Koebler, Jason (February 25, 2016). "Wikimedia Foundation Executive Director Resigns Amid a Community Revolt". Vice. Archived from the original on February 26, 2016.
  15. ^ a b "Online-Enzyklopädie: Chefin der Wikipedia-Stiftung tritt zurück". Spiegel Online (in German). February 26, 2016. Archived from the original on March 5, 2016.
  16. ^ a b Orlowski, Andrew (January 14, 2014). "Google stabs Wikipedia in the front". The Register. Archived from the original on November 13, 2017.
  17. ^ a b c Kleinz, Torsten (February 15, 2016). "Wikipedia plant Suchmaschine, aber keinen Google-Konkurrenten". Heinz Heise (in German). Archived from the original on February 17, 2016.
  18. ^ a b "Knowledge Engine: Wikimedia Foundation takes aim at Google with $US2.5m search project". Radio Australia. February 17, 2016. Archived from the original on March 6, 2016.
  19. ^ Orlowski, Andrew (February 11, 2016). "Move over, Google. Here's Wikipedia's Search Engine – Full of On-Demand Smut". The Register. Archived from the original on July 8, 2017.
  20. ^ Orlowski, Andrew (February 12, 2016). "Reluctant Wikipedia lifts lid on $2.5m internet search engine project". The Register. Archived from the original on September 1, 2017.
  21. ^ Shah, Jaymi (February 16, 2016). "Wikimedia Foundation Secures $250,000 Grant For Search Engine Development". Technoledger. Archived from the original on March 2, 2016.
  22. ^ a b Mullin, Joe (February 29, 2016). "Wikimedia Foundation director resigns after uproar over "Knowledge Engine"". Ars Technica. Archived from the original on March 1, 2016.
  23. ^ McGee, Matt (February 16, 2016). "Wikimedia Foundation: "We're Not Building A Global Crawler Search Engine"". Search Engine Land. Archived from the original on February 17, 2016.
  24. ^ a b Koebler, Jason (February 16, 2016). "Wikimedia: We're Really Really Not Building a Search Engine". Vice. Archived from the original on February 17, 2016.
  25. ^ a b c d e McCambridge, Ruth (February 16, 2016). "Knight Foundation Grant Request Tears at Wikipedia's Community". Nonprofit Quarterly. Archived from the original on February 24, 2016.
  26. ^ Singh, Manish (February 16, 2016). "Knowledge Engine: Wikimedia Foundation takes aim at Google with $3.5m search project". ABC News (Australia). Archived from the original on February 16, 2016.
  27. ^ a b Noisette, Thierry (February 26, 2016). "Crise à la fondation Wikimedia : sa directrice démissionne". L'Obs (in French).
  28. ^ Price, Rob (February 26, 2016). "The executive director of the nonprofit behind Wikipedia just resigned". Business Insider. Archived from the original on February 28, 2016.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Knowledge_Engine_(search_engine)&oldid=967646077"