View source for Wikipedia:Bots/Requests for approval/LkolblyBot

<noinclude>[[Category:Denied Wikipedia bot requests for approval|LkolblyBot]]</noinclude><div class="boilerplate metadata" style="background-color:
#FFEAEA; margin:2em 0 0 0; padding:0 10px 0 10px; border:1px solid #AAAAAA;">
:''The following discussion is an archived debate. <span style="color:red">'''Please do not modify it.'''</span> To request review of this BRFA, please start a new section at [[WT:BRFA]].'' The result of the discussion was {{BotDenied}}<!-- from Template:Bot Top-->

==[[User:LkolblyBot|LkolblyBot]]==
{{Newbot|LkolblyBot|}}
<!-- ^^^ Ignore everything above this line ^^^ -->

<!-- Leave this alone -->
'''Operator:''' {{botop|Lkolbly}}

<!-- And leave this alone -->
'''Time filed:''' 18:57, Monday, December 24, 2018 ([[UTC]])

<!-- A SHORT bot function overview (max 2–3 lines); place the in-depth explanation in the "function details" section below-->
'''Function overview:''' This bot automatically updates Alexa rankings in website infoboxes by querying the Alexa Web Information Service.

<!-- Automatic: edits are not reviewed, except when issues arise -->
<!-- Supervised: all edits are reviewed soon after being made -->
<!-- Manual: all edits are reviewed before being made -->
'''Automatic, Supervised, or Manual:''' Automatic

<!-- Programming Language: The computer language that this bot will be written in. E.g. Python, Java, C, VB, AutoWikiBrowser. NOT English etc. -->
'''[[Programming language]](s):''' Python

<!-- Source code available: e.g. a link to the source code, "Standard pywikipedia"/"AWB"/etc. Be sure the bot account's password is not given out! -->
'''[[Source code]] available:''' https://github.com/lkolbly/alexawikibot (presently, the actual saving is commented out, for testing)

<!-- Bot tasks require consensus in order to be approved. Please list any relevant discussions here to indicate consensus for the task. If such input is not necessary (for instance, a task that is duplicating or closely matching an existing bot) leave this blank-->
'''Links to relevant discussions (where appropriate):''' Previous bot that performed this task: [[Wikipedia:Bots/Requests_for_approval/OKBot_5|OKBot_5]]

<!-- e.g. Continuous, daily, one time run, etc. -->
'''Edit period(s):''' Monthly or so

<!-- Should be a reasonable guess as to how many distinct pages you'll be editing. For open-ended tasks, estimate pages per some reasonable time period.-->
'''Estimated number of pages affected:''' 4,560 articles are in the current candidate list. A subset of these pages will be updated each month. Other pages could be pulled into the fray over time if someone adds alexa information to a page. Also, there will be a whitelist copied from [[User:OsamaK/AlexaBot.js]] of pages that will be edits (presently containing 1,412 pages).

<!-- Which namespace(s) will the bot edit? Mainspace/Articles, Categories, Files, ...-->
'''Namespace(s):''' Articles

<!-- Whether the bot skips articles and other pages with the {{bots}} template. If "no", please provide a reason in the Function details or Discussion.-->
'''[[Template:bots|Exclusion compliant]]''' ''(Yes/No)''''':''' Yes (via whatever functionality is already in pywikipedia)
<!-- Bots with Administrative powers are rare and run by Users who are already Administrators -- >

'''Adminbot''' ''(Yes/No)''''':''' Yes <!-- close the comment above to enable -->

<!-- List full and complete function details here. Please be precise and explicit, describing all changes the bot would make, including those "bundled" with bot frameworks like AWB or pywikipedia genfixes. Bots cannot be approved for open-ended tasks, so ensure the details cover all cases. Consider making several BRFAs for tasks with large independent changes. Vague or incomplete details can delay the BRFA process. Straight-forward details and examples will speed up the approval. If you need to modify these details after a discussion starts, it is recommended that you use del tags – <del></del> – to remove text and ins tags – <ins></ins> – for additions (this preserves coherence of the discussion), or show/hide boxes as you see fit. -->
'''Function details:''' 
This bot will scan all pages (using a database dump as a first pass) to find pages which have the "Infobox website" template with both "url" and "alexa" fields.

It will parse the domain from the url field using a few heuristics, and query the domain with AWIS. Domains that have subdomains return incorrect results from AWIS (e.g. mathmatica.wolfram.com returns the result for just wolfram.com), so these domains are discarded (and the page not touched). It will then perform an AWIS query to determine the current website rank and trend over 3 months.

Websites will be classified into <nowiki>{{Increase}}</nowiki>, <nowiki>{{Decrease}}</nowiki>, and <nowiki>{{steady}}</nowiki> ({{Increase}}, {{Decrease}}, and {{steady}}, respectively). A site increasing in popularity will gain it the {{Increase}} tag, even though it is numerically decreasing (previously, many sites were also classified into IncreaseNegative and DecreasePositive that I didn't understand)

Then, in the text of the article, whatever the current alexa data is will be replaced by something like:

<code><nowiki>{{Increase}} 169,386 ({{as of|2018|12|24}})<ref name="alexa">{{cite web|url= http://www.alexa.com/siteinfo/darwinawards.com | publisher= [[Alexa Internet]] |title=Darwinawards.com Traffic, Demographics and Competitors - Alexa |accessdate= 2018-12-24 }}</ref> <!-- Updated monthly by LkolblyBot --></nowiki></code>

(e.g. {{Increase}} 169,386 ({{as of|2018|12|24}})<ref name="alexa">{{cite web|url= http://www.alexa.com/siteinfo/darwinawards.com | publisher= [[Alexa Internet]] |title=Darwinawards.com Traffic, Demographics and Competitors - Alexa |accessdate= 2018-12-24 }}</ref> <!-- Updated monthly by LkolblyBot -->)

There are two as-yet untested test cases that I'll test (and fix if necessary) before any full-scale deployment:
* Apparently some infoboxes have multiple <code>|alexa=</code> parameters? I have to go find one and see what the bot does with it. (probably the right thing to do is to not touch the page at all in that situation)
* Some pages have an empty <code>|alexa=</code> parameter, which should be fine, but worth testing anyway.
{{reflist-talk}}

===Discussion===
<!-- This is not a vote.  It is a discussion -->

Please make the bot's talk page.

"''whatever the current alexa data is will be replaced''" - how do you know there isn't more than just the previous value? Or that there isn't a reference that is used elsewhere?

I imagine many pages that copy-paste the template code will have an empty <code>|alexa=</code> parameter. This would not be any different to not having it at all.

Do you preserve template's formatting?

The particular citation style the bot uses may not match the article's, especially the date format. (I wonder why we don't have an Alexa citation template still.) —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color: #B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 21:26, 24 December 2018 (UTC)

: The format of the template code overall is preserved, the value is replaced by replacing the regex <code>r"\|\s*alexa\s*=\s*{}".format(re.escape(current_alexa))</code>, so the rest of the template is unaffected. (the number of spaces before the equal sign goes from "any number" to "exactly one", though)

: Yeah, I was debating having it skip empty alexa parameters. There's value in adding it (as much as updating it), though for very small sites the increase/decrease indicator may not be particularly useful.

: I didn't think to check whether there's more than the previous value, though I can't think of what else would be there. There's at least two common formats for this data, basically the OKBot format, and a similar format with parenthesized month/year instead of the asof (see https://en.wikipedia.org/wiki/Ethnologue - note lack of a reference). I guess it would be safest to check that the value is in a whitelisted set of alexa formats to replace, I'll bet a small number of regexes could cover 90% of cases (and the remaining 10% could be changed to a conforming case by hand :D)

: The reference is interesting, because it's basically a lie. It's a link to the alexa page, but that isn't where the data was actually retrieved from, it was retrieved from their backend API. As for if someone's already using that reference, it shouldn't be too hard to check for that, I would think. I imagine (with only anecdotal evidence) that most of those cases will be phrases like "as of 2010, foobar.com had an alexa rank of 5". Updating that reference to the present value may not make sense in the context of the article (myspace isn't as big as it used to be, an article talking about how big it was in 2008 won't care how big it is now). But either way they should probably be citing a source that doesn't automatically change as soon as you read it.

: The ethnologue page already looks like it has diverging date formats? I don't know how common that is, I'll have to go dig up the style guide for citations (maybe we should have a bot to make that more uniform). What would it take to make a template? (also, would that solve the uniformity issue? I guess at least it'd be uniform across all alexa rankings)

: [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 14:52, 25 December 2018 (UTC)

:: [[WP:CITEVAR]] and [[WP:DATEVAR]] is the relevant stuff on date and citation differences. On English wiki, changing or deviating from citation or date style without a good reason is very controversial. The short answer is "don't". Bots are generally expected to do the same, although minor deviations are somewhat tolerated. But bots are expected to follow templates, like {{tl|use dmy dates}} or {{para|df}} parameters. —&nbsp;<small>&nbsp;[[user:Hellknowz|<span style="color: #B00;">HELL</span>KNOWZ]]&nbsp;&nbsp;&nbsp;▎[[User talk:Hellknowz|TALK]]</small> 16:36, 25 December 2018 (UTC)

::: Okay, it looks like it should be pretty straightforward to just check for the two <code><nowiki>{{use ... dates}}</nowiki></code> tags and set the {{para|df}} parameter. [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 14:39, 26 December 2018 (UTC)

::: Updated the bot so that it follows mdy/dmy dates, updating the accessed date and asof accordingly. Also constrained the pages that will be updated to a handful of matching regexes and also pulled a list from [[User:OsamaK/AlexaBot.js]], which eventually I'll make a copy of. [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 18:20, 1 January 2019 (UTC)
*{{BotTrial|edits=50}} [[User:Primefac|Primefac]] ([[User talk:Primefac|talk]]) 00:43, 20 January 2019 (UTC)

:{{BotTrialComplete}} Ran bot to edit 50 randomly selected pages. So far I've noticed two bugs that cropped up, one involving leading zeros in the access dates and another where the comment "Updated by LKolblyBot" got repeated. I'm going to go through and fix the issues by hand for now and apply fixes to the bot. [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 20:20, 27 January 2019 (UTC)

: Also, looking closer, some pages got a "Retrieved" date format that doesn't match the rest of the page (e.g. [[Iraqi News]]), but I'm pretty sure it's because those pages aren't annotated with dmy or mdy. [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 20:47, 27 January 2019 (UTC)
::I have questions.
::*First, [[Special:Diff/880480890]] - is there a reason it chooses http over https?
::*Second, why do [[Special:Diff/880480302|some diffs]] use ISO formatting for the date while others [[Special:Diff/880480336|actually change it to dmy]]?
::*Third, are OKBot and Acagastya still updating these pages, and would it make sense to remove those names from the comments?
::My fourth/fifth questions were going to be what you were going to do about duplicate names, but it looks like you noticed that and are taking care of it, along with a lack of leading zeros issue with the 2019-'''0'''1-27. 
::Also, as a minor point, even if you've only done 44 edits with the bot, please make sure when you finish with a trial that you link to the specific edits, since while "Contribs" might only show those 44 edits now, after you've made thousands they won't be the first thing to look at. 
::Actually, I ''do'' have another thought - for brevity, it might be best to have a wikilink in the edit summary instead of a full URL. [[User:Primefac|Primefac]] ([[User talk:Primefac|talk]]) 20:12, 28 January 2019 (UTC)

:::I have answers.
:::* There's no particular reason it uses http over https for the alexa.com link, I hadn't given it a second thought. I can change it to https.
:::* The variations in date formatting are an attempt to stick with the articles predominant style. The default style being ISO format, and if there's a use dmy or use mdy tag it uses the respective format.
:::* OKBot appears defuct, I wasn't aware of Acagastya, though from their [[User:Acagastya|user page]] it looks like they've left English Wikipedia at least. It does make sense to remove the (now duplicate) comments, that was ultimately the goal but it didn't work as planned.
:::* Good point on the making a list of the trial edits, conveniently it looks like I can search the contribs to make a view of [https://en.wikipedia.org/w/index.php?limit=50&title=Special%3AContributions&contribs=user&target=LkolblyBot&namespace=&tagfilter=&start=2019-01-27&end=2019-01-27 just the trial edits].
:::* Yeah, the wikilink idea occurred to me a few minutes too late, it looks terrible in the commit message :/ [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 23:32, 28 January 2019 (UTC)
::::With the constant modification that Alexa goes through, it is not a good idea to put manual labour for updating the ranks.<br>[[user talk:acagastya|<span style="color:#b00">acagastya</span>]] 08:53, 29 January 2019 (UTC)

*Regarding the 'Updated monthly by ...' lines - as is being demonstrated here there are stale entries - and it can be expected as no bot should ever be expected to operate in the future.  To that end I don't think this should be added, and would support having the next update remove any existing such comment codes. — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 15:21, 7 February 2019 (UTC)
*:{{BotExtendedTrial|edits=50}} Please implement the above changes in this run. [[User:Primefac|Primefac]] ([[User talk:Primefac|talk]]) 21:30, 14 February 2019 (UTC)
: was the trial completed? What are the results (please link to diffs as well). — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 18:51, 12 March 2019 (UTC)

::Sorry I've been dead in the water this last month, time hasn't been on my side (I figured I'd re-architect my server before I ran the trial, and have everything nice and containerized, but that didn't work out and then one thing led to another). I haven't done the trial yet, I plan to run it this coming weekend though. [[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 19:10, 12 March 2019 (UTC)

:{{BotTrialComplete}} Okay, ran the bot on [https://en.wikipedia.org/w/index.php?limit=50&title=Special%3AContributions&contribs=user&target=LkolblyBot&namespace=&tagfilter=&start=2019-03-18&end=2019-03-20 these] 50 pages. Some notes:
:*r.e. the "Updated by" comments: So it turns out the framework I'm using (pywikibot) strips out the comments, which is why they were being duplicated. This run did not add "updated by" comments. Removing existing comments could be done but would have to be a separate script.
:*I think I'll change the change comment to "Bot: Update Alexa ranking (link to a list of sites that the bot maintains)"
:*Some sites (e.g. [[Gothamist]]) list a URL in the infobox that is not ostensibly the site's actual (or main) URL, which gives an inaccurate alexa ranking. I think this is beyond my control though.
:*The original formatting of the infobox is unfortunately lost in pywikibot. The spacing varies - some ([[Adventure Gamers]]) use no spaces after the vertical bar, most one space, some align the equals signs, some don't (or do so inconsistently). Regardless, the information is gone at rewrite time.
:*A large number of sites had an "April 2014" style alt text specified for the "as of" tag. This script eliminates those.
:*One page ([[Shutterfly]]) had the "alexa" ref specified in a separate infobox references section at the bottom of the infobox, which led to a duplicate reference name error.
:Otherwise, everything seemed to run fairly smoothly. The last point I might be able to handle by searching for <code><nowiki>name="alexa"</nowiki></code> or something in the page text. I think it's a fairly rare occurrence though.
:[[User:Lkolbly|Lkolbly]] ([[User talk:Lkolbly|talk]]) 02:13, 20 March 2019 (UTC)
:{{ping|Lkolbly}} regarding the increase vs increasenegative difference, my reading is that this is a Numerical/Desirable field (see [[Template:Infobox website]] and related talk pages).  Moving a Ranking from 5th to 1st is a "decrease" in value, but an increase in desirability.  Arguably "1st place" is an increase from "2nd place" though so normal increase could be fine here - this should be sorted out at [[Template talk:Infobox website]], and that template documentation should be updated to match before this begins.  You don't want your bot to be warring with human editors over the direction of a triangle. — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 13:59, 28 March 2019 (UTC)
:{{OperatorAssistanceNeeded|D}}  please let us know any results from: [[Template_talk:Infobox_website#Bot_Job_and_arrows]]. — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 14:05, 2 April 2019 (UTC)
*See also [[Wikipedia:Village_pump_(policy)#Alexa_rank_question]] asking about the utility of this data at all. — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 14:59, 2 April 2019 (UTC)
:{{BotDenied}} task denied.  The discussion at VPP in [[Special:PermaLink/894265275#Alexa_rank_question]] is indicating that these edits are not widely supported.  Should this change through future discussions feel free to open a new BRFA referencing this one, and linking to the supportive discussion. Best regards, — [[User:Xaosflux|<span style="color:#FF9933; font-weight:bold; font-family:monotype;">xaosflux</span>]] <sup>[[User talk:Xaosflux|<span style="color:#009933;">Talk</span>]]</sup> 23:58, 26 April 2019 (UTC)
:''The above discussion is preserved as an archive of the debate.  <span style="color:red">'''Please do not modify it.'''</span> To request review of this BRFA, please start a new section at [[WT:BRFA]].''<!-- from Template:Bot Bottom --></div>