Benutzer:DrTrigonBot/ToDo-Liste

Related to DrTrigonBot-"framework":

runbotrun.py
- sum_disc.py
- clean_sandbox2.py $\rightarrow$ clean_user_sandbox.py
- wikipediaAPI.py (und wikipedia.py?) $\rightarrow$ dtbext/wikipedia.py, dtbext/pagegenerators.py und dtbext/query.py
new_user.py
replace_tmpl.py
irc_client.py

Für Konfiguration siehe

config.py
sum_disc_conf.py (ACHTUNG: gibt 2 Versionen; die für den toolserver und eine lokale zum Testen)

BUGS

Hinweis: Du darfst diesen Abschnitt editieren!

Ja, wirklich. Es ist schön, wenn jemand vorbeikommt und Fehler oder Links korrigiert und diesen Abschnitt verbessert. Sollten deine Änderungen aber der innehabenden Person dieser Benutzerseite nicht gefallen, sei bitte nicht traurig oder verärgert, wenn sie rückgängig gemacht werden.
Wikipedia ist ein Wiki, sei mutig!

Hier bitte Fehler im Bot melden.

Id

Was

Gemeldet von

B44

Für mehrere Seiten können von der API können keine section (heading) Informationen erhalten werden. Ist kein </references> Problem ich befürchte es sind API bugs, die gemeldet werden müssen?! Noch prüfen und im chat besprechen...!

-- Dr Trigon 23:19, 9. Jul. 2009 (CEST)

B49

Vorlage:Permalink

Ist das mit neuer Bot Version immer noch ein Problem?

--Fake4d 19:20, 2. Aug. 2009 (CEST)

B50

...

FEATURE REQUESTS

Hinweis: Du darfst diesen Abschnitt editieren!

Ja, wirklich. Es ist schön, wenn jemand vorbeikommt und Fehler oder Links korrigiert und diesen Abschnitt verbessert. Sollten deine Änderungen aber der innehabenden Person dieser Benutzerseite nicht gefallen, sei bitte nicht traurig oder verärgert, wenn sie rückgängig gemacht werden.
Wikipedia ist ein Wiki, sei mutig!

Hier bitte Anfragen nach neuen Funktionen stellen.

Id Was Gemeldet von

F39

Automatische archivierung alter Bot-Nachrichten, so dass immer nur die neuesten x Nachrichten in der Bot-Vorlage angezeigt werden (siehe Benutzer Diskussion:Benzen#Fragen wegen Bot-Modifikation). Möglichkeiten:

Überschrift und ein {{Erledigt|1=~~~~}} dazuschreiben und von ArchivBot/SpBot machen lassen
<noinclude> so dazuschrieben, dass nur die gewünschten Nachrichten in Vorlage sichtbar
herkömmliches archivieren, der Einträge auf wählbare Archiv-Seite

Auf Vorschlag von Benzen ^C₆H₆ auf seiner Diskussionsseite.

Für nächste Bot Version, derzeit läuft nur eine Bereinigung (feature close)

-- Dr Trigon 15:41, 22. Jun. 2009 (CEST)

F41

Bugzilla:19417

~~Wenn/falls gelösst kann der code schön vereinfacht werden.~~ Ist anscheinend schon etwas in der Art in Arbeit, siehe Vorlage:Permalink und action=parse&prop=sections (translatewiki.net) (ist schon ver. 52241 anstelle von ver. 48544).

-- Dr Trigon 14:20, 27. Jun. 2009 (CEST)

F45

Einfaches Hinzufügen weiterer überwachter Seiten durch Benutzer (z.B. wie bei Benutzer:CopperBot/Überwachte Seiten)

Ist in neuer Bot Version integriert, jeder Benutzer kann für sich die Liste modifizieren

Merlissimo

F47

Vorlage:Permalink (oder auch FEATURE REQUEST), damit auch Unterseiten von "Benutzer Diskussion:...", also "Benutzer Diskussion:.../Unterseite" als Ausgabeseiten verwendet werden können.

Gemeldet von Benutzer:AwOc.

-- Dr Trigon 12:18, 14. Jul. 2009 (CEST)

F48 ... ...

TODO

Benutzer:DrTrigon/Entwurf/Vorlage:Du darfst nicht

Id	Priorität	Was
37	1	bug fixes für Version 0.1.0013 und testen von bzw. vorbereiten von update auf Version 0.1.0014, durch ausführliches testen aller Funktionen und lange Vergleiche mit aktueller Version. Ev. ändern zu Version 0.2.0001...?! (_localizeDateTime, history)
20		Erstellen eines Zeitplans.
36		CPU last auf toolserver kontrollieren mit /usr/bin/time -v python ... wenn über lange Zeit sehr hoch, dann versuchen die Last zu reduzieren oder die Admins vom toolserver anfragen, was zu tun ist? (nice wird schon verwendet)

MID-/LONG-TERM FEAT.

Benutzer:DrTrigon/Entwurf/Vorlage:Du darfst nicht

Id	Was
17	history selbst verwaltend, d.h. history ist script das mit 'eval(buf)' ausgeführt werden kann und automatisch alles so liefert (in variable verpackt) wie gewollt. dann könnte konvertierungcode für andere formate einfach einmal dazugeschrieben werden und problem gelösst! / Ev. pickle verwenden speed-up durch multi threading für abrufe (siehe dazu auch 'threadpool.py') oder auch 'ThreadedGenerator' (in 'pagegenerators.py') für asynchrones vorauslesen der seiten (read/get) verwenden (output der alten und neuen read methoden vergleichen, sollen das selbe leifern!) page write vorgänge auch über API abwickeln: action=edit&text= für write und action=edit&appendtext= für append ~~(könnten gebündelt alle aufs mal ganz am ende des bot-laufs gemacht werden, etwas doof bei fehlern)~~. Bis rev. 6977 ist nichts wie 'append' (für page) integriert ev. können API Aufrufe mit 'generatexml' verbessert/vereinfacht werden (bzw. sauberer gemacht werden?!)...
18	paramter/optionen erweitern, mögliche neue optionen: history; umleiten auf wählb. wikiseite (z.B. Benutzer Diskussion:DrTrigon/History). Siehe dazu auch Vorlage:Permalink, Benutzer:DrTrigon/Entwurf/Vorlage:SumDiscEntry und http://toolserver.org/~drtrigon/cgi-bin/chksum_check.py, durch die Verwendung dieses Konstruktes (v.a. der Vorlage) können zusätzliche Daten (und Funktionen) in die History integriert werden. report; blockweise oder einzeln (ob jeder eintrag einz. sign. wird oder alles als block) tolerance; nicht hinzufügen wenn nach kurzer zeit (hohe aktivität) neuer eintrag kommt ignoreuser; ausschluss von disk-seiten die von bots geändert wurden (z.B. User:ArchivBot)
30 $\uparrow$	autoexclude; intelligente exclude-liste; für mich (ev. für alle) z.B. alle Hinweise auf Bot-Ausgabelisten von anderen Benutzern unterdrücken, weil er so was meldet, da ich vor langer Zeit mal die Seite bearbeitet habe und der letzte Bearbeiter meiner Diskussion war. (einfach Benutzer:DrTrigonBot/Diene Mir! als exclude-liste verwenden? oder die signaturen in bot-ausgaben verändern, dass der bot sie wiedererkennt?)
19	mehrsprachig/multi-language-support damit allg. funktionsfähig und auch für andere wikis brauchbar, ev. "contributen" an pywikipedia-bot-framework zuerst aber mit neuestem bot frameworkversion mergen/vergl. $\rightarrow$ look also at id B45
25	wenn 'sum_disc.py' ohne grössere/akute BUGS läuft und alle wichtigen und sinnvollen FEATURES integriert sind, weiter zu anderer bot funkt.: 'replace_tmpl.py' (für Individuelle Vorlagenersetz- und Bearbeitungsaufträge) weiterentwickeln mit Funktionen aus 'template.py' erweitern (kombinieren mit den unveränderten scripts 'cosmetic_changes.py'/config.cosmetic_changes, 'spellcheck.py', u.a. zudem: fixes.py, weblink.py siehe Benutzer Diskussion:Benzen#Fragen wegen Bot-Modifikation, die wiki kann auch seiten vergleichen, siehe z.B. revid und diff zw revids) mal kurz analysieren, was für ein Bot überhaupt sinnvoll ist, bzw. gebraucht wird?! (mal rumfragen) andere idee wäre status/dhtml-bot der links oder inhalte ausserhalb der wiki überwacht und bei änderungen diese in der wiki nachträgt könnte auch sum_disc.py dazu bringen, dass er noch unsigniertes nachsigniert (wobei sign-bot scheint schwierig ohne live irc und dann aufwändig, bzw. mach viele edits und ist gefährlich falls er amok läuft) Wikipedia:Verbesserungsvorschläge#Bot-Verständigung bei Löschanträgen (je nach Bot Typ unbedingt `{{`Vorlage:Bots`}}` berücksichtigen/integrieren)
22 $\uparrow$	new_user.py: nur user mit einer minestzahl an beiträgen (z.B. 1) begrüssen, bzw. auflisten
23 $\uparrow$	new_user.py: ev statt auf feste seite schreiben über template integrieren Vorlage:NewUserList z.B. ...
F44	Was hat https://fisheye.toolserver.org/ alles zu bieten? Doku siehe http://confluence.atlassian.com/display/FISHEYE016/FishEye+Documentation+Home.

DONE (engl. only!)

Benutzer:DrTrigon/Entwurf/Vorlage:Du darfst nicht Included from: Benutzer:DrTrigonBot/ToDo-Liste/DONE

Files related to DrTrigonBot-"framework" (look also at Botwiki:Python:DrTrigonBot scripts):

Configuration: config.py, user-config.py, (others?)

Data: (BoW training, ...)

Look also at Benutzer:DrTrigonBot#Source.

Aktuelle Version

Aktuelle Revision (FishEye): Activity • Files • Users • RSS feed

Id	What	Script
	Bugfix for: Sicherheitslücke im Anmeldeprozess wurde geschlossen - das Login via API wurde ebenfalls angepasst so the bot (with the old framework) was not able to write to the wiki anymore, so changed to revision 8072 but the sulution there is not that nice, should be improved as I understand this, but I hope I is ok for a while... ;)	framework, dtbext_query.py, runbotrun.py
25	Spezial:Permanentlink/64368286 sandbox for new created status/subster bot. This bot watches text on external (or internal) pages, the text is specified by regex. If this text changes, the bot copies the new/actual text into the wiki. Tag model to mark places of text drop is borrowed from MerlBot (and this sub-bot could also be used on my local wiki with an anacron-job ;).	subster.py

Version 0.2.0000

Revision 1 (WebSVN): Log

Id What Script

Feature/Bugfix/... for:

Experimentally introduced a new bot part 'page_disc.py' for Spezial:Permanentlink/77229254#Disk-Zusammenfassung_auf_WD:ADT, test is running on Benutzer:DrTrigon/Spielwiese#Test 3. For this several small changes had to be made on 'dtbext_query.py' and 'dtbext_wikipedia.py' which should also return some benefit to other bot parts, e.g. 'sum_disc.py' (parser and others - look at the comments in the mentioned files).
MailerBot; the bugfixes in 'dtbext_query.py' enabled to send longer mails and thus it looks like the earlier introduced long mail support becomes obsolete. May be for very special needs it can still be helpful, but at the moment it is disabled.

page_disc.py, dtbext_query.py, dtbext_wikipedia.py, mailer.py

Feature for:

MailerBot; enabled the bot to create and sent long mail through the toolserver for users, that have registered themselves.

mailer.py, mailerreg.py

Feature for:

SubsterBot; enabled the bot to retrieve wiki text content, not only html page content. Used for wiki to wiki text substitutions.

subster_beta.py, substersim.py

Bugfix for:

Small bug in MailerBot; the 'content_maxlen' has to include the subject size and thus must be calculated for every page (else some mails for certain pages cannot be sent).

mailer.py

Bugfix for:

Small bug in 'Pages.getVersionHistory(...)'; if the API omitts some entries (and sometimes it does) and does not return at least a 'missing' for them, the class does recognize this now and generate and add dummy results for the missing pages.

dtbext_wikipedia.py

Bugfix for:

Sicherheitslücke im Anmeldeprozess wurde geschlossen - das Login via API wurde ebenfalls angepasst so the bot (with the old framework) was not able to write to the wiki anymore, so changed to revision 8072 but the sulution there is not that nice, should be improved as I understand this, but I hope I is ok for a while... ;)

framework, dtbext_query.py, runbotrun.py

Bugfix for:

Spezial:Permanentlink/72481024#Dein Bot: komische Einträge Notification links (section matching); strange behaviour. When using wiki API to check sections, both methods are now used always, instead of only when the first returned nothing and the longer output is used (since the other might not be complete). This should prevent problems with pages like Spezial:Permanentlink/72550929 with any kind of include-tags. This method now is slower since always 2 calls are made, this should be improved. (bot run now takes about 1.5h instead of about 1h before; 50% slower...)

dtbext_wikipedia.py

B50 Spezial:Permanentlink/64115999 Notification links; don't work. Used target link from http://toolserver.org/~merl/UserPages/query.php?user=DrTrigon&format=xml (parameter/value 'url'). But the handling of foreign-wiki-links in framework could still be improved (interwikimap). dtbext_pagegenerators.py, sum_disc.py

37

Creation of a stable release v0.2.0000 based on bug fixes, new feature and improvements in current semi-stable release v0.1.0013. Testing of this version and update of bot software on toolserver to this new version. Extensive testing of all new functions and whole code like output of times (correct timezone setting) and others.

Setup of new CRON-job for compression of history once all 14 days.

(toolserver)

MailerBot created. This sub-bot is very experimental at the moment; it does it's job for me, probably/hopefully for you to, but do not expect miracles at this current state (but ideas, hints and bug reports are still very welcome). mailer.py

36

Check CPU load of whole bot script with:

/usr/bin/time -v python /home/drtrigon/pywikipedia/runbotrun.py -cron

because the bot runs every day at least for 1 hour, so the load must not be too high! If it is, try to delay the expensive calls and/or contact the toolserver admins what to do, because 'nice' is already in use. The load of history compression has not to be checked, since the script runs not that often.

Result:

drtrigon@nightshade:~$ /usr/bin/time -v python /home/drtrigon/pywikipedia/runbotrun.py -cron
	Command being timed: "python /home/drtrigon/pywikipedia/runbotrun.py -cron"
	User time (seconds): 32.23
	System time (seconds): 2.86
	Percent of CPU this job got: 1%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 46:18.79
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 0
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 42316
	Voluntary context switches: 204822
	Involuntary context switches: 3707
	Swaps: 0
	File system inputs: 0
	File system outputs: 1816
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

should be ok with 1% CPU load.

(toolserver)

B44

For several pages related with MerlBot the bot was not able to retrieve section (heading) info. Those issues appeared on following pages:

and were related to <noinclude> and <onlyinclude> tags, and have been solved. But the pages are handeled like templates because of 'action=parse&text={{%s}}__TOC__&prop=sections' is used for recognition. Therefore it could become necessary in future to use a more sophisticated way; get page content, preprocess (handle <noinclude>, <onlyinclude>) it, pass it to 'action=parse&text=%s__TOC__&prop=sections' and then postprocess (handle the rest like <pre> and others) the result.

dtbext_wikipedia.p

F45

List of parameters/options enhanced:

checkedit_list: each User has the ability to add additional patterns for pages that should be monitored by the bot.

Mentioned by Benutzer:Merlissimo (was already implemented in bot code).

(publishing)

B49

Bugfix for:

Spezial:Permanentlink/63464336#Problem_mit_DrTrigon_Bot has solved itself, according to Spezial:Permanentlink/63713171.

34 Poll/survey under the bot users about happiness and feature requests. (my talk page)

F38

New feature integrated, by using data from http://toolserver.org/~merl/UserPages/query.php?user=DrTrigon&format=xml the bot is now able to give users hints if any of their other wiki talk pages was changed, just for the case of getting messages there.

Mentioned by Benutzer:Merlissimo in chat.

dtbext_wikipedia.py, dtbext_pagegenerators.py, sum_disc.py, sum_disc_conf.py

B47

Bugfix for:

Spezial:Permanentlink/62327426 look at Benutzer Diskussion:Holger.Ellgaard; issue with correct signature recogition, it's necessary to find the most recent out of a set of signatures.

Reported by [[ Forrester ]].

sum_disc.py

F46

As long as F45 is not implemented yet, please add

to the surveillance list.

Mentioned by Benutzer:Merlissimo.

sum_disc_conf.py

B48

Bugfix for:

Spezial:Permanentlink/63048704#Es_gibt_leider_noch_Probleme_mit_deiner_Datei: Internationalization issue.

$\rightarrow$ look at id B45

Reported by [[ Forrester ]].

(look there)

B46

Bugfix for:

Spezial:Permanentlink/62158091 / 25. entry; the brackets '[' and ']' have to be replaced in some parts of the output (namely the url part of the links).

sum_disc.py

B45

Bugfix for:

Spezial:Permanentlink/62087384 / 2nd last entry; is reported despite the fact it is signed/not necessary to report. The problem was that a user used english page links e.g. 'User:...' in its signature, those describers were added now.

sum_disc_conf.py

From id 44 splitted into B?? and F?? for simpler counting.

33

Bot runs all 24 hours (instead of all 48 hours).

Proposed by Benutzer:Forrester (for BLUbot).

toolserver

List of paramters/options enhanced:

altsign_list: especially for BLUbot introducted parameter to give alternative user names to check for signature, e.g. you have a representative or something else.

sum_disc.py, sum_disc_conf.py

40.2

Bugfix for:

Spezial:Permanentlink/61457254 / last block, 1st entry; the last part of '_checkThData()', check for text after signature recognizes the '|}' as additional text, code to prevent this was tested but not activated beacuse the better way would be to change the welcome template and suppress additional newlines at the end.

sum_disc.py

42, 43

Bugfix for:

Spezial:Permanentlink/61674143 / 2nd entry
Spezial:Permanentlink/61871974 / last entry

The bot reports redirects (created by users) as new discussions (which of course are non-existent). Maybe redirects should be resolved and the resulting pages checked against the ignore-list? This change was mentioned by Benzen ^C₆H₆ 09:20, 5. Jul. 2009 (CEST) (probably a hamster?).

dtbext_wikipedia.py, dtbext_query.py, sum_disc.py

17.2

Wanted to change the history format such that each time, at each bot run the last history file entry is read but contains the full actual history. This data is updated and checked for changings since last run, the actual version history for each page is retrieved and saved - this would serve a simple programming interface, however it uses way to lot of hard drive space (and memory). Now using the old incremental format, but with the data updated and checked for changings since last run and 'compressHistory' does no additional relevancy check, just rewrites same history content (bot behaviour keeps the same after compression). So fast compression was removed completely.

Dogma: 'compressHistory()' must not do any kind of relevancy/signature check, and not alter the bots behaviour.

At bot run (not compression) all discussions are checked for signature and if it is completely missing the discussion for this page is closed and it is removed from history.
Additionally 'rollbackHistory()' was implmented to make the bot to use older history entries only.
Different pagestates introduced to make the new history handling practible:
- _PS_warning: serious or no classified warnings/errors that should be reported
- _PS_changed: changed page (if closed, will be removed)
- _PS_unchanged: unchanged page
- _PS_new: new page
- _PS_closed: closed page (remove it from history)
- _PS_maintmsg: maintenance message
Split of 'self._news' into 'self._news' (_PS_changed, _PS_new, _PS_maintmsg) and 'self._others' (_PS_unchanged, _PS_warning) and furthermore parallel (to 'self._news') updating of 'self._history' (_PS_changed, _PS_new, _PS_maintmsg, _PS_unchanged, OHNE _PS_closed) which is saved in the end to the history file.
History should be dropped at least all two week, because it is bigger now (could be zip-ed)!

sum_disc.py

35.2

Method of choice for wiki content access is to use the API and retrieve pages in bunches (of e.g 10 pages). Dumps are very fast too but not available on a daily base (all 2 days only), look at http://download.wikimedia.org/dewiki/ also (Benutzer:MerlBot claims to use dumps). So switched to handling of page read access with API (it looked to me that the recent rev. of bot-framework have switched partly to access over API, but that is wrong; it are a few commands only, mostly user or config related), this was already implemented with different methods in dtbext.wikipedia.Page().get(). Now it was implemented for simultaneous read of a bunch of pages with prop=revisions (rv) in dtbext.wikipedia.Pages().get(). But this has a small drawback, the parameter/option 'directgetfull_switch' becomes a kind of useless or obsolete (because it is faster to re-read single pages per block, then splitting the block and all handling related with this), therefore removed parameter/option:

~~directgetfull_switch~~

It is important that the original and the new page read methods deliver the same page content!

$\rightarrow$ look also at id 17

dtbext_wikipedia.py, dtbext_query.py, sum_disc.py

21

Merged with most recent pywikipedia-framework 6977 and testing of new functions, as well as update of wikipedia.py (because of lines 725-728). The bot reports now on every run the most recent framework rev. because of verbosity and information (download via svn could be automated, look also at Benutzer Diskussion:Benzen#Fragen wegen Bot-Modifikation). The extensions needed for the bot are now techn. adapted to the framework and packed into one single package with same structure like the framework:

wikipediaAPI $\rightarrow$ dtbext.wikipedia and dtbext.pagegenerators (analogue to the framework)
wikipediaAPI.ContributionsGen $\rightarrow$ dtbext.pagegenerators.UserContributionsGenerator (is now a generator and returns a list of pages instead of a list of tupels)
wikipediaAPI.BacklinksGen $\rightarrow$ dtbext.pagegenerators.ReferringPageGenerator (as above), furthermore dtbext.wikipedia.Page().getReferences() with API implemented. If the API is used for call is not related to the generator but to the given page-typ. The new generator supports setting of limit/number additionally.
wikipediaAPI.GlobalWikiNotificationsGen $\rightarrow$ dtbext.pagegenerators.GlobalWikiNotificationsGenerator (experimental only)
wikipediaAPI.Page()._getParsedString() $\rightarrow$ dtbext.wikipedia.getParsedString()
wikipediaAPI.Page().getFull(), wikipediaAPI.Page().getParsedContent() and wikipediaAPI.Page().get() (the same as wikipedia.Page().get()) $\rightarrow$ dtbext.wikipedia.Page().get() with mode='full' , mode='parse' or without any setting (default)
wikipedia.sectionencode for bot messages could be possible to solve the old special char issue
dtbext.wikipedia.html2notagunicode introduced (as addition to 'Unicode library functions')
clean_sandbox2.py $\rightarrow$ clean_user_sandbox.py
dtbext.query introduced

(various)

35.1

Introduced bundeling of API calls. A new class 'Pages' performs all actions on a set of pages simultaneously. Currently page version history information can be retrieved this way, this can be enhanced on other functions too.

$\rightarrow$ look also at id 17, 35.2

wikipediaAPI.py, sum_disc.py

Full compression (not fast) improved by adding a part that removes all old/removed headings from history page entries. wikipediaAPI.py, sum_disc.py

New history format (v5) introduced; use dict (again) is the one and only intelligent/correct way to hold unique data
Conversion of old history formats in '_getHistoryPYF' during read-in, instead of different places spread all over the code. This causes all history data to be keept in up-to-date format, now.

Dogma: history format for headings (with checksums) has to be a dict, with unique keys. (if not possible to make unique, then add a timestamp or something like this)

sum_disc.py

Heavily improved parameter/option handling, all internal used user specific bot parameters/options can be modified by the users. The behaviour either to overwrite or preserve the defaults can easily be configured internally. sum_disc.py, sum_disc_conf.py

18

List of paramters/options enhanced:

ignorepage_list: here you can add pages to ignore, to this list added are your own discussion pages. if you do not define this parameter your archiv pages are used by default.

sum_disc.py, sum_disc_conf.py

17.1

Speed-up by:

pre-compile of all often used and static regex (probably do some more later, if it makes sense)

Dogma: frequently used regex must be pre-compiled.

wikipediaAPI.py, sum_disc.py

28

Clean the code by removing all unnecessary 'try' constructs, replace '(... == None)' contructs by the shorter (and cleaner) version and add documentation.

$\rightarrow$ look also at id 26, 27

sum_disc.py

32 Compression could be done by comparing against the backlinks. This was implemented and tested, here the observations: the original method depends on '_SearchForSignature(...)', the alternative method on a properly configured 'backlinks_list'. if properly configured the alternative mode is a lot faster, but it depends on this config, whereas the original method works always! the statement all pages listed on backlinks have currently a signature is correct if 'backlinks_list' is properly configured (how to check this?) sum_disc.py

27 The checksum checker in '_checkThData(...)' improved (no need for 'try' structure anymore) and the different modes documented. Mode v1 was removed because it is very old, if there is any history entry with v1, then it is ignored, the page is reported and logged again. This should cause not that much problems (since there are very few or no such entries existent anymore). sum_disc.py

31

Checkout of API functions/calls, here some examples:

meta=userinfo (ui) with 'hasmsg': gives only a hint if there is a new message or not (already implemented in bot framework) but how to set it read after loading the page to make the message disappear is still left unclear / function not needed at the moment
meta=allmessages (am): can be used to translate some describtions and other things from English to this wiki / function useful
meta=siteinfo (si): a lot of site, wiki and server info / function useful
action=paraminfo: obtain information about certain API parameters / function useful

$\rightarrow$ look also at id 13, 24

24

The page Spezial:Änderungen an verlinkten Seiten (Special:RecentChangesLinked) could be implemented as an addition to Spezial:Beiträge (Special:Contributions) here is an example for DrTrigon. Unfortunately I was not able to find an API for this, but list=backlinks (bl) which is the same as Spezial:Linkliste (Special:WhatLinksHere) and this was implemented as an addition. The use of this is configurable by a new parameter/option:

getbacklinks_switch: read the backlinks to user discussion page and check them additionally

wikipediaAPI.py, sum_disc.py

29

All pages 'Wikipedia:Löschkandidaten/<DATE>' are read twice (second time with 'getFull()'), therefore a parameter/option was introduced:

directgetfull_switch: read all pages with 'Wikipedia:Löschkandidaten/\d' at first attempt with 'getFull()'

$\rightarrow$ look also at id 17

(This option may later become obsolete, e.g. when updating to new bot framework; if the default read method 'get()' is changed)

sum_disc.py

26 Bot should run without any use of 'fall back' mode in '_getThContent(...)' therefore this obsolete, old and unreliable mode was removed but replaced with a warning message (just in case). sum_disc.py

Version 0.1.0013

Id

What

Script

List of paramters/options enhanced:

altsign_list: here you can add other signatures to check/handle equivalent like the users signature. Needed for Benutzer:BLUbot and WP:DÜP service.

sum_disc.py

40.1

Bugfix for:

Spezial:Permanentlink/61457262 / last entry; 1st link should not be reported because the 2nd is the actual discussion.
Spezial:Permanentlink/61457270 / last block
- 1st entry; is already covered by problems above.
- last entry; is an error, should not be reported don't know why it is reported.
Spezial:Permanentlink/61457254 / last block
- 3rd entry; is an error, should not be reported don't know why it is reported.
- and last entry; same problem, same origin?

All those are related with the different history formats and the way they were read in and updated. All older formats are converted now during read and in to v5. Took from Version 0.1.0014.

sum_disc.py

Bugfix for: Spezial:Permanentlink/61457165 / 1st entry; head recognition out of sync, added another check for validity (to decide to use full read mode). Took from Version 0.1.0014.

wikipediaAPI.py

Bugfix for: html tag heading recognition; tags with additional values are now recognized too.

wikipediaAPI.py

Bugfix for: heading recognition; e.g. Spezial:Permanentlink/61381636 / last block, 1st entry; because of some <nowiki>-tags the function had some problems. All '=' within those tags are substituted by '=' as work-a-round.

wikipediaAPI.py

Bugfix for: re-reported discussions; e.g. Spezial:Permanentlink/61311960 / last entry or Spezial:Permanentlink/61309424 / 1st entry of last block; this is related with the different history formats and the way they were read in and updated. Some older formats are converted now during read and in that way the data can be updated properly.

sum_disc.py

Bugfix for: twice appearing headings; the API returns wrong links they are not complete, the numbering (from 2 upwards) is missing. So a fix for this was implemented now the bot should really be able to get all head links correct.

wikipediaAPI.py

The DrTrigonBot status panel was added to the Toolserver Wiki as a tool (with description page, a.o.) and a bit modified (added a link to User:DrTrigon).

panel.py

35.3

Bugfix for: Spezial:Permanentlink/61105710 / 1st entry of last block; the page was processed without any heading info, in fact it could not retrieve the headings (sections). Added an additional/alternative method if the first fails.

$\rightarrow$ look also at id 28

wikipediaAPI.py

35.4

Bugfix for: Spezial:Permanentlink/61106764 / 2nd entry; the headings got out of sync, in the 2nd half it was shifted about 1. The problem was that headings recognized by wiki parser has to start at the first char of a line!

$\rightarrow$ look also at id 28

wikipediaAPI.py

5

testing of new 0.1.0013 release. upload of it and rearrange data on toolserver.

copy 'local:pywikipedia-devel_0.1.0013/' to 'toolserver:./pywikipedia/'
copy 'toolserver:./pywikipedia/' to 'toolserver:./public_html/source/pywikipedia/'
copy 'toolserver:./pywikipedia/' to 'local:toolserver/pywikipedia/' and 'local:toolserver/public_html/source/pywikipedia/'

local, toolserver

15

it would be a good idea to use https://jira.toolserver.org/secure/BrowseProjects.jspa the Toolserver issue tracker if the amount of bugs reported and to handle, exceed a certain number, but for the moment it is not needed. the problem is you cannot add a new project as normal user, so the system here has to deliver this service.

BUGS

16

had issues with creation of BOT-MESSAGE containing some special chars (e.g. ':'). the algorithm should now be more stable, but it estimates the correct heading only, so could still make some problems.

Dogma: use as simple bot maintenance message texts as possible.

sum_disc.py

13

sample bot output:

* Processing User List (wishes): Benutzer:DrTrigonBot/Diene Mir!
NOTE: You have new messages on wikipedia:de
...

if such a message apears again, the bot now read its discussion page Benutzer Diskussion:DrTrigonBot and prints/show the content to output (log file). that informs you about the message and should lead to supress further display (of the same old message).

runbotrun.py

14

a very intressting fact is the page Spezial:Änderungen an verlinkten Seiten (Special:RecentChangesLinked) could be able to replace or be an alternative to Spezial:Beiträge (Special:Contributions). here an example for DrTrigon (linked to Benutzer Diskussion:DrTrigon) to show the capabilities.

sum_disc.py

9

if the bot is executed by CRON job on toolserver the preprocessing/filtering of log file output does not work properly. have added another regex to the output handler, hope that helps.

runbotrun.py

10.3

purge page cache through wiki API according to example http://toolserver.org/~drtrigon/wiki-purge.html (has to be deleted from the toolserver now) implemented as wikipediaAPI.Page(...).purgePageCache(). the issues occured here earlier have disappeared or were due miss-programming/-understanding of the API by myself.

wikipediaAPI.py, sum_disc.py

10.2

the render given in http://svn.wikimedia.org/viewvc/pywikipedia/trunk/pywikiparser/ looks to be (very) old; the newest contributions are Modified Mon Oct 29 06:41:05 2007 UTC (19 months, 1 week ago) and the wiki API seems to do a quite good job. so the bot is using the API at the moment.

wikipediaAPI.py, sum_disc.py

11

because of improvements in 10.1 before the bot is now able to detect level 1 headings (through a work-a-round, but it is possible) with the old it was nearly impossible. so level 1 headings are taken into account except the bot has to fall-back into the old mode so the wrong display of discussions in front of level 1 headings (on problematic pages such as Löschdiskussionen bei Benuterseiten, WP:FzW, WP:Auskunft, Portale and others) should stop now.

wikipediaAPI.py, sum_disc.py

10.1

wikipediaAPI.Page(...).getSections() improved; now it is on a good level and able to find and synchronize the subsection headings in pages
wikipediaAPI.Page(...).getFull() introduced; to help ...getSections() it uses 'expandtemplates' API and therefore should be able to give a low level output of wikitext on pages, however it seems to hide HTML comments for example...
_checkRelevancyTh(...) improved with help of ...getSections() but it still has full fall-back capabilities.
wikipediaAPI.Page(...)._getParsedString(...) introduced and wikipediaAPI.Page(...).getParsedContent() improved; they should work now.
_parseNews improved because of the two mentioned before it was possible to replace _renderWiki(...) (and with it _renderWikiTempl(...) too) with _getParsedString(...) and they are useless now and can be deleted.

wikipediaAPI.py, sum_disc.py

7

'_setUser(...)' and '_getUsers()' used both two seperate parameter/option handling mechanisms. since the entry in Benutzer:DrTrigonBot/Diene Mir! without and parameters in brackets '{...}' offers already a config option; you can specify any target page within your userspace. this option and the "real" options in brackets where handeled in 'self.userListInfo' and 'self.param'. those two mechanisms were now merged and the target page in 'self.userListInfo' is added to 'self.param' as 'userResultPage'.

Dogma: 'self.param' is the only option/parameter handling mechanism (especaly for params changable by all users through wiki)

sum_disc.py

6

config management/system for running bot in seperate file; all options (global vars) moved there. the variables could be named nicer (more self-explainig) but the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!

Dogma: the external options names (parameters that can be set within Benutzer:DrTrigonBot/Diene Mir!) must not be changed! never!

sum_disc.py, sum_disc-conf.py

8

extended search/check range/context:

Wikipedia:Auskunft
Wikipedia:Café
Wikipedia:Verbesserungsvorschläge (could make problems cause of level 1 headings)
Wikipedia:Tellerrand
Wikipedia:Urheberrechtsfragen
Wikipedia:Vandalismusmeldung
Wikipedia:Administratoren/Anfragen
Wikipedia:Administratoren/Notizen
Wikipedia:Diskussionen über Bilder

added. some of them could be extended by Wikipedia:Administratoren/* (but this would trigger more traffic with few benefits only). this change was mentioned by Benutzer:Benzen in Benutzer Diskussion:DrTrigon/Archiv#Bot: Reklamation.

sum_disc.py

Version 0.1.0012

Id

What

Script

18

parameters/options introduced with configuration directly on Benutzer:DrTrigonBot/Diene_Mir!

currently possible/implemented options are:

checkedit_count: how many recent edits should be checked (mainly for use with BLUbot)
reportchanged_switch: should changed (and already as new reported) discussions on change be listed again (mainly for use with BLUbot)

sum_disc.py

12

format/system of saved checksums had to be changed once again (3rd or 4th time) because of the dict used with subheadings as key was not unique (all subheadings on a page may have the same name); now the bot is using a list again, but a list of tuples (there are still some issues open with subheadings, this may make another change necessary...)

sum_disc.py

4

add interface for history's compression; was done initially but does not make sense since the permissions of user 'apache' will not allow to to re-write the history. the good possibilities are to do it by hand (manual) with a toolserver login (from time to time) or setup a e.g. monthly or weekly CRON job

Dogma: history compression should be done by hand or CRON job (if not by hand, do auto backup)

panel.py

3

improvement of admin log interface: 'filelist' and 'checkfiles' are not necessary, the whole path can be generated only once just before the call of 'os.remove'!

panel.py

2

instead of using 'runbotrun.py -auto', now a CRON job was created on toolserver, this should in order take care on the bot timing and be able to deal with server reboots. with this action all issues related with this are solved. call is of bot is 'runbotrun.py -cron'

runbotrun.py

1

admin interface added for removal of old log files instead of using cyclic logs (5 files e.g.)

Dogma: admin interface is used for log file removal - it's link is not published

panel.py