Wikipedia:WikiProject edit counters/Flcelloguy's Tool
From Wikipedia, the free encyclopedia
This is an old revision of this page, as edited by AySz88 (talk | contribs) at 01:36, 16 April 2006 (lots of updates (using "capabilities" as a todo list for now, might be better moved somewhere else....)). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.Revision as of 01:36, 16 April 2006 by AySz88 (talk | contribs) (lots of updates (using "capabilities" as a todo list for now, might be better moved somewhere else....))
 | Current status of toolserver and other edit counting tools: Interiot's Tool: Currently not replicating from en.wiki. Contributions aren't updating; effectively down. Please use Interiot's Tool if it is working; this is only a backup. See also: Kate's Tool and Interiot's Javascript |
|
Attention users: We need your help to improve this tool. We're particularly looking for someone with Java knowledge or ideas for a good GUI. Feel free to make a feature request. |
This is an extension of Flcelloguy's Tool which processes the HTML from a contribs file and parses it into a file readable by the tool. It is not finished, but the core is done, so I'm publishing it as v1.00.
Titoxd(?!? - did you read this?) 04:21, 8 December 2005 (UTC)[reply]
Compiling and executing the tool requires the Java Development Kit (JDK) for Java 5.0 (aka 1.5.0).
Requirements
- Minimal requirements: Java Development Kit (JDK) from Sun Microsystems. This comes with a Java compiler (javac.exe) that can be executed from the command line, and the program can be executed using java.exe.
- With IDE: You must have a Java IDE and JDK (freeware available; see below). Thus, you must make two downloads - one from Sun Microsystems, which developed Java, and another from a Java run-time development program. We recommend JCreator, BlueJ, or Eclipse.
- Please see the help page for more information.
Quick-start directions
(Note: these directions may become outdated in the near-future.)
- Download and install Java version 5 (see links below)
- Download the JAR from here. Double-click it to run it.
- If this does not work:
- Check the java version (5).
- Download the batch file here to the same folder (you may need to right-click and click "Save As..."), then run the batch file.
Links
Capabilities
Current Capabilities
Count the number of edits- Break down count by project namespace
- Count percentage and number of "minor" flag usage
- Correct parsing of edit summaries (through PurgeContribs.java)
- Displaying of edit summary statistics (Titoxd)
- Adding support for direct analysis of Special:Contributions (beta version can process in 5,000-edit intervals) (done, will be available in 4.0)
- Add "kill bit" functionality for turning off tool remotely when server load requires it
- Make choosing files simpler by using JFileChooser() (done)
Under development
Critical: Clicking "Continue: No" does not stop the tool.- Add more detailed statistics with the parsing of timestamps, etc. available in Titoxd's code
- Add more buttons to UI, make more user friendly
- Better and more detailed instructions on how to download and use (perhaps a JAR?)
- JAR file done, will be distributed after 4.0, just needs hosting
- Output to
txtfile for debugging (already done) and also to make copying results to the project easier
Future developments (see also Feature requests)
- Merge the two pages (this page and User:Titoxd/Flcelloguy's Tool, perhaps under a WP:TOOLS subfile; rename to reflect that it's not "my" tool, but both my and Titoxd's tool (Flcelloguy)
- Add a progress bar (like this?)
- Add "First edit" date (easy)
- Saving contributions in StatBundle to a local file, perhaps a CSV or similar file?
- Processing of user logs (Titoxd)
- Output user access level (i.e. sysop, bureaucrat, CheckUser, etc.) like how Interiot's tool did - parse
http://en.wikipedia.org/w/index.php?title=Special%3AListusers&group=&username=USERNAMEfrom Special:Listusers and parse what's between parenthesis, if any, and then display that.
Known bugs and limitations
- User UI is not completely user-friendly, needs improvements and suggestions
- Need to create/update documentation to reflect move to WikiProject (Flcelloguy)
- Exceptions not handled sometimes, need to process correctly
- The application appears to hang while it is downloading contribs, but it is not - add an indeterminate progress bar to tell the user to keep waiting
- Attempt to make counting less crude by using a query (Outside scope - use Interiot's Tool)
Revisions
- v1.00: Original version, parses contribs to HTML file
- v1.01: Revision, split into a separate class, removed print command to system buffer (slowing down tool, only used for debugging)
- v2.00: begun processing the raw HTML file, parsed date/time stamp and page name into a special "Contrib" class. Minor edits, edit summaries and most recent edits still to be implemented.
- v2.10: Minor edits implemented, some code for edit summaries created (not operational yet).
- v2.15: Contrib class parses namespaces.
- v3.00: Merged my code with Flcelloguy's code.
- v3.10: Removed 5,000-edit limit (the program now allows input from multiple sources) and added a basic Graphical user interface.
- v3.20: Enhanced the GUI, output now sent to a window, not to console.
- v3.30: Enabling parsing of edit summaries and automatic section-edit summaries.
- v3.31: Bug fix in edit summary parsing: now processes one-word edit summaries correctly
- v3.32: Minor bug fix.
- v3.40: Major code commit (fixed edit summary parsing bug), added new function to PurgeContribs.java to analyze edit summaries specifically.
- v4.00: Major release, added semi-automatic parsing capabilities, enhanced user interface, more classes and fixed unhandled exceptions.
Help
Please see the help page for help.
Code
For the current version (v4.00), please use the JAR (see instructions above) if possible, or get the code below. Any further enhancements to the tool will be available at the Java Sandbox before being copied here.
Source code. Click show to expand, and open the edit view to copy without problems.
Stats.java (same file as Flcelloguy's tool, only updated)
/**
* @author Flcelloguy et al.
* @program Flcelloguy's Tool (Stats.java)
* @version 4.10a; released April 13, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot code from http://en.wikipedia.org/wiki/User:Flcelloguy/Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
* Capabilities: Count edits, break down by namespace, count minor edits and calculate percentage
* Please leave this block in.
* Note: This new version does not require cut-and-pasting.
* Just go to
* http://en.wikipedia.org/w/index.php?
* title=Special:Contributions&target={{USERNAME}}&offset=0&limit=5000
* where {{USERNAME}} is the name of the user you want to run a check on.
*/
import javax.swing.JOptionPane;
import java.awt.Component;
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
import java.net.URL;
import java.util.HashMap;
import java.util.Iterator;
import java.util.TreeMap;
//import java.util.FileReader;
public class Stats
{
// Function added: provides human-readable output
//private static HashMap<String, Namespace> cleanMap = Namespace.getFullMap("en");
private static HashMap<String, StatBundle> statMap = new HashMap<String, StatBundle>();
private static final String CURRENTSTATUS =
"Current status: \n "
+ "Editcount \n Breakdown by namespace \n Minor edits usage \n Edit summary usage \n"
+ "Coming soon: \n "
+ "User friendly version \n First edit date";
protected StringBuilder console;
private Component frame;
public Stats(Component inFrame)
{
console = new StringBuilder();
frame = inFrame;
}
/*public void reset()
{
if (cleanMap == null)
cleanMap = Namespace.getFullMap("en");
console = new StringBuilder();
nsMap = cleanMap;
}*/
public static void main(String args[]) throws IOException
{
/**
* the GUI is too complex to screw up clean, crisp code like this, so
* I'm moving it to a separate class. --Titoxd
*/
MainGUI.main(null);
}
public void mainDownload(String targetUser) throws IOException
{
if (targetUser == null)
targetUser = JOptionPane.showInputDialog("Input file:", targetUser);
// JOptionPane
// .showMessageDialog(
// frame,CURRENTSTATUS,
// "Information", JOptionPane.INFORMATION_MESSAGE);
URL nextPage = new URL(
"http://en.wikipedia.org/wiki/Special:Contributions/"
+ targetUser.replace(" ", "_")
+ "?limit=5000");
StatBundle sb;
if (statMap.containsKey(targetUser))
{
sb = statMap.get(targetUser);
sb.parseFromConnection(GetContribs.getSource(nextPage));
}
else
{
sb = new StatBundle(nextPage, Namespace.getFullMap("en"), frame);
statMap.put(sb.username, sb);
}
buildConsole(sb);
System.out.print(console);
}
public void mainSingle(String inFile$) throws IOException
{
if (inFile$ == null)
inFile$ = JOptionPane.showInputDialog("Input file:", inFile$);
// JOptionPane
// .showMessageDialog(
// frame,CURRENTSTATUS,
// "Information", JOptionPane.INFORMATION_MESSAGE);
editcount(inFile$);
// JOptionPane.showMessageDialog(null, "Number of edits: "
// + editcount(inFile$), "Results",
// JOptionPane.INFORMATION_MESSAGE);
}
public void mainMulti(String[] args) throws IOException
{
String outFile$ = null;
String inFile$ = null;
outFile$ = JOptionPane.showInputDialog(frame,
"Enter the filename of the output file:", outFile$,
JOptionPane.QUESTION_MESSAGE);
FileWriter writer = new FileWriter(outFile$);
BufferedWriter out = new BufferedWriter(writer);
out.write("<ul>", 0, "<ul>".length());
out.newLine();
inFile$ = JOptionPane.showInputDialog(frame,
"Enter the filename of the next contributions file:", inFile$,
JOptionPane.QUESTION_MESSAGE);
while (inFile$ != null)
{
FileReader reader = new FileReader(inFile$);
BufferedReader in = new BufferedReader(reader);
String inString = "";
boolean endContribs = false; // marks whether all the
// contributions have been parsed
inString = in.readLine(); // read from file and discard
do
{
if ((inString.trim().compareTo("<ul>") == 0)
&& (endContribs == false)) // until the <ul> tag is
// reached,
{
do
{
inString = in.readLine(); // then start reading and
// recording
if ((inString.trim().compareTo("</ul>") != 0))
{
out.write(inString.trim(), 0, inString.length());
out.newLine();
// System.out.println(inString.trim());
}
else
{
endContribs = true;
}
}
while (endContribs != true);
}
inString = in.readLine(); // read from file and discard
}
while (inString != null);
in.close();
inFile$ = JOptionPane.showInputDialog(frame,
"Enter the filename of the next contributions file:",
inFile$, JOptionPane.QUESTION_MESSAGE);
}
out.write("</ul>", 0, "</ul>".length());
out.newLine();
out.close();
mainSingle(outFile$);
}
public void editcount(String inFile$) throws IOException
{
// TODO: inFile --> URL --> edit count using other code
// need getContribs to return fileReader/bufferedReader-like things
System.out.println("Computing...");
FileReader reader = new FileReader(inFile$);
BufferedReader in = new BufferedReader(reader);
// FileWriter writer = new FileWriter(outFile$); //for debugging
// BufferedWriter out = new BufferedWriter(writer); //for debugging
String inString = "";
Contrib outContrib;
boolean endContribs = false; // marks whether all the contributions
// have been parsed
inString = in.readLine(); // read from file and discard
do
{
if ((inString.trim().compareTo("<ul>") == 0)
&& (endContribs == false)) // until the <ul> tag is
// reached,
{
do
{
inString = in.readLine(); // then start reading and
// recording
if ((inString.trim().compareTo("</ul>") != 0))
{
// System.out.println(inString.trim());
outContrib = PurgeContribs.Parse(inString.trim());
// System.out.println(outString.trim());
//addContrib(outContrib); FIXME
// out.newLine();
}
else
{
endContribs = true;
}
}
while (endContribs != true);
}
inString = in.readLine(); // read from file and discard
}
while (inString != null);
in.close();
// out.close();
// Prints out statistics
buildConsole(statMap.get("")); // FIXME
System.out.print(console);
}
public void buildConsole(StatBundle sb)
{
console = new StringBuilder();
HashMap<String, Namespace> nsMap = sb.getNamespaceMap();
if (sb.badUsername)
{
console.append("No such user.\n(Check spelling and capitalization)");
return;
}
console.append("Statistics for: " + sb.username + "\n");
if (sb.noEdits)
{
console.append("Account exists, but no edits.\n(Check spelling and capitalization)");
return;
}
console.append("- Total: " + sb.total + " -\n");
Namespace ns; // reusable memory location
Iterator<String> iter1 = nsMap.keySet().iterator();
// Convert arbitrary Name->Namespace HashMapping to
// sorted index->Namespace TreeMapping
TreeMap<Integer, Namespace> indexNamespaceMap = new TreeMap <Integer, Namespace>();
while (iter1.hasNext())
{
ns = nsMap.get(iter1.next());
indexNamespaceMap.put(ns.getMediawikiIndex(), ns);
}
Iterator<Integer> iter2 = indexNamespaceMap.keySet().iterator();
int next = iter2.next();
for (; next < 0 && iter2.hasNext(); next = iter2.next());
// next has the value 0 or larger than 0 here, or hasNext() false
while (iter2.hasNext())
{
ns = indexNamespaceMap.get(next);
int count = ns.getCount();
if (count > 0)
console.append(ns.getEnglishName() + ": " + count + "\n");
next = iter2.next();
}
// still one more after it drops out
{
ns = indexNamespaceMap.get(next);
int count = ns.getCount();
if (count > 0)
console.append(ns.getEnglishName() + ": " + count + "\n");
}
console.append("-------------------" + "\n"
+ "Total edits: " + sb.total + "\n");
console.append("Minor edits: " + sb.minor + "\n");
console.append("Edits with edit summary: " + sb.summary + "\n");
console.append("Edits with manual edit summary: " + sb.manualSummary + "\n");
console.append("Percent minor edits: "
+ ((int)((float)(sb.minor * 10000) / (float)(sb.total))/100.0) + "% *\n");
console.append("Percent edit summary use: "
+ ((int)((float)(sb.summary * 10000) / (float)(sb.total))/100.0) + "% *\n");
console.append("Percent manual edit summary use: "
+ ((int)((float)(sb.manualSummary * 10000) / (float)(sb.total))/100.0) + "% *\n");
console.append("-------------------\n");
console.append("* - percentages are rounded down to the nearest hundredth.\n");
console.append("-------------------\n");
//return total;
}
}
PurgeContribs.java
/**
* @author Titoxd
* @program HTML -> ContribFile converter for Flcelloguy's Tool
* @version 4.10; released April 13, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.io.IOException;
import java.util.StringTokenizer;
public class PurgeContribs
{
public static void main(String[] args)
{
try
{
System.out.println(
getNextDiffs(
"<p>(Newest | <a href=\"/w/index.php?title=Special:" +
"Contributions&go=first&limit=5000&target=" +
"AySz88\">Oldest</a>) View (Newer 5000) (<a href=\"" +
"/w/index.php?title=Special:Contributions&offset=" +
"20060329014125&limit=5000&target=AySz88\">Older" +
" 5000</a>) (<a href=\"/w/index.php?title=Special:" +
"Contributions&limit=20&target=AySz88\">20</a>" +
" | <a href=\"/w/index.php?title=Special:Contributions" +
"&limit=50&target=AySz88\">50</a> | <a href=\"" +
"/w/index.php?title=Special:Contributions&limit=100" +
"&target=AySz88\">100</a> | <a href=\"/w/index.php?" +
"title=Special:Contributions&limit=250&target=" +
"AySz88\">250</a> | <a href=\"/w/index.php?title=Special" +
":Contributions&limit=500&target=AySz88\">500</a>)" +
".</p>")
);
}
catch (Exception e)
{
System.out.println(e);
}
}
/**
* @param purgedLine
* (input line in raw HTML, leading and trailing whitespace
* removed)
* @return Contrib class object: for analysis
* @throws IOException
*/
public static Contrib Parse(String purgedLine) throws IOException
{
/**** Take out the <li> tags ****/
String midString1;
String timeStamp;
String editSummary = null;
String autoSummary = null;
boolean minorEdit = false;
boolean endLoop = false;
boolean newestEdit = false;
//boolean sectionParsed = false;
midString1 = purgedLine.substring(4, purgedLine.length() - 5);
/**** Process the time stamp ****/
StringTokenizer token;
token = new StringTokenizer(midString1.trim());
{
String time = token.nextToken();
String day = token.nextToken();
String month = token.nextToken();
String year = token.nextToken();
timeStamp = time + " " + day + " " + month + " " + year;
}
/**** Process the page name ****/
String dummy = token.nextToken(); // get rid of (<a
/*String URL =*/ token.nextToken();
StringBuilder titleBuilder = new StringBuilder();
//String pageName = URL.substring(25, URL.length() - 20);
///**** Get rid of a few extra tokens ****/
// start with the "title" piece
do
{
dummy = token.nextToken();
titleBuilder.append(dummy); titleBuilder.append(" ");
endLoop = false;
if (dummy.lastIndexOf('<') != -1)
{
if (dummy.substring(dummy.lastIndexOf('<'),
dummy.lastIndexOf('<') + 3).compareTo("</a>") != 0)
endLoop = true;
}
}
while (endLoop == false);
String title = titleBuilder.toString();
String pageName = title.substring(7, title.length() - 11);
/**** Do the same with the diff link ****/
dummy = token.nextToken(); // get rid of (<a
String diffURL = token.nextToken(); // this URL is not needed, so it is dummied out
String diffIDString = diffURL.substring(diffURL.lastIndexOf("=")+1, diffURL.length()-1); // ditto
long diffID = Long.parseLong(diffIDString);
do
{
endLoop = false;
dummy = token.nextToken();
if (dummy.lastIndexOf('<') != -1)
{
if (dummy.substring(dummy.lastIndexOf('<'),
dummy.lastIndexOf('<') + 3).compareTo("</a>") != 0)
endLoop = true;
}
}
while (endLoop == false);
String dummyPageName;
/**** Determine if edit is minor or not ****/
dummy = token.nextToken(); // get rid of (<span
dummy = token.nextToken(); // read the next token; it should be class="minor">m</span> if a minor edit
if (dummy.compareTo("class=\"minor\">m</span>") == 0)
{
minorEdit = true;
dummyPageName = null;
}
else
{
minorEdit = false;
dummyPageName = dummy;
}
if (dummyPageName == null) // if it was a minor edit, advance token
// cursor to match non-minor edits
{
dummy = token.nextToken(); // get rid of <a
dummyPageName = token.nextToken();
}
do
{
endLoop = false;
dummy = token.nextToken();
if (dummy.lastIndexOf('<') != -1)
{
if (dummy.substring(dummy.lastIndexOf('<'),
dummy.lastIndexOf('<') + 3).compareTo("</a>") != 0)
endLoop = true;
}
}
while (endLoop == false);
/**** flush the StringTokenizer ****/
StringBuilder tokenDump = new StringBuilder();
String dump;
if (token.hasMoreTokens() == true) //
{
do
{
tokenDump.append(token.nextToken());
tokenDump.append(' ');
} while (token.hasMoreTokens()==true);
tokenDump.trimToSize();
dump = tokenDump.toString();
}
else //not top edit, no edit summary
{
dump = null;
}
/**** Top edit? ****/
if (dump != null && dump.contains("<strong> (top)</strong>") == true)
{
newestEdit = true;
dump = dump.substring(0,dump.indexOf("<strong> (top)</strong>")); //truncate to remove rollback links and other garbage
dump = dump.trim();
}
else newestEdit = false;
/**** Process edit summary ****/
String[] summaries = ParseSummary(dump);
autoSummary = summaries[0];
editSummary = summaries[1];
Contrib contrib = new Contrib(timeStamp, pageName, minorEdit,
editSummary, autoSummary, newestEdit, diffID);
return contrib;
}
/**
* @param dump
* @return String[2] array, String[0] is the auto summary, String[1] is the manual summary
*/
private static String[] ParseSummary(String dump)
{
// TODO: clean this up
/****Check that there is an edit summary to begin with ****/
if (dump == null || dump.equals("")) return new String[] {null, null};
String[] summaryArray = new String[2];
if (dump.indexOf("<span class=\"autocomment\">") != -1) //autocomment present
{
String autoSummary = // everything within the <span class="autocomment">
dump.substring(
dump.indexOf("<span class=\"autocomment\">"),
dump.indexOf("</span>")+7);
summaryArray[0] = autoSummary.substring(autoSummary.indexOf("<a href="));
summaryArray[1] = dump.substring(0,dump.indexOf(autoSummary)) + dump.substring(dump.indexOf(autoSummary)+ autoSummary.length()).trim();
summaryArray[0] = summaryArray[0].substring(0,summaryArray[0].lastIndexOf("<"));
summaryArray[0] = summaryArray[0].substring(summaryArray[0].lastIndexOf(">")+1);
if (summaryArray[0].endsWith(" -")== true)
{
summaryArray[0] = summaryArray[0].substring(0,summaryArray[0].length()-1);
}
}
else
{
if (dump != "") summaryArray[1] = dump;
}
if (summaryArray[1] != null && summaryArray[1].length() != 0)
{
summaryArray[1] = summaryArray[1].substring(summaryArray[1].indexOf(">")+1,summaryArray[1].lastIndexOf("<"));
summaryArray[1] = summaryArray[1].trim();
if (summaryArray[1].equals("()")) summaryArray[1]=null;
}
if (summaryArray[0]=="") summaryArray[0]=null;
if (summaryArray[1]=="") summaryArray[1]=null; //so the edge cases don't trigger exceptions
if (summaryArray[0] != null) summaryArray[0] = summaryArray[0].trim();
return summaryArray;
}
/**
* "Next 5000" contributions link parser
* @param inLine (<code>String</code> object that contains the raw HTML for the Contributions line
* @return String with the relative URL of the link if the link is available, null if it is not
*/
public static String getNextDiffs(String inLine) throws IOException
{
// if no such user, it would have been caught already
if (inLine.contains("<p>No changes were found matching these criteria."))
throw new IOException(StatBundle.NO_EDITS);
StringTokenizer midToken = new StringTokenizer(inLine,"()");
String midLine[] = new String[midToken.countTokens()];
for (int i = 0; i < midLine.length; i++)
{
midLine[i] = midToken.nextToken();
}
StringTokenizer token = new StringTokenizer(midLine[5],"<>");
String tag = token.nextToken();
String link = null;
boolean diffObtained = false;
//FIXME: Internationalize this function
do
{
if (tag.contains("href=\"/w/index.php?title=Special:Contributions&") == true)
{
if (tag.contains("limit=5000") == true)
{
if (token.nextToken().contains("Older") == true)
link = tag.split("\"")[1];
link = link.replace("&","&");
diffObtained = true;
}
}
if (token.hasMoreTokens() == true)
{
tag = token.nextToken();
}
else
{
diffObtained = true;
}
} while (diffObtained == false);
return link;
}
public static String getUsername(String line) throws IOException
{
if (!line.contains("title=\"User:"))
throw new IOException(StatBundle.NO_SUCH_USER);
return line.substring(
line.indexOf("title=\"User:") + 12,
line.indexOf("\">", line.indexOf("title=\"User:")));
}
}
Contrib.java
/**
* @author Titoxd
* @program Contribution class for Flcelloguy's Tool
* @version 4.04a; released February 20, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.util.Calendar;
import java.util.GregorianCalendar;
import java.util.HashMap;
import java.util.StringTokenizer;
public class Contrib implements Comparable
{
protected String timeStamp;
protected String pageName, namespace, shortName;
protected boolean minorEdit;
protected String editSummary, autoSummary;
// editSummary is only the manual part
protected boolean topEdit;
protected long editID;
protected Calendar date = new GregorianCalendar();
protected static final String[][] NAMESPACE_ARRAY = setArray();
public Contrib(String inStamp, String inName, boolean inMin, String inSummary, String inAuto, boolean inTop, long inEditID)
{
timeStamp = inStamp;
pageName = inName;
String[] nameArray=pageName.split(":",2);
if (nameArray.length == 1)
{
namespace = Namespace.MAINSPACENAME;
shortName = pageName;
}
else
{
namespace = nameArray[0];
shortName = nameArray[1];
}
minorEdit = inMin;
editSummary = inSummary;
autoSummary = inAuto;
topEdit = inTop;
editID = inEditID;
setDate(timeStamp);
}
private static String[][] setArray()
{
String[] NAMESPACE_ARRAY =
{"Media", "Special", "", "Talk", "User", "User talk",
"Wikipedia", "Wikipedia talk", "Image", "Image talk", "MediaWiki", "MediaWiki talk",
"Template", "Template talk", "Help", "Help talk", "Category", "Category talk",
"Portal", "Portal talk"};
int[] INDEX_ARRAY =
{-2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 100, 101};
String tempArray[][] = new String[NAMESPACE_ARRAY.length][2];
for (int i = 0; i < NAMESPACE_ARRAY.length; i++)
{
tempArray[i][0] = NAMESPACE_ARRAY[i];
tempArray[i][1] = String.valueOf(INDEX_ARRAY[i]);
}
return tempArray;
}
/**
* Constructor for the internal Calendar object
*/
private void setDate(String dateString)
{
StringTokenizer stamp = new StringTokenizer(dateString," :,");
int hour = Integer.parseInt(stamp.nextToken());
int minute = Integer.parseInt(stamp.nextToken());
int day = Integer.parseInt(stamp.nextToken());
String month = stamp.nextToken();
int year = Integer.parseInt(stamp.nextToken());
int monthNo = 0;
{
if (month.equalsIgnoreCase("January")) monthNo = Calendar.JANUARY;
else if (month.equalsIgnoreCase("February")) monthNo= Calendar.FEBRUARY;
else if (month.equalsIgnoreCase("March")) monthNo = Calendar.MARCH;
else if (month.equalsIgnoreCase("April")) monthNo = Calendar.APRIL;
else if (month.equalsIgnoreCase("May")) monthNo = Calendar.MAY;
else if (month.equalsIgnoreCase("June")) monthNo = Calendar.JUNE;
else if (month.equalsIgnoreCase("July")) monthNo = Calendar.JULY;
else if (month.equalsIgnoreCase("August")) monthNo = Calendar.AUGUST;
else if (month.equalsIgnoreCase("September")) monthNo = Calendar.SEPTEMBER;
else if (month.equalsIgnoreCase("October")) monthNo = Calendar.OCTOBER;
else if (month.equalsIgnoreCase("November")) monthNo = Calendar.NOVEMBER;
else if (month.equalsIgnoreCase("December")) monthNo = Calendar.DECEMBER;
}
date.set(year, monthNo, day, hour, minute);
date.set(Calendar.SECOND,0); //these fields shouldn't be used, so they're zeroed out
date.set(Calendar.MILLISECOND,0);
}
protected void checkCorrectNamespace(HashMap<String, Namespace> nsMap)
{
if (!nsMap.containsKey(namespace))
{
System.out.println("Page: "+ namespace+":"+shortName+" set to main namespace."); //for debug purposes
namespace = Namespace.MAINSPACENAME; //set to main namespace by default
}
}
public int compareTo(Object o) //may throw ClassCastException
//sorts by editID (same as sorting by date)
{
Contrib con = (Contrib) o;
if (editID > con.editID) return 1;
if (editID == con.editID) return 0;
return -1;
}
public String toString()
{
String returnString = "Time: " + timeStamp + "\r" +
"Page: " + pageName + " (Namespace: " + namespace + "; Article: " + shortName + ")\r" +
"Minor edit: " + minorEdit + "\r" +
"Edit Summary: " + editSummary + "\r" +
"Most recent edit: " + topEdit;
return returnString;
}
}
MainGUI.java
/**
* @author Titoxd
* @program Graphical User Interface shell for Flcelloguy's Tool
* @version 4.00; released January 15, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import javax.swing.JDesktopPane;
import javax.swing.JMenu;
import javax.swing.JMenuItem;
import javax.swing.JMenuBar;
import javax.swing.JFrame;
import javax.swing.KeyStroke;
import javax.swing.UIManager;
import java.awt.event.*;
import java.awt.*;
public class MainGUI extends JFrame
implements ActionListener {
static JDesktopPane desktop;
public MainGUI() {
super("Flcelloguy's Tool");
//Make the big window be indented 50 pixels from each edge
//of the screen.
int inset = 100;
Dimension screenSize = Toolkit.getDefaultToolkit().getScreenSize();
setBounds(inset, inset,
screenSize.width - inset*2,
screenSize.height - inset*2);
//Set up the GUI.
desktop = new JDesktopPane(); //a specialized layered pane
createFrame(); //create first "window"
setContentPane(desktop);
setJMenuBar(createMenuBar());
//Make dragging a little faster but perhaps uglier.
desktop.setDragMode(JDesktopPane.OUTLINE_DRAG_MODE);
}
protected JMenuBar createMenuBar() {
JMenuBar menuBar = new JMenuBar();
//Set up the lone menu.
JMenu menu = new JMenu("Document");
menu.setMnemonic(KeyEvent.VK_D);
menuBar.add(menu);
//Set up the first menu item.
JMenuItem menuItem = new JMenuItem("New Query");
menuItem.setMnemonic(KeyEvent.VK_N);
menuItem.setAccelerator(KeyStroke.getKeyStroke(
KeyEvent.VK_N, ActionEvent.ALT_MASK));
menuItem.setActionCommand("new");
menuItem.addActionListener(this);
menu.add(menuItem);
//Set up the second menu item.
menuItem = new JMenuItem("Quit");
menuItem.setMnemonic(KeyEvent.VK_Q);
menuItem.setAccelerator(KeyStroke.getKeyStroke(
KeyEvent.VK_Q, ActionEvent.ALT_MASK));
menuItem.setActionCommand("quit");
menuItem.addActionListener(this);
menu.add(menuItem);
return menuBar;
}
//React to menu selections.
public void actionPerformed(ActionEvent e) {
if ("new".equals(e.getActionCommand())) { //new
createFrame();
} else { //quit
quit();
}
}
//Create a new internal frame.
protected void createFrame() {
QueryFrame frame = new QueryFrame();
frame.setVisible(true);
desktop.add(frame);
try {
frame.setSelected(true);
} catch (java.beans.PropertyVetoException e) {}
}
protected static void createTextFrame(String passedString) {
ResultsFrame frame = new ResultsFrame(passedString);
frame.setVisible(true);
desktop.add(frame);
try {
frame.setSelected(true);
} catch (java.beans.PropertyVetoException e) {}
}
protected void quit() {
System.exit(0);
}
/**
* Create the GUI and show it. For thread safety,
* this method should be invoked from the
* event-dispatching thread.
*/
private static void createAndShowGUI()
{
try
{
UIManager.setLookAndFeel(
UIManager.getSystemLookAndFeelClassName());
} catch (Exception e) { }
//Make sure we have nice window decorations.
JFrame.setDefaultLookAndFeelDecorated(true);
//Create and set up the window.
MainGUI frame = new MainGUI();
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
//Display the window.
frame.setVisible(true);
}
public static void main(String[] args) {
//Schedule a job for the event-dispatching thread:
//creating and showing this application's GUI.
javax.swing.SwingUtilities.invokeLater(new Runnable() {
public void run() {
createAndShowGUI();
}
});
}
}
QueryFrame.java
/**
* @author Titoxd
* @program Query Graphical User Interface for Flcelloguy's Tool
* @version 4.00; released February 23, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import javax.swing.BorderFactory;
import javax.swing.JButton;
import javax.swing.JComboBox;
//import javax.swing.JFrame;
import javax.swing.Box;
import javax.swing.BoxLayout;
import javax.swing.JFileChooser;
import javax.swing.JInternalFrame;
import javax.swing.JLabel;
import javax.swing.JPanel;
//import javax.swing.JScrollBar;
import javax.swing.JScrollPane;
import javax.swing.JSplitPane;
import javax.swing.JTabbedPane;
import javax.swing.JTextArea;
import javax.swing.JTextField;
import javax.swing.SpringLayout;
import java.awt.event.*;
import java.awt.*;
import java.io.IOException;
/* Used by MainGUI.java. */
public class QueryFrame extends JInternalFrame implements ActionListener
{
static int openFrameCount = 0;
static final int xOffset = 10, yOffset = 10;
static final Dimension MINIMUM_TEXTPANEL_SIZE = new Dimension(250, 250);
private JLabel
topLabel = new JLabel("Online help"),
label = new JLabel("Flcelloguy's Tool: Statistics for editcounters.");
private JTextField helpURL = new JTextField("http://en.wikipedia.org/wiki/WP:WPEC/FT/H"),
nameInput = new JTextField();
private static final String NA_TEXT = "No data available.";
private String[] phases = {
"Download",
"Single local (\u22645,000 edits)", // "\u2264" is "?" (less than or equal to sign)
"Multiple local (\u22655,000 edits)" // "\u2265" is "?" (greater than or equal to sign)
};
private JComboBox methodBox = new JComboBox(phases);
private JButton button = new JButton("Proceed");
private CardLayout locationOption = new CardLayout();
private JPanel row3 = new JPanel(locationOption);
private JLabel fileChooseDesc = new JLabel();
private JTextField filePathField = new JTextField();
private JFileChooser fc = new JFileChooser();
private JButton browseButton = new JButton("Browse...");
private JTabbedPane outputTabs = new JTabbedPane();
private JPanel textOutputPanel = new JPanel();
private JTextArea textOutput = new JTextArea(20, 20);
private JScrollPane areaScrollPane = new JScrollPane(textOutput);
private JPanel graphOutputPanel = new JPanel();
private JPanel treeOutputPanel = new JPanel();
private Stats statStorage = new Stats(this.getRootPane());
public QueryFrame()
{
super("New Query " + (++openFrameCount), true, // resizable
true, // closable
true, // maximizable
true);// iconifiable
// ...Create the GUI and put it in the window...
JPanel panel = (JPanel) createComponents();
panel.setBorder(BorderFactory.createEmptyBorder(20, // top
30, // left
10, // bottom
30) // right
);
getContentPane().add(panel);
// ...Then set the window size or call pack...
pack();
// Set the window's location.
setLocation(xOffset * openFrameCount, yOffset * openFrameCount);
}
public Component createComponents()
{
GridBagConstraints optionC = new GridBagConstraints();
GridBagConstraints c = new GridBagConstraints();
//label.setLabelFor(button);
button.setMnemonic('i');
button.addActionListener(this);
methodBox.setSelectedIndex(0);
methodBox.addActionListener(this);
JPanel mainPanel = new JPanel(new BorderLayout());
JPanel optionPanel = new JPanel();
optionPanel.setLayout(new GridBagLayout());
optionC.gridx = 0; optionC.gridy = 0;
optionC.weightx = 1; optionC.weighty = .5;
optionC.anchor = GridBagConstraints.WEST;
optionC.fill = GridBagConstraints.HORIZONTAL;
{
JPanel row1 = new JPanel(new GridBagLayout());
GridBagConstraints tempC = new GridBagConstraints();
tempC.gridx = 0; tempC.gridy = 0;
tempC.weightx = 0;
row1.add(topLabel, tempC);
tempC.gridx++;
tempC.weightx = 1;
tempC.fill = GridBagConstraints.BOTH;
row1.add(helpURL, tempC);
helpURL.setEditable(false);
optionPanel.add(row1, optionC);
}
{
/*
JPanel row2 = new JPanel(new GridBagLayout());
GridBagConstraints tempC = new GridBagConstraints();
tempC.gridx = 0; tempC.gridy = 0;
tempC.weightx = 0;
row2.add(new JLabel("Statistics for:"), tempC);
tempC.gridx++;
tempC.weightx = 1;
tempC.fill = GridBagConstraints.BOTH;
row2.add(nameInput, tempC);
tempC.fill = GridBagConstraints.NONE;
tempC.weightx = 0;
tempC.gridx++;
JLabel via = new JLabel("via");
via.setBorder(BorderFactory.createEmptyBorder(0, 2, 0, 2));
row2.add(via, tempC);
tempC.gridx++;
row2.add(methodBox, tempC);
tempC.gridx++;
row2.add(button, tempC);
*/
/*
JPanel row2 = new JPanel();
row2.setLayout(new BoxLayout(row2,BoxLayout.X_AXIS));
row2.add(new JLabel("Statistics for:"));
row2.add(nameInput);
JLabel via = new JLabel("via");
via.setBorder(BorderFactory.createEmptyBorder(0, 2, 0, 2));
row2.add(via);
row2.add(methodBox);
row2.add(button);
*/
SpringLayout layout = new SpringLayout();
JPanel row2 = new JPanel(layout);
JLabel lineStart = new JLabel("Statistics for:");
JLabel via = new JLabel("via");
nameInput.setMinimumSize(new Dimension(
methodBox.getPreferredSize().width + via.getPreferredSize().width,
nameInput.getMinimumSize().height));
nameInput.setPreferredSize(nameInput.getMinimumSize());
methodBox.setEditable(false);
row2.add(lineStart);
row2.add(nameInput);
row2.add(via);
row2.add(methodBox);
row2.add(button);
row2.add(label);
layout.putConstraint(
SpringLayout.WEST, lineStart,
0,
SpringLayout.WEST, row2);
layout.putConstraint(
SpringLayout.WEST, nameInput,
0,
SpringLayout.EAST, lineStart);
layout.putConstraint(
SpringLayout.WEST, via,
0,
SpringLayout.EAST, lineStart);
layout.putConstraint(
SpringLayout.WEST, methodBox,
2,
SpringLayout.EAST, via);
layout.putConstraint(
SpringLayout.WEST, button,
5,
SpringLayout.EAST, nameInput);
layout.putConstraint(
SpringLayout.EAST, row2,
2,
SpringLayout.EAST, button);
layout.putConstraint(
SpringLayout.EAST, label,
0,
SpringLayout.EAST, row2);
layout.putConstraint(
SpringLayout.NORTH, lineStart,
5,
SpringLayout.NORTH, row2);
layout.putConstraint(
SpringLayout.NORTH, nameInput,
2,
SpringLayout.NORTH, row2);
layout.putConstraint(
SpringLayout.NORTH, via,
7,
SpringLayout.SOUTH, nameInput);
layout.putConstraint(
SpringLayout.NORTH, methodBox,
2,
SpringLayout.SOUTH, nameInput);
layout.putConstraint(
SpringLayout.NORTH, button,
10,
SpringLayout.NORTH, row2);
layout.putConstraint(
SpringLayout.NORTH, label,
0,
SpringLayout.SOUTH, methodBox);
layout.putConstraint(
SpringLayout.SOUTH, row2,
0,
SpringLayout.SOUTH, label);
optionC.gridy++;
//optionC.gridwidth = GridBagConstraints.REMAINDER;
optionC.fill = GridBagConstraints.HORIZONTAL;
optionPanel.add(row2, optionC);
optionC.fill = GridBagConstraints.NONE;
}
/*
{
JPanel row2a = new JPanel();
row2a.setLayout(new BoxLayout(row2a, BoxLayout.X_AXIS));
row2a.add(Box.createHorizontalGlue(), c);
row2a.add(label, c);
optionC.gridx = 0; optionC.gridy++;
optionC.fill = GridBagConstraints.HORIZONTAL;
optionPanel.add(row2a, optionC);
optionC.fill = GridBagConstraints.NONE;
}
*/
// row3 was already declared
JPanel filePanel = new JPanel(new GridBagLayout());
c.gridx = 0; c.gridy = 0;
c.fill = GridBagConstraints.NONE;
c.weightx = 0;
filePanel.add(fileChooseDesc);
c.gridx++;
c.weightx = 1;
c.fill = GridBagConstraints.BOTH;
filePanel.add(filePathField);
c.fill = GridBagConstraints.NONE;
c.weightx = 0;
c.gridx++;
browseButton.addActionListener(this);
filePanel.add(browseButton);
filePathField.setPreferredSize(new Dimension( // FIXME: band-aid
150,
filePathField.getPreferredSize().height));
row3.add(new JLabel("[en.wikipedia.com, es, Wikimedia sites in en, etc.] [New Set]"), phases[0]);
//row3.add(filePanel, phases[1]);
row3.add(filePanel, phases[2]);
locationOption.show(row3, phases[0]);
optionC.gridy++;
optionC.fill = GridBagConstraints.BOTH;
optionC.weightx = 1;
optionPanel.add(row3, optionC);
optionC.fill = GridBagConstraints.NONE;
{
JPanel row4 = new JPanel(new GridBagLayout());
row4.setBorder(BorderFactory
.createTitledBorder("Filter by date (inclusive)"));
c.gridx = 0; c.gridy = 0;
c.gridheight = 1;
row4.add(new JLabel("From:"), c);
c.gridy++;
row4.add(new JLabel("To:"), c);
c.gridy = 0; c.gridx++;
row4.add(new JLabel("[mm/dd/yyyy]"), c);
c.gridy++;
row4.add(new JLabel("[mm/dd/yyyy]"), c);
c.gridy = 0; c.gridx++;
c.gridheight = 2;
c.weightx = 1;
row4.add(Box.createHorizontalGlue());
c.weightx = 0;
c.gridx++;
row4.add(new JLabel("or"), c);
c.gridx++;
c.weightx = 1;
row4.add(Box.createHorizontalGlue());
c.weightx = 0;
c.gridx++; //c.gridy = 0;
c.gridheight = 1;
row4.add(new JLabel("[n] [days/months/edits]"), c);
c.gridy++;
row4.add(new JLabel("[before/after] [mm/dd/yyyy]"), c);
optionC.gridy++;
optionPanel.add(row4, optionC);
}
{
JPanel row5 = new JPanel();
row5.setLayout(new BoxLayout(row5, BoxLayout.X_AXIS));
JPanel graphTypes = new JPanel(new GridBagLayout());
graphTypes.setBorder(BorderFactory
.createTitledBorder("Graph Type"));
c.anchor = GridBagConstraints.WEST;
c.gridx = 0; c.gridy = 0;
graphTypes.add(new JLabel("O Stacked"), c);
c.gridy++;
graphTypes.add(new JLabel("O Unstacked"), c);
c.gridy++;
graphTypes.add(new JLabel("O Proportion"), c);
c.gridy++;
c.gridwidth = 2;
c.anchor = GridBagConstraints.CENTER;
graphTypes.add(new JLabel("O Pie"), c);
c.anchor = GridBagConstraints.WEST;
c.gridwidth = 1;
c.gridx++; c.gridy = 0;
graphTypes.add(new JLabel("O Line"), c);
c.gridy++;
graphTypes.add(new JLabel("O Area"), c);
c.gridy++;
graphTypes.add(new JLabel("O Histogram"), c);
row5.add(graphTypes);
row5.add(Box.createHorizontalGlue());
JPanel graphAnalyses = new JPanel(new GridBagLayout());
graphAnalyses.setBorder(BorderFactory
.createTitledBorder("Time Axis"));
c.anchor = GridBagConstraints.WEST;
c.gridx = 0; c.gridy = 0;
graphAnalyses.add(new JLabel("O Continuous"), c);
c.gridy++;
graphAnalyses.add(new JLabel("O Sum over week"), c);
c.gridy++;
graphAnalyses.add(new JLabel("O Sum over day"), c);
c.gridy++;
graphAnalyses.add(new JLabel("Resolution: [n] [hours/days/edits]"), c);
c.gridx = 1;
row5.add(graphAnalyses);
optionC.gridy++;
optionPanel.add(row5, optionC);
}
{
JPanel row6 = new JPanel(new GridBagLayout());
row6.setBorder(BorderFactory
.createTitledBorder("Filters and splits"));
c.gridx = 0; c.gridy = 0;
row6.add(new JLabel("O Major/minor split X Only Major X Only Minor"), c);
c.gridy++;
row6.add(new JLabel("O Namespaces [groups, exceptions, and colors]"), c);
c.gridy++;
row6.add(new JLabel("X Top [n] [% or articles] edited"), c);
c.gridy++;
row6.add(new JLabel("X Exclude articles with less than [n] edits"), c);
c.gridy++;
row6.add(new JLabel("O Edit summary split"), c);
optionC.gridy++;
optionPanel.add(row6, optionC);
}
/*
{
JPanel row7 = new JPanel();
row7.add(new JLabel("O Text")); // use tabs instead?
row7.add(new JLabel("O Graph"));
row7.add(new JLabel("O Tree"));
optionC.gridy++;
optionPanel.add(row7, optionC);
}
*/
textOutput.setFont(new Font("Ariel", Font.PLAIN, 12));
textOutput.setLineWrap(true);
textOutput.setWrapStyleWord(true);
textOutput.setText(NA_TEXT);
//textOutput.setPreferredSize(new Dimension(
//
// textOutput.getPreferredSize().height));
areaScrollPane
.setVerticalScrollBarPolicy(JScrollPane.VERTICAL_SCROLLBAR_ALWAYS);
areaScrollPane.setMinimumSize(MINIMUM_TEXTPANEL_SIZE);
textOutputPanel.setLayout(new BorderLayout());
textOutputPanel.add(areaScrollPane, BorderLayout.CENTER);
outputTabs.addTab("Text", textOutputPanel);
outputTabs.addTab("Graph", graphOutputPanel);
outputTabs.addTab("Tree", treeOutputPanel);
optionC.gridx = 1;
optionC.gridheight = GridBagConstraints.REMAINDER;
optionC.gridy = 0;
//optionC.gridwidth = GridBagConstraints.REMAINDER;
optionC.weightx = 1; optionC.weighty = 1;
optionC.fill = GridBagConstraints.BOTH;
//optionPanel.add(outputTabs, optionC);
JSplitPane splitPane = new JSplitPane(JSplitPane.HORIZONTAL_SPLIT,
optionPanel, outputTabs);
mainPanel.add(splitPane, BorderLayout.CENTER);
return mainPanel;
}
public void actionPerformed(ActionEvent event)
{
if (event.getSource() == methodBox)
{
if (methodBox.getSelectedItem().equals(phases[0]))
{
locationOption.show(row3, phases[0]);
label.setText("Please put the username in the upper-left and the site below");
}
else if (methodBox.getSelectedItem().equals(phases[1]))
{
fileChooseDesc.setText("File path:");
locationOption.show(row3, phases[2]); // FIXME: "phases[2]" is band-aid fix
label.setText("Please indicate the file to load below");
}
else if (methodBox.getSelectedItem().equals(phases[2]))
{
fileChooseDesc.setText("First file path:");
locationOption.show(row3, phases[2]);
label.setText("Please indicate the first file to load below");
}
}
else if ("Proceed".equals(event.getActionCommand()))
{
try
{
setCursor(Cursor.getPredefinedCursor(Cursor.WAIT_CURSOR));
//if (statMap.containsKey(nameInput))
//statStorage.reset();
if (methodBox.getSelectedItem().equals(phases[0]))
{
statStorage.mainDownload(nameInput.getText());
}
else if (methodBox.getSelectedItem().equals(phases[1]))
{
statStorage.mainSingle(filePathField.getText());
}
else if (methodBox.getSelectedItem().equals(phases[2]))
{
statStorage.mainMulti(null);
}
setCursor(Cursor.getPredefinedCursor(Cursor.DEFAULT_CURSOR));
textOutput.setText(statStorage.console.toString());
textOutput.setCaretPosition(0);
//The following didn't work:
//JScrollBar scroll = areaScrollPane.getVerticalScrollBar();
//scroll.setValue(scroll.getMinimum());
}
catch (IOException e)
{
// TODO Auto-generated catch block
e.printStackTrace();
}
}
else if (event.getSource() == browseButton)
{
if (fc.showOpenDialog(this) == JFileChooser.APPROVE_OPTION)
{
filePathField.setText(fc.getSelectedFile().toString());
}
}
}
}
GetContribs.java
/**
* @author AySz88, Titoxd
* @program Remote source reader for Flcelloguy's Tool
* @version 4.00; released April 13, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.io.BufferedInputStream;
import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.util.Calendar;
import java.util.Date;
import java.util.HashMap;
public final class GetContribs
{
//TODO: return fileReader/bufferedReader-like things
private static long killbitCacheSaveTime = 0;
// 0 = the epoch, a few hours before 1970, so killbit expired by a safe margin :p
private static boolean cachedKillValue = true;
// initialize to true to avoid loophole from rolling back time to before 1970
private static final String KILLURL = "http://en.wikipedia.org/w/index.php?title=Wikipedia:WikiProject_edit_counters/Flcelloguy%27s_Tool/Configs&action=raw";
private static final long CACHETIME = 1000*60*10; //10 minutes (arbitrary) in milliseconds
// CACHETIME will bug out if > ~35 yrs (see comment next to killbitCacheSaveTime)
private static final int PARCELSIZE = 4096; //arbitrary number
private static HashMap<URL, CachedPage> cache = new HashMap<URL, CachedPage>();
public static void main(String[] args) // testing purposes only
{
try
{
//String content =
getSource(new URL(
"http://en.wikipedia.org/"));
getSource(new URL(
"http://en.wikipedia.org/"));
// getSource(new URL(
// "http://en.wikipedia.org/w/index.php?title=Special:Contributions&target=AySz88&offset=0&limit=5000"));
// getSource(new URL(
// "http://en.wikipedia.org/wiki/Special:Contributions/Essjay"));
// getSource(new URL(
// "http://en.wikipedia.org/wiki/Special:Contributions/Titoxd"));
//System.out.println(content);
}
catch (MalformedURLException e)
{
e.printStackTrace();
}
}
// different ways to call getSource:
public static String getSource(URL location, boolean overrideCache) throws MalformedURLException
{
if (killBit() == false)
return getSourceDirect(location, overrideCache);
else
{
System.out.println("Killbit active; Scraper is disabled. Try again later.");
return "Killbit active; scraper is disabled. Try again later.";
}
}
public static String getSource(URL location) throws MalformedURLException
{
return getSource(location, false);
}
public static String getSource(String location, boolean overrideCache) throws MalformedURLException
{
return getSource(new URL(location), overrideCache);
}
public static String getSource(String location) throws MalformedURLException
{
return getSource(new URL(location), false);
}
//Actual loading of page:
private static String getSourceDirect(URL location, boolean overrideCache) throws MalformedURLException//bypasses Killbit
{
//Simulating Internet disconnection or IO exception:
//if (!KILLURL.equals(location.toString())) throw new MalformedURLException();
if (
!overrideCache &&
cache.containsKey(location) &&
!cache.get(location).isExpired()
)
{
CachedPage cp = cache.get(location);
System.out.println("Loading " + location.toString() +
"\n\tfrom cache at time " + new Date(cache.get(location).time).toString() +
"\n\t-ms until cache expired: " + ((cp.expire+cp.time) - Calendar.getInstance().getTimeInMillis()));
return cache.get(location).source;
}
try
{
System.out.println(" Loading -- ");
StringBuilder content = new StringBuilder();
// faster: String concatination involves StringBuffers anyway
// ...and StringBuilders are even faster than StringBuffers
int bytesTotal = 0;
BufferedInputStream buffer = new BufferedInputStream(location.openStream());
int lengthRead=0;
byte[] nextParcel;
do
{
nextParcel = new byte[PARCELSIZE];
/*
* Don't try to use buffer.available() instead of PARCELSIZE:
* then there's no way to tell when end of stream is reached
* without ignoring everything anyway and reading a byte.
*
* Also, if nextParcel is full (at PARCELSIZE),
* content.append(nextParcel) will add an address
* into the content (looks something like "[B&1dfc547")
* so avoid using content.append(byte[]) directly.
*/
lengthRead = buffer.read(nextParcel);
bytesTotal += lengthRead;
//if (lengthRead == PARCELSIZE)
//content.append(nextParcel); // would have been faster
//else
if (lengthRead > 0)
content.append(new String(nextParcel).substring(0, lengthRead));
// TODO: any better way to append a subset of a byte[]?
System.out.print("Bytes loaded: " + bytesTotal + "\n");
}
while (lengthRead != -1);
bytesTotal++; // replace subtracted byte due to lengthRead = -1
System.out.print(" -- DONE! Bytes read: " + bytesTotal + "; String length: " + content.length() + "\n");
String source = content.toString();
cache.put(location, new CachedPage(location, source, Calendar.getInstance().getTimeInMillis(), CACHETIME));
return source;
}
catch (IOException e)
{
e.printStackTrace();
}
return null;
}
// checks the killBit
public static boolean killBit()
{
Calendar now = Calendar.getInstance();
if (killbitCacheSaveTime + CACHETIME > now.getTimeInMillis())
return cachedKillValue;
String configs = null;
try
{
configs = getSourceDirect(new URL(KILLURL), false);
}
catch (Exception e)
{
e.printStackTrace();
cacheKillbit(true, now);
return true;
// if killbit cannot be read for any reason, killbit = true
}
String[] configArray = configs.split("\n");
for (int i = 0; i < configArray.length; i++)
{
String setting = configArray[i];
System.out.println(configArray[i]);
if (setting.compareTo("killBit = true;") == 0)
{
cacheKillbit(true, now);
return true;
}
}
cacheKillbit(false, now);
return false;
}
public static void clearKillCache()
{
killbitCacheSaveTime = 0;
}
public static void restartSession()
{
killbitCacheSaveTime = 0;
}
private static void cacheKillbit(boolean bool, Calendar time)
{
cachedKillValue = bool;
killbitCacheSaveTime = time.getTimeInMillis();
}
}
Namespace.java
/**
* @author AySz88
* @program Namespace Loader for Flcelloguy's Tool
* @version 4.00; released March 25, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.net.URL;
import java.util.AbstractMap;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.TreeMap;
import java.util.TreeSet;
//For now, the name of the "main" namespace is interpreted as "Main"
// This may be changed as necessary
//The index of the main namespace must be 0
public class Namespace
{
// data fields and functions
private String englishName, localName;
private int indexNo; // as shown in the Special:Export// page
private TreeSet<Contrib>[][][] countMatrix;
private TreeSet<Contrib> contribSet;
public Namespace(String eng, int ind, String loc)
{
englishName = eng;
indexNo = ind;
localName = loc;
// Two methods to init countMatrix:
//Type-safe way:
countMatrix =
createArray(
createArray(
createArray(new TreeSet<Contrib>(), new TreeSet<Contrib>()),
createArray(new TreeSet<Contrib>(), new TreeSet<Contrib>())
),
createArray(
createArray(new TreeSet<Contrib>(), new TreeSet<Contrib>()),
createArray(new TreeSet<Contrib>(), new TreeSet<Contrib>())
));
//Type-unsafe way:
// countMatrix = new TreeSet[2][2][2];
//
// for (int i = 0; i < countMatrix.length; i++)
// for (int j = 0; j < countMatrix[i].length; j++)
// for (int k = 0; k < countMatrix[i][j].length; k++)
// countMatrix[i][j][k] = new TreeSet<Contrib>();
contribSet = new TreeSet<Contrib>(); //sorts automagically
}
public static <T> T[] createArray(T... items)
//type-safe arrays with arrays of known fixed size
{
return items;
}
public boolean addContrib(Contrib con)
{
if (contribSet.contains(con)) return false;
int minor = 0, auto = 1, manu = 1;
if (con.minorEdit) minor = 1;
if (con.autoSummary == null) auto = 0;
if (con.editSummary == null) manu = 0;
countMatrix[minor][auto][manu].add(con);
contribSet.add(con);
return true;
}
public int getCount() {return contribSet.size();}
public int getMinorCount()
{
return // minor = 1
countMatrix[1][0][0].size() +
countMatrix[1][0][1].size() +
countMatrix[1][1][0].size() +
countMatrix[1][1][1].size();
}
public int getMajorCount()
{
return // minor = 0
countMatrix[0][0][0].size() +
countMatrix[0][0][1].size() +
countMatrix[0][1][0].size() +
countMatrix[0][1][1].size();
}
public int getSummaryCount()
{
return // auto == 1 || manual == 1
countMatrix[0][0][1].size() +
countMatrix[0][1][0].size() +
countMatrix[0][1][1].size() +
countMatrix[1][0][1].size() +
countMatrix[1][1][0].size() +
countMatrix[1][1][1].size();
}
public int getManualSummaryCount()
{
return // manual == 1
countMatrix[0][0][1].size() +
countMatrix[0][1][1].size() +
countMatrix[1][0][1].size() +
countMatrix[1][1][1].size();
}
public int getAutoSummaryCount()
{
return // auto == 1
countMatrix[0][1][0].size() +
countMatrix[0][1][1].size() +
countMatrix[1][1][0].size() +
countMatrix[1][1][1].size();
}
public double getMinorProportion() {return ((double) getMinorCount() / (double) getCount()); }
public double getSummaryProportion() {return ((double) getSummaryCount() / (double) getCount()); }
public double getManualSummaryProportion() {return ((double) getManualSummaryCount() / (double) getCount()); }
// public int newArticleCount() {return newArticleArray.size();}
// public TreeSet newArticleSet() {return newArticleArray;}
// public String[] newArticleArray() {return newArticleArray.toArray(new String[1]);}
public String getEnglishName() {return englishName;}
public String getLocalName() {return localName;}
public int getMediawikiIndex() {return indexNo;}
static void addContrib(HashMap<String, Namespace> map, Contrib con)
{
Namespace ns;
if (con.namespace.equals("")) ns = map.get(MAINSPACENAME);
else ns = map.get(con.namespace);
ns.addContrib(con);
}
// static fields and functions
public static final String head = "http://",
foot = ".wikipedia.org/w/index.php?title=Special:Export//";
public static final int ENDTAGLENGTH = "</namespace>".length(); // this'll evaluate at compile-time right?
public static final String MAINSPACENAME = "Main";
public static void main(String[] args)
{
/*TreeMap<Integer,Namespace> test =*/ getDefaultNamespaceTreeMap();
//getNamespaces("ko");
getFullMap("ko");
}
// TODO: allow any two languages to convert to one another (currently one language must be English)
public static HashMap<String, Namespace> getFullMap(String local)
{
TreeMap<Integer, Namespace> localMap = getNamespaceTreeMap(local);
Namespace[] engNamespaces = getNamespaces("en");
HashMap<String, Namespace> finishedMap = new HashMap<String, Namespace>();
for (int i = 0; i < engNamespaces.length; i++)
{
Namespace ns = localMap.remove(engNamespaces[i].indexNo);
if (ns == null) System.out.println("The " + local + " wiki does not have the equivalent of English Wikipedia namespace: " + engNamespaces[i].localName);
else
finishedMap.put(ns.localName, new Namespace(engNamespaces[i].localName, ns.indexNo, ns.localName));
}
if (localMap.size() != 0)
{
System.out.println("The " + local + " wiki has namespaces not seen in the English Wikipedia");
Iterator iter = localMap.keySet().iterator();
while (iter.hasNext())
{
Namespace ns = localMap.remove(iter.next());
finishedMap.put(ns.localName, new Namespace("Translation Unknown - " + ns.indexNo, ns.indexNo, ns.localName));
}
}
return finishedMap;
}
private static Namespace[] getNamespaces(String local)
//Does NOT populate the english names
{
try
{
String[] lineArray = GetContribs.getSource(new URL(head + local + foot)).split("\n");
int i = arrayStepPast(lineArray, "<namespaces>");
ArrayList<Namespace> nsArray = new ArrayList<Namespace>();
while (!lineArray[i].trim().equals("</namespaces>"))
{
String[] parts = lineArray[i].trim().split("\"");
int number = Integer.parseInt(parts[1]); // 2nd part
String name = "";
if (number == 0)
name = MAINSPACENAME;
else
name = parts[2].substring(1, parts[2].length()-ENDTAGLENGTH);
nsArray.add(new Namespace("", number, name));
System.out.println(number + " " + name);
i++;
}
return (Namespace[]) nsArray.toArray(new Namespace[1]); // the Namespace[1] is to convey the type of array to return
}
catch (Exception e)
{
e.printStackTrace();
return getDefaultNamespaceTreeMap().values().toArray(new Namespace[1]);
}
}
/* Currently unused
private static HashMap<String, Namespace> getNamespaceHashMap(String local)
//HashMap uses local names of namespaces as the key
//Does NOT populate the english names
{
try
{
String[] lineArray = GetContribs.getSource(new URL(head + local + foot)).split("\n");
int i = arrayStepPast(lineArray, "<namespaces>");
HashMap<String, Namespace> nsMap = new HashMap<String, Namespace>();
while (!lineArray[i].trim().equals("</namespaces>"))
{
String[] parts = lineArray[i].trim().split("\"");
int number = Integer.parseInt(parts[1]); // 2nd part
String name = "";
if (number == 0)
name = MAINSPACENAME;
else
name = parts[2].substring(1, parts[2].length()-ENDTAGLENGTH);
nsMap.put(name, new Namespace("", number, name));
System.out.println(number + " " + name);
i++;
}
return nsMap;
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}*/
private static TreeMap<Integer, Namespace> getNamespaceTreeMap(String local)
//TreeMap uses index numbers as key
//Does NOT populate the english names
//Just in case we can eventually access data using the index numbers
//Also used to match local names to english names
{
try
{
String[] lineArray = GetContribs.getSource(new URL(head + local + foot)).split("\n");
int i = arrayStepPast(lineArray, "<namespaces>");
TreeMap<Integer, Namespace> nsMap = new TreeMap<Integer, Namespace>();
while (!lineArray[i].trim().equals("</namespaces>"))
{
String[] parts = lineArray[i].trim().split("\"");
int number = Integer.parseInt(parts[1]); // 2nd part
String name = "";
if (number == 0)
name = MAINSPACENAME;
else
name = parts[2].substring(1, parts[2].length()-ENDTAGLENGTH);
nsMap.put(number, new Namespace("", number, name));
System.out.println(number + " " + name);
i++;
}
return nsMap;
}
catch (Exception e)
{
e.printStackTrace();
return getDefaultNamespaceTreeMap();
}
}
private static TreeMap<Integer, Namespace> getDefaultNamespaceTreeMap()
//TreeMap uses index numbers as key
//Does NOT populate the english names
//Just in case we can eventually access data using the index numbers
//Also used to match local names to english names
{
System.out.println("Error encountered - using default namespaces");
try
{
TreeMap<Integer, Namespace> nsMap = new TreeMap<Integer, Namespace>();
String name = "";
String[][] inArray = Contrib.NAMESPACE_ARRAY;
for (int j = 0; j < inArray.length ; j++)
{
int number = Integer.parseInt(inArray[j][1]);
if (number == 0)
name = MAINSPACENAME;
else name = inArray[j][0];
nsMap.put(number, new Namespace("", number, name));
System.out.println(number + " " + name);
}
return nsMap;
}
catch (Exception e)
{
e.printStackTrace();
return null;
}
}
public static int sumAllCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getCount();
return sum;
}
public static int sumAllMinorCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getMinorCount();
return sum;
}
public static int sumAllMajorCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getMajorCount();
return sum;
}
public static int sumAllSummaryCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getSummaryCount();
return sum;
}
public static int sumAllManualSummaryCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getManualSummaryCount();
return sum;
}
public static int sumAllAutoSummaryCounts(AbstractMap<String, Namespace> map)
{
int sum = 0;
Iterator iter = map.keySet().iterator();
while (iter.hasNext())
sum += map.get(iter.next()).getAutoSummaryCount();
return sum;
}
private static int arrayStepPast(String[] array, String o)
// was originally going to allow to use with any Comparable object
// currently only works with Strings due to use of trim()
{
int i = 0; // keep the value of i after for loop
for (; i < array.length && !array[i].trim().equals(o); i++);
return ++i; // step *past* first, then return
}
}
StatBundle.java
/**
* @author AySz88
* @program Remote source reader for Flcelloguy's Tool
* @version 4.00; released April 13, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.awt.Component;
import java.io.IOException;
import java.net.URL;
import java.util.HashMap;
import javax.swing.JOptionPane;
public class StatBundle
{
private static final String NAVIGATION_BAR_START = "<input type=\"submit\" value=",
CONTRIB_LIST_START = "<ul>",
CONTRIB_LIST_END = "</ul>",
USERNAME_BAR_START = "<div id=\"contentSub\">For"; //TODO:Internationalize
protected static final String NO_SUCH_USER = "No such user.",
NO_EDITS = "No edits from this user.";
private HashMap<String, Namespace> nsMap;
protected int total, minor, summary, manualSummary;
protected String username;
private boolean allContribsParsed;
private Component frame;
protected boolean badUsername, noEdits;
public StatBundle(String user, HashMap<String, Namespace> ns, Component inFrame)
{
username = user;
allContribsParsed = false;
nsMap = ns;
frame = inFrame;
badUsername = false;
noEdits = false;
}
public StatBundle(URL someURL, HashMap<String, Namespace> ns, Component inFrame) throws IOException
{
allContribsParsed = false;
nsMap = ns;
frame = inFrame;
badUsername = false;
noEdits = false;
parseFromConnection(GetContribs.getSource(someURL));
}
public void parseFromConnection(String source) throws IOException
{
try
{
if (allContribsParsed)
{
allContribsParsed = false;
URL nextPage = appendFromConnection(source);
while (nextPage != null)
{
JOptionPane.showConfirmDialog(frame,"5000 edits loaded. Continue?",
"Confirmation",JOptionPane.YES_NO_OPTION);
nextPage = appendFromConnection(GetContribs.getSource(nextPage));
}
}
else
{
URL nextPage = parseAllFromConnection(source);
while (nextPage != null)
{
JOptionPane.showConfirmDialog(frame,"5000 edits loaded. Continue?",
"Confirmation",JOptionPane.YES_NO_OPTION);
nextPage = parseAllFromConnection(GetContribs.getSource(nextPage));
}
}
}
catch (IOException e)
{
if (e.getMessage().equals(NO_SUCH_USER))
{
badUsername = true;
username = NO_SUCH_USER;
}
else if (e.getMessage().equals(NO_EDITS))
{
noEdits = true;
}
else throw e;
}
}
private URL parseAllFromConnection(String source) throws IOException
{
String linkString = null;
System.out.println("Computing...");
String[] array = source.split("\n");
Contrib outContrib;
int i = 1;
for (; i < array.length && ! array[i].trim().equals(CONTRIB_LIST_START); i++)
{
if (array[i].trim().startsWith(USERNAME_BAR_START))
username = PurgeContribs.getUsername(array[i]);
if (array[i].startsWith(NAVIGATION_BAR_START))
linkString = PurgeContribs.getNextDiffs(array[++i]);
}
// if (!foundURL) // bad username or url or other error
// {
// System.out.println("StatsBundle: Could not find navigation links, assume bad username");
// allContribsParsed = true;
// return null;
// }
i++; // increment past
while (i < array.length && !array[i].trim().equals(CONTRIB_LIST_END))
{
// then start reading and recording
outContrib = PurgeContribs.Parse(array[i].trim());
addContrib(outContrib);
i++;
}
updateTotals();
if (linkString == null) // finished parsing
{
allContribsParsed = true;
return null;
}
return new URL("http://en.wikipedia.org" + linkString);
}
private URL appendFromConnection(String source) throws IOException
{
String linkString = null;
System.out.println("Computing...");
String[] array = source.split("\n");
Contrib outContrib;
int i = 1;
for (; i < array.length && ! array[i].trim().equals(CONTRIB_LIST_START); i++)
{
if (array[i].trim().startsWith(USERNAME_BAR_START))
username = PurgeContribs.getUsername(array[i]);
if (array[i].startsWith("NAVIGATION_BAR_START"))
linkString = PurgeContribs.getNextDiffs(array[++i]);
}
if (linkString != null) linkString = "http://en.wikipedia.org" + linkString; // TODO:Internationalize
//complete URL here
i++; // increment past
while (i < array.length && !array[i].trim().equals(CONTRIB_LIST_END))
{
// then start reading and recording
outContrib = PurgeContribs.Parse(array[i].trim());
boolean newContrib = addContrib(outContrib);
if (!newContrib)
{
updateTotals();
allContribsParsed = true;
return null; // all new contribs parsed, exit
}
i++;
}
updateTotals();
// Prints out statistics
URL returnURL;
if (linkString!=null)
{
returnURL = new URL(linkString);
}
else
{
allContribsParsed = true;
returnURL = null;
}
return returnURL;
}
private boolean addContrib(Contrib con)
{
con.checkCorrectNamespace(nsMap);
return nsMap.get(con.namespace).addContrib(con);
}
private void updateTotals()
{
total = Namespace.sumAllCounts(nsMap);
minor = Namespace.sumAllMinorCounts(nsMap);
summary = Namespace.sumAllSummaryCounts(nsMap);
manualSummary = Namespace.sumAllManualSummaryCounts(nsMap);
}
public HashMap<String, Namespace> getNamespaceMap() {return nsMap;}
}
CachedPage.java
/**
* @author AySz88
* @program Remote source reader for Flcelloguy's Tool
* @version 4.00; released February 25, 2006
* @see [[User:Flcelloguy/Tool]]
* @docRoot http://en.wikipedia.org/wiki/User:Titoxd/Flcelloguy's_Tool
* @copyright Permission is granted to distribute freely, provided attribution is granted.
*/
import java.net.URL;
import java.util.Calendar;
public class CachedPage
{
protected URL url;
protected String source;
protected long time, expire;
public CachedPage(URL u, String s, long t)
{
url = u;
source = s;
time = t;
}
public CachedPage(URL u, String s, long t, long e)
{
url = u;
source = s;
time = t;
expire = e;
}
/*public CachedPage(URL u, String s)
{
url = u;
source = s;
time = Calendar.getInstance().getTimeInMillis();
}*/
public boolean isExpired(long now)
{
return (now > (expire+time));
}
public boolean isExpired()
{
return (Calendar.getInstance().getTimeInMillis() > (expire+time));
}
}