Module:Sandbox/Innesw/PopulationFromWikidata-upgrade/doc

This page contains

• descriptions of the proposed changes to the module, implemented in this new version
• the proposed documentation for the module once the changes are implemented.

Changes and fixes to this module have been made (in part) in association with a proposed upgrade of the Infobox template. Therefore some changes are only relevent if particular parts of that upgrade are implemented.

Examples and testcases for the changes documented here are at Infobox template upgrade population testcases.

Changes to the module.
The existing, released version of the module is referred to here as the 'current' version, and the proposed version after these changes are implemented as the 'new' version.

Changed Behaviour - multiple population values

The new version makes a significant change to the behaviour of the module regarding how it displays population figures if they are available for a number of different geographies, ie: if they are available in the linked Wikidata item for more than one of Urban Centres and Localities (UCL), Suburbs and Localities (SAL) and Indigenous Communities (ILOC).

Current Behaviour

Having gathered a list of all Wikidata population claims that meet certain criteria, then reduced that list to the latest claim for each geography, the logic in the current version is:

• for each |type= value, there is a single preferred geography. If a claim for that geography is available, return only that claim.
• if there are no claims for the prefered geography:
  • if |type=town, return latest claims for SAL and / or ILOC if available
  • for all other |type= values, return latest claims for all available geographies

This means that, if a claim for the preferred geography exists, no claims for any other geography are ever returned.

Discussion

The suggestion to show multiple population figures has been made in a number of forums (eg: at the module talk page and at the infobox talk page). On the other hand there has started to be use of the module outside the Infobox, for tables of suburbs in LGAs (see City_of_Rockingham, Town_of_Katherine), where showing the population for the wrong geography, or for multiple geographies, would seem to be undesirable.

There are a number of possible solutions for making available options to show populations for either (a) multiple geographies if they are available, or (b) a single specific geography:

1. make the ListForInfobox function by default return a single population, but return all available geographies on specific request (|multi=yes)
2. make the ListForInfobox function by default return all available geographies, but return a single population for a specified geography on request (eg: |geog=ucl)
3. make different functions available, ie: ListForInfobox to return all available geographies, and LatestPopulation(geog=ucl) to return a specified geography

Option 3 has been implemented in the new version of the module. Function ListForInfobox will show all available geographies (unless explicitly requested not to with the |geog= parameter. New function LatestPopulation - intended for use outside the infobox - requires a geography to be specified, and will return the single latest population for that geography.

New Behaviour

Having gathered a list of all Wikidata population claims that meet certain criteria, then reduced that list to the latest claim for each geography, the logic in function ListForInfobox is, in the new version:

• if a geography has been specified in |geog=, and a latest claim for that geography exists, return that claim. (There is no reference here to the |type= value.)
  • if there is no claim for the specified geography, return blank
• for each |type= value, a list of possible geographies is specified
• latest claims that exist for any of these geographies are returned
• if there are no such claims, return blank

Other Fixes and Enhancements

Changes made in this upgrade to the module

Issue	Suggested at	Comments
integrated citation references	next steps module talk 1, also module talk 2	When a population is retrieved for article text, as well as in the infobox, the current version produces separate references to the same source. In the new version, links for the same claim in Wikidata now combine into the same citation reference. The existing version constructed a correct name for the references, but did not pass that name correctly to the citation reference process.
fix for merged claims in wikidata	Template_talk:Infobox_Australian_place/Archive_9#Strange_census_behaviour_on_Gootchie,_Queensland, also Template_talk:Infobox_Australian_place#Referring_to_population_in_the_body_text	When two census uploads for the same place (but different dates) had the same value (population number), the two claims were merged in wikidata, and incorrect references could be returned. The module now recognises a claim with two point-in-time qualifiers, and splits it (internally) into two normal claims with valid references.
latest population		New function, called by {{#invoke:PopulationFromWikidata|LatestPopulation}}, returns the latest population figure for the requested geography. Always returns the population figure with the reference, but with options to also show the year, and to put supplied text (usually a full stop) between the other returned text and the reference. Intended for use in the text of articles.
historic populations	next steps	New function, called by {{#invoke:PopulationFromWikidata|HistoricPopulations}}, returns a table of population figures for all years where census data is in wikidata. By default it has columns for each of the specified geographies (for the |type=) that have any population data. |geog= can be specified, to return only population figures for that geography.
no output for some |type= values	next steps	If |type=cadastral or |type=protected, the new version explicitly returns blank output. It also returns blank if |type= is not recognised.
links to explanations of ABS geographic units	next steps	The geographic unit abbreviation is now a link to the Australian Bureau of Statistics article, and a specific anchor for the explanation of the unit.
census link linkrot		For census data for 2016 and earlier, data uploaded to Wikidata has reference URLs that are no longer valid. The module now uses eg: {{Census_2016_AUS}} to produce a citation reference with the correct URL. For claims in Wikidata that are not from census data, the existing reference URL is used as-is.
new |type= values		New |type= values have been suggested for the infobox, and implemented in the proposed upgrade of it. The new version of PopulationFromWikidata recognises these values as valid, and allocates ABS geographies to them: |type=settlement - SAL and ILOC |type=locality - SAL and ILOC |type=townandlocality - UCL, ILOC and SAL

Unresolved Issues

Issue	Suggested at	Comments
calculate population densities	next steps	Suggestion was to automatically calculate densities from Wikidata information. To do this each area claim in Wikidata will need an 'applies to part' property that matches the population claim, and these area claims will need to be kept up-to-date after each census.
indigenous communty populations	next steps	The initial suggestion was to get ABS ILOC IDs into Wikidata items, as the first step to uploading ILOC census data. The new version of this module will accept |geog=iloc, if that is the preferred geography for populations.
|type=region	next steps	For |type=region, the current version - and the new version - return census data for the LGA. The suggestion is SA3 populations may be better. There are 167 articles with |type=region in the infobox. It has not yet been investigated what populations currently appear in the infobox for these articles.
ABS data issue	next steps	Possible differences occur between ABS datapacks (the source for data uploaded to Wikidata) and Quickstats (which reference URLs link to), especially for places with very low populations. This means a link to Quickstats may not support the stated population value. This seems to be an issue that needs to be taken up with the ABS.
Create template?		Is creating a template that calls this module worth doing? proposed template name: LatestPopulationFromWikidata it would be intended for use outside the infobox, like in the lede of an article: 'Place is a town in Victoria. It has a population of {{LatestPopulationFromWikidata|type=town|geog=ucl}}.' it would produce the single latest population figure for the requested geography, optionally with year, and always a reference. the parameter |geog= would be required, and |wikidata= optional

Below is the proposed documentation for the module once the above changes are implemented.

Description

This module returns population figures for Australian places, getting its data from the Wikidata item linked to the place article. Its most common use is in {{Infobox Australian place}}, where it will return the latest populations for all Australian Bureau of Statistics (ABS) geographic areas relevant to the place.

It can also return a single latest population figure for a specified geography, or a table of all the historic populations found in the Wikidata.

All returned population figures are referenced - if a population claim in Wikidata does not have a reference, it will not be returned.

The aim of the module is to make it easier to keep population values (and associated references) up-to-date in Australian place article Infoboxes, and in other locations in those articles.

Usage

The module exposes three functions.

ListForInfobox( type, wikidata, geog )

{{#invoke:PopulationFromWikidata |ListForInfobox |type=t |wikidata=w |geog=g }}

Returns the lastest populations for all relevant ABS geographies that are available. If there is more than one, they are returned as bulleted items.

Parameters:

• |type= the type parameter from {{Infobox Australian place}}. Required.
• |wikidata= Wikidata ID to override that of the current article. Optional.
• |geog= Geography abbreviation for a single geography whose population is to be returned. Valid values are ucl, sal, iloc, lga. Optional.

LatestPopulation( geog, wikidata, year, punc )

{{#invoke:PopulationFromWikidata |LatestPopulation |geog=g |wikidata=w |year=y |punc=p }}

Returns the lastest census population for the the requested ABS geography.

Parameters:

• |geog= Geography abbreviation for a single geography whose population is to be returned. Valid values are ucl, sal, iloc, lga. Required.
• |wikidata= Wikidata ID to override that of the current article. Optional.
• |year= Any value except <blank> or 'no' will cause the output to include the year of the Wikidata population claim. If |year= is not specified, the year is not shown. Optional.
• |punc= Any value will be placed between the rest of the returned text and the reference number. Usually this will be |punc=.. Optional.

HistoricPopulations( type, wikidata, geog )

{{#invoke:PopulationFromWikidata |HistoricPopulations|type=t |wikidata=w |geog=g }}

Returns a table of census populations by year and ABS geography.

Parameters:

• |type= the type parameter from {{Infobox Australian place}}. Required.
• |wikidata= Wikidata ID to override that of the current article. Optional.
• |geog= Geography abbreviation for a single geography whose populations are to be returned. Valid values are ucl, sal, iloc, lga. Optional.

Categories

The module adds all articles that use it - including via Infobox Australian place - to Category:Australian_place_articles_using_Wikidata_population_values.

What the module does

Population sources

The module is designed to be invoked from the Infobox Australian place template and gathers data from the Wikidata item linked to each article.

This module is only invoked by Infobox Australian place if a population is not manually supplied in its |pop= argument.

Assumptions

The module works with the following assumptions:

• That all Australian place Wikipedia articles are linked to relevant Wikidata items (true because Canley and 99of9 have done this work).
• Relying on the type field of the Infobox Australian place template being a required field and assuming it always has a value specified.
• We're only considering population values associated with the Australian Bureau of Statistics' defined Australian Statistical Geographic Standard areas.
• Assuming that the linked Wikidata item will likely have population statements for multiple Australian Bureau of Statistics geographic areas that encompass the item place.
• We’re ignoring any ranking of population statements.

Population selection

The high level steps of the module work flow are decribed below. There are three major steps in the process of selecting the population figure to return.

Step 1. Check which population claims have enough information to be considered

As a minimum they are required to have:

1. A point in time qualifier date (this helps to choose the most recent population figures).
2. An applies to part qualifier value (this states which ABS geography type the population is for and helps choose the most appropriate geographic area for the place article).
3. A determination method qualifier item (this specifies if it is a census population figure or a non-census population estimate and helps define the reference components).
4. Some reference information (it is a requirement to have something with which to build a reference but more than the minimum is recommended - see the Population data in Wikidata section).

After filtering for these requirements a subset of population claims is carried forward.

Step 2. Check which population claims have the most recent figures for each geography

Check within the list of claims and find the most recent population figure for each applies to part value, ie: for each applies to part value (UCL, SAL etc) we find the claim with the latest point in time value (2016, 2011 etc).

Step 3. Specify the geography types that can be returned for each place type

If a geography is specified in the |geog= parameter, only the latest population for that geography will be returned.

If the |geog= parameter is not specified, the possible geographies for the provided |type= are:

City	Urban Centres and Localities (UCL)
Suburb	Suburbs and Localities (SAL)
Town	Urban Centres and Localities (UCL), Indigenous Locations (ILOC), Suburbs and Localities (SAL)
Settlement	Suburbs and Localities (SAL), Indigenous Locations (ILOC)
Locality	Suburbs and Localities (SAL), Indigenous Locations (ILOC)
TownAndLocality	Urban Centres and Localities (UCL), Indigenous Locations (ILOC), Suburbs and Localities (SAL)
LGA	Local Government Areas (LGA)
Region	Local Government Areas (LGA) (for now)

Compile the module output, using only latest claims in specified geographies

For all possible geographies, the latest claims found will be returned.

If there is only one matching claim, it is simply output:

100 (UCL 2021)^[1]

If there is more than one claim, they are output as a bulleted list:

• 90 (ILOC 2021)^[1]
• 100 (SAL 2021)^[2]

References

The references for actual census data are formatted using the census template for the appropriate year, eg: Census 2021 AUS. All other references use the Cite web template.

The census population figure references take this form: Australian Bureau of Statistics (28 June 2022). "Cosmo Newberry (Indigenous Location)". 2021 Census QuickStats. Retrieved 28 June 2022.

The non-census derived population figure references take this form: Australian Bureau of Statistics (29 March 2022). "Population estimates by SA2 and above, 2001 to 2021 (Greater Capital City Statistical Areas)". Australian Regional Population. Retrieved 28 June 2022.

The references are named: name = refwork .. "_" .. year .. "_" .. appliespart .. "_" .. reftitle. This is long-winded because we are aiming for a unique reference name for each population claim.

Example outputs

There are some example outputs in the Infobox Australian place Sandbox Test Cases page here.

What it doesn't do - next steps

There are some issues that we are aware of, and have considered but haven't dealt with yet. These will be tackled in time in collaboration with other place article contributors. (No doubt there are many more to add to the list - please do).

• Some tidying up the output within the infobox:
  • removing unnecessary bullet points when there's only one item
  • add links to information about the relevant Census
  • adding tooltip description
  • change geography to abbreviation
  • adding links to explanations of ABS geographic boundaries (add this info to the Census articles and link to sections there)
    • Done. The links are to an ASGS section in Australian Bureau of Statistics.
• Make a table of historic population values (from those available in Wikidata and that meet other module criteria) and test this as a new addition to place articles. As part of possible methods of preserving historic population figures in articles. Possibly a better solution than having multiple old values listed in Infoboxes (eg. Basket_Range,_South_Australia) or having to maintain individually in-text? Not the same idea, but there is a table of historic population values listed in this article).
  • Done. Use {{#invoke:PopulationFromWikidata|HistoricPopulations}}.
• Figure out the case of two Infoboxes: Jimbour East, Queensland
• Suppress the population figures for protected areas. (eg, no output for type = protected). Yes?
  • Done. Populations for |type=protected, and |type=cadastral are now explicitly suppressed.
• Population density figures need to be computed and added to the Infobox using the same population (and area from corresponding geography) as this module outputs now. With area data uploaded to Wikidata?
• Test that city rank can still be displayed in the Infobox if population coming from the module.
  • Done, but not in this module - done in {{Infobox Australian Place}}
• How to integrate (merge correctly) named references from the module with those used in-text. And how to retain historic population values (and references) as the Infobox population automatically updates with the most current figures. The module produces named references that are unique to the population value, but there are currently reference merging bugs associated with references from templates (and modules).
  • Done.
• Should we change it so that pop2 still displays even if pop is replaced by the module population? So you can have both the automated population and a specific other population that's relevant to the article for some reason.
  • Done, but not in this module - done in {{Infobox Australian Place}}
• Add some more documentation to Wikipedia:WikiProject Australian places/Population data.
• Figure out interactions with the Coord template that's used in the majority of Australian place articles. The Coord template takes a population argument and uses that to determine the display scale of the Coordinates interactive map. Should we make an equivalent module (similar to this one) to bring place coordinates (with appropriate map zoom scales) from Wikidata to the Infobox Australian place template? Then the coordinates (and map scale) can be kept up-to-date with Wikidata. This would require parallel work to determine most appropriate place coordinate definition (eg centroid? of which geographic area?) so coordinates can be bulk imported to Wikidata? Or just rely on people adding the coordinates values to Wikidata manually but cut out the need to use the Coord template to set map zoom scale? Or keep using the Coord template but give it the appropriate population value as selected by (a modified version) of this PopulationFromWikidata module.
• Connect Aboriginal and Torres Strait Islander community Wikidata items with ABS ILOC IDs so ILOC population counts can be uploaded in bulk. Will then need to revisit the ILOC vs SAL preferencing because for some towns ILOC will be more appropriate than SAL (due to geographic area covered).
• Need to revisit mapping of regions to ABS geographies. Maybe they should be mapped to SA3s? Eg: Kimberley region article Also, need to update this article and other equivalents.
• Discrepancy with places with zero population, such as Essendon Fields (Q5399482):
  • In QuickStats it says: "No information can be provided because the area selected had no people or a very low population in the 2021 Census."
  • But it does have data in Wikidata (population 13, for SAL20886, which is what is in the DataPack).
  • This means that the reference URL ends up not backing up the displayed population figure.
  • There may be a difference in how this is handled between 2016 and 2021. For example, Savannah (Q55771929) has zero population and doesn't show in either 2016 or 2021 QuickStats — but Tyenna (Q7859816) had 3 people in 2016 (shown in QuickStats) and 4 people in 2021 (not shown in QuickStats). Both places have both figures in the DataPack.

What if the outputs are incorrect

All the references produced by this module are followed by an Edit at Wikidata pencil icon which links to the relevant Wikidata item (and specific population claim). This is where people should go to fix any errors in the population figure outputs or references. See next section for lists of what should ideally be included in a Wikidata population claim.

Population data in Wikidata

Wikipedia - Wikidata links

In parallel to development of this module User:99of9 and User:Canley have been working on ensuring all Australian place Wikipedia articles are linked to corresponding Wikidata items (describing that same place). This has largely been done. This enables the use of this module.

Census data

Population data has historically been manually entered to individual Wikidata items. Recently (since ~2017) User:99of9, User:Canley and others have used QuickStatements to do bulk imports of population data from Australian Bureau of Statistics datasets. Part of developing this module was to refine the list of metadata (qualifiers and reference fields) that should be imported alongside the population values.

As at July 2022 the first release of the 2021 census population data has been uploaded for the geographic areas relevant to Australian place Infoboxes. This includes data for Suburbs and Localities (SAL), Indigenous Locations (ILOC) and Local Government Areas (LGA). The Urban Centres and Localities (UCL) data is due to be released in October 2022.

The module requires these qualifiers and reference components to have values in the Wikidata population claim.

• applies to part
• point in time
• determination method
• reference: reference URL
• reference: title
• reference: published in
• reference: retrieved
• reference: Australian Statistical Geography 2021 ID (optional)

An example of a Wikidata item with a correctly filled 2021 population claim (using Census data) is:d:Q2821571#P1082.

Non-census data

Bulk uploads have been done for census data. They have not been done for between-census estimated residential population (ERP) or Data by Region figures, for example. These estimates are useful for capital cities, LGAs and regions.

The module requires that non-census population claims have these components:

• applies to part
• point in time
• determination method
• reference: reference URL
• reference: title
• reference: published in
• reference: retrieved
• reference: publication date
• reference: Australian Statistical Geography 2021 ID (optional)

An example of a Wikidata item with a correctly filled 2021 estimated resident population claim (not the other population claims) is:d:Q11568#P1082. An example of a Wikidata item with a correctly filled 2020 LGA Data by Region population claim (not the other population claims) is:d:Q704257#P1082.

Who made the module

Wikimedia Australia initially designed this project to coincide with the first release of the 2021 census data (in June 2022). This module was created as part of a funded project with work done by m:User:MaiaCWilliams in collaboration with (really...HUGE amounts of help from) User:Samwilson, User:99of9 and User:Canley. The project was coordinated by User:tenniscourtisland.

It is an ongoing project and we will continue to refine the module. Of course anyone is welcome to contribute!

Head to the Module_talk:PopulationFromWikidata page if you have anything to discuss.

We wrote a summary of the project for the Wikimedia Australia blog here.