How to access structured data - Revision history

GavinRobinson: Text replacement - ", UK" to ", British Army"

2017-11-11T20:23:54Z

Text replacement - ", UK" to ", British Army"

GavinRobinson: /* Limitations of the data */

2016-08-15T19:22:16Z

‎Limitations of the data

GavinRobinson: /* Limitations of the data */ added some division-corps-army relationships

2016-05-24T18:22:59Z

‎Limitations of the data: added some division-corps-army relationships

GavinRobinson at 19:28, 18 March 2016

2016-03-18T19:28:50Z

GavinRobinson: Created page with "''This page is a work in progress. Please help to improve it if you can, especially if you're familiar with the MediaWiki API.'' This wiki aims to create pages for military u..."

2016-03-18T19:05:23Z

Created page with "''This page is a work in progress. Please help to improve it if you can, especially if you're familiar with the MediaWiki API.'' This wiki aims to create pages for military u..."

New page

''This page is a work in progress. Please help to improve it if you can, especially if you're familiar with the MediaWiki API.''

This wiki aims to create pages for military units in the First World War, with each unit's page containing structured data in infobox templates. You are free to re-use this data for any purpose according to our Creative Commons Attribution Share Alike licence. This page gives some tips on how to access the data.

== Data structures ==

Data about military units is structured using these templates:

* {{tl|infobox military unit}} (one per page)
* {{tl|infobox command structure}} (repeatable)
* {{tl|infobox theatre of war}} (repeatable)

See the documentation of each template for more information about parameters and values. The templates are adapted from Wikipedia but are not quite the same.

Unit pages also have some semi-structured data outside the infoboxes. They mostly have the same headings for personal narratives, media and other sources. References to official war diaries are under a standard heading controlled by {{tl|War diary heading}}. There are several templates for information about war diaries, listed in [[:Category:War diary templates]].

British administrative units often have references to campaign medal rolls in a wikitable with a standard structure. For example, see [[Lincolnshire Regiment, UK#Medal rolls]].

== MediaWiki action API ==

This site runs on MediaWiki. The action API is enabled for read queries, and you don't need to be logged in to use it. The API allows you to access wiki pages using the programming language of your choice.

The action API is fully documented in the [https://www.mediawiki.org/wiki/API:Main_page MediaWiki manual].

The endpoint for this wiki is http://collaborativecollections.org/WorldWarOne/api.php

== Special:Export ==

You can export pages as XML files using [[Special:Export]], which is documented in the [https://www.mediawiki.org/wiki/Help:Export Mediawiki manual]. You don't have to be logged in to use this method. If you want to download large numbers of pages, it's quicker and easier than using API calls.

You can download all the pages in a category by putting the category name in the 'Add pages from category' box and then clicking 'Add'. You can add another category after this in the same way.

The quickest way to get every unit's page is to use [[:Category:All units]]. If you only want a subset of units, you can use a more specific category. You can see what unit categories are available by looking at [[:Category:Units]] and manually clicking through its sub-categories.

If you check the 'Save as file' box on the [[Special:Export]] page and then click the 'Export' button, you will get an XML file containing the source text of each page wrapped in simple XML tags.

These are the most important tags:

* <nowiki><page></nowiki> contains all downloaded revisions of a page, and metadata about the page.
** <nowiki><title></nowiki> is the page name.
** <nowiki><revision></nowiki> is one revision of the page, including source text and metadata. If you unchecked the box 'Include only the current revision, not the full history' on the [[Special:Export]] page, there might be more than one revision per page. If you leave the box checked, you only get the latest revision of each page but it will still be wrapped in a <nowiki><revision></nowiki> tag.
*** <nowiki><timestamp></nowiki> is the timestamp of the revision.
*** <nowiki><contributor></nowiki> contains details of the user who made the revision.
*** <nowiki><text></nowiki> contains the full source text of the page, as it would appear in the edit box if you were editing a page manually. These is no XML markup within the page text, so you can't access template parameters just by parsing the XML. Angle brackets inside the page text are escaped as &lt; and &gt;

You can use the programming language of your choice to either parse the XML or just split the file into substrings.

== Scraping HTML ==

You can scrape data straight out of HTML pages instead of accessing the source wikitext. The infoboxes are marked up as tables. The first row spans two columns and contains:

* {{tl|infobox military unit}}: the page name inside <nowiki><b></nowiki> tags.
* {{tl|infobox command structure}}: the text 'Command structure'.
* {{tl|infobox theatre of war}}: the text 'Theatre of war'.

Each subsequent row has two columns.

* the first column is a <nowiki><th></nowiki> tag containing the parameter label (this does not always match the parameter's actual name in the template source code).
* the second column is a <nowiki><td></nowiki> tag containing the parameter value. For parameters that are turned into wikilinks, the value will be inside <nowiki><a></nowiki> tags. Some values can contain <nowiki><ref></nowiki> tags, which will be rendered in HTML as <nowiki><sup></nowiki> tags.

== Limitations of the data ==

This site is a work in progress. See [[Populating lists of military units]] for more details of coverage and progress. We currently have pages for over 5,000 units. These are mostly [[infantry]] and [[mounted]] units from the British Empire and the American Expeditionary Force.

The data on most existing pages is still quite rough. Many tactical relationships are probably missing, and dates for relationships and theatres are often only accurate to the nearest month (typically the start date will be given as the first of the month and end date as the last of the month regardless of the actual date).

Tactical relationships between divisions, corps and armies are nearly all missing.

British administrative relationsips are mostly complete for the units that have pages but dates are likely to be missing. Most pages don't have a complete history of the unit's full names.

[[Category:Help]]

@@ Line 15: / Line 15: @@
 Unit pages also have some semi-structured data outside the infoboxes. They mostly have the same headings for personal narratives, media and other sources. References to official war diaries are under a standard heading controlled by {{tl|War diary heading}}. There are several templates for information about war diaries, listed in [[:Category:War diary templates]].
-British administrative units often have references to campaign medal rolls in a wikitable with a standard structure. For example, see [[Lincolnshire Regiment, UK#Medal rolls]].
+British administrative units often have references to campaign medal rolls in a wikitable with a standard structure. For example, see [[Lincolnshire Regiment, British Army#Medal rolls]].
 == MediaWiki action API ==

@@ Line 62: / Line 62: @@
 == Limitations of the data ==
-This site is a work in progress. See [[Populating lists of military units]] for more details of coverage and progress. We currently have pages for over 5,000 units. These are mostly [[infantry]] and [[mounted]] units from the British Empire and the American Expeditionary Force.
+This site is a work in progress. See [[Populating lists of military units]] for more details of coverage and progress. We currently have pages for over 7,000 units. These are mostly [[infantry]] and [[mounted]] units from the British Empire and the American Expeditionary Force.
 The data on most existing pages is still quite rough. Many tactical relationships are probably missing, and dates for relationships and theatres are often only accurate to the nearest month (typically the start date will be given as the first of the month and end date as the last of the month regardless of the actual date).
-Some tactical relationships between divisions, corps and armies have now been added but the dates are nearly all missing, and some relationships are probably still missing because we don't have a complete list.
+American division-corps-army relationships should be more or less complete, with exact dates.
 British administrative relationsips are mostly complete for the units that have pages but dates are likely to be missing. Most pages don't have a complete history of the unit's full names.
 [[Category:Help]]

@@ Line 31: / Line 31: @@
 You can download all the pages in a category by putting the category name in the 'Add pages from category' box and then clicking 'Add'. You can add another category after this in the same way.
-The quickest way to get every unit's page is to use [[:Category:All units]]. If you only want a subset of units, you can use a more specific category. You can see what unit categories are available by looking at [[:Category:Units]] and manually clicking through its sub-categories.
+The quickest way to get every unit's page is to use [[:Category:All units]]. If you only want a subset of units, you can use a more specific category. You can see what unit categories are available by looking at [[:Category:Units]] or [[:Category:Countries]] and manually clicking through their sub-categories.
 If you check the 'Save as file' box on the [[Special:Export]] page and then click the 'Export' button, you will get an XML file containing the source text of each page wrapped in simple XML tags.