Wikidata crawling
Sun 26 April 2020
Graph database representation (Photo credit: Wikipedia)
I wish to have reliable data about vehicles. I decided to rely on one large source, namely Wikipedia. I chose it because it is reviewable and most of the time reviewed, and regularly updated and completed.
Wikipedia - Wikidata relationship
Wikidata items are made to designate one entity and its relationship to other entities. One entity can is directly link to one or more Wikipedia pages, one per language.
The link Wikipedia \(\to\) Wikidata is assured via a hyperlink named "Wikidata item" in english, and the link Wikidata \(\to\) Wikipedia is present into a box named "Wikipedia".
Wikipedia to Wikidata link
Wikidata to Wikipedia link
Wikipedia data
Characteristics are mainly put in tables. I decided to retrieve them using classic crawling technics with BeautifulSoup.
def table_to_dict(table):
"""
Args:
table (bs4.element.tag): html table tag
Returns:
(dict)
"""
res = {}
excluded_table = ("navbox", "navbox-inner", "navbox-subgroup")
if any(word in table.attrs.get("class", [""]) for word in excluded_table):
return res
for row in table.find_all("tr"):
prop = row.find("th")
value = row.find("td")
if prop and value:
res[prop.text.strip()] = value.text.strip().replace("\xa0", " ")
for row in table.find_all("tr"):
try:
prop, value = row.find_all("td")
except ValueError:
# not the right number of <td>
continue
prop = prop.text.strip().strip(":")
value = value.text.strip().replace("\xa0", " ")
res[prop] = value
return res
def html_to_props(content):
"""
Args:
content (str): html
Returns:
(dict)
"""
res = {}
soup = BeautifulSoup(content)
for table in soup.find_all("table"):
res.update(table_to_dict(table))
return res
def get_wikipedia_props(url):
"""Retrieve characteristic tables from wikipedia pages
Args:
url (str): url
Returns:
(dict)
"""
req = requests.get(url)
return html_to_props(req.content)
>>> get_wikipedia_props("https://fr.wikipedia.org/wiki/Hermione_(1779)")
{'Autres noms': '« La frégate de la liberté »',
'Type': 'Frégate de 12',
'Classe': 'Concorde',
'Fonction': 'Navire de guerre',
'Gréement': 'Trois-mâts carré',
'Architecte': 'Henri Chevillard',
'Chantier naval': 'Arsenal de Rochefort',
'Fabrication': 'Coque en chêne',
'Lancement': '1779 (coule en 1793)',
'Équipage': '255 à 316 marins',
'Longueur': '66 m (mât de pavillon compris)',
'Longueur de coque': '44,20 m',
'Maître-bau': '11,5 mètres (11,24 ?)',
"Tirant d'eau": '4,94 m (5,78 ?)',
"Tirant d'air": '46,9 m',
'Déplacement': '1 166 tonnes à vide',
'Hauteur de mât': '56,5 m (grand mât)35 m (artimon)',
'Voilure': '2 200 à 3 315 m2',
'Vitesse': '14,5 nœuds (27 km/h)',
'Armement': '26 canons de 12 livres 8 canons de 6 livres',
'Armateur': 'Marine française',
'Pavillon': 'Marine royale française Marine de la République'}
Those characteristics are often available in multiple languages, requiring post processing to improve uniformity.
Wikidata structure
Wikidata is a graph database linked to Wikipedia. Each entity is identified by a "Q number", an ID beginning with the "Q" letter followed by a variable number of digits. This graph is oriented with relationship being themselves objects (e.g. "member of" is the item Q379825).
Wikidata exploration
I created the following function to get a JSON representation of an item.
from collections import defaultdict
import requests
def get_item(ref):
"""get widikada item
Args:
ref (str): wikidata item reference
Returns:
(dict): json
"""
base_url = "https://query.wikidata.org/sparql?query="
query="""SELECT DISTINCT ?p ?property_label ?value
WHERE {{
wd:{} ?p ?value .
?property wikibase:directClaim ?p ;
rdfs:label ?property_label .
FILTER(LANG(?property_label)="en")
}}""".format(ref)
headers = {"Accept": "application/json"}
req = requests.get(base_url + query + "&format = JSON", headers=headers)
return req.json()
def wikidata_to_dict(item):
"""wikidata item
Args:
item (str): json output from get_item
Returns:
(dict): comprehensive dictionary
"""
result = defaultdict(list)
for prop in item["results"]["bindings"]:
label = prop["property_label"]["value"]
value = prop["value"]["value"]
if prop["value"]["type"] == "uri":
value = value.split('/')[-1]
result[label].append(value)
return result
>>> wikidata_to_dict(get_item("Q498614"))
defaultdict(list,
{'image': ['Combatlouisbourg400%20004210900%201924%2014072007.jpg'],
'instance of': ['Q11446'],
'country': ['Q142'],
'named after': ['Q658523'],
'Freebase ID': ['/m/04jf4sf'],
'inception': ['1779-01-01T00:00:00Z'],
'Encyclopædia Universalis ID': ['hermione-fregate'],
'derivative work': ['Q5502184'],
'Commons category': ['Hermione (ship, 1779)']})>
I used lists as characteristics can be mutlievaluated. This request gives numerical characteristics such as beam and draft for boats.
Retrieve item ID belonging to a specific category
Code speaks by itself:
import requests
def get_membership(ref, prop="P31"):
base_url = "https://query.wikidata.org/sparql?query="
query = """SELECT ?item
WHERE
{{
?item wdt:{} wd:{}.
}}
""".format(
prop, ref
)
headers = {"Accept": "application/json"}
req = requests.get(base_url + query + "&format = JSON", headers=headers)
res = req.json()
for member in res["results"]["bindings"]:
url = member["item"]["value"]
yield url.split("/")[-1]
I decided to retrieve id from URL.
Relationship Wikidata - Wikipedia
Wikidata to Wikipedia
To get Wikipedia page URLs, I coded the following function using BeautifulSoup.
from bs4 import BeautifulSoup
from bs4.element import Tag
def wikidatapage_to_wikipedialinks(item):
"""
Args:
item (str): ref
Yields:
(str): url
"""
req = requests.get("https://www.wikidata.org/wiki/{}".format(item))
soup = BeautifulSoup(req.content)
for tag in soup.find_all("div"):
if "wikibase-sitelinklistview" not in tag.attrs.get("class", ""):
continue
for href in tag.find_all("a"):
url = href.attrs["href"]
if "wikipedia.org" in url:
yield url
Wikipedia to Wikidata
Here again, I used BeatifulSoup.
def wikipedia_to_wikidata(url):
"""Retrieve wikidata item number
Args:
url (string): wikipedia url
Returns:
(string): reference
"""
req = requests.get(url)
soup = BeautifulSoup(req.content)
wikilink = next(
li
for li in soup.find_all("li")
if "id" in li.attrs and li.attrs["id"] == "t-wikibase"
)
tag = wikilink.find("a")
try:
return tag.attrs["href"].split("/")[-1]
except IndexError:
return ""
>>> wikipedia_to_wikidata("https://en.wikipedia.org/wiki/French_frigate_Hermione_(2014)")
'Q5502184'
Put it all together
From a Q item ID, if we try to get all properties, we end with:
def get_all_props(ref):
"""
Args:
ref (str): Q number
Returns:
(dict): all properties
"""
res = wikidata_to_dict(get_item(ref))
wikipedia_urls = wikidatapage_to_wikipedialinks(ref)
for url in wikipedia_urls:
res.update(get_wikipedia_props(url))
return res
Be careful, some properties are lists (those coming from Wikidata), other are strings (those coming from Wikipedia).
Mass crawling
Let's start with an iterable of Q items IDs. I'll put them into a set to
assure unicity but any iterable is OK. I won't go in details into the retrieving
of those IDs, but you may imagine the function get_membership() defined
earlier is useful.
qitems = {
"Q877343",
"Q3079847",
"Q3445980",
"Q3017039",
"Q16222749",
"Q3153932",
}
Wikidata limits the query rate, and is the list of items is long monitoring is desirable.
from datetime import datetime
from time import sleep
all_results = {}
for i, ref in enumerate(qitems):
if ref in all_results:
# speedup restarts if all_results is partially filled
continue
try_nb = 0
while try_nb < 10 and ref not in all_results:
try:
all_results[ref] = get_all_props(ref)
except: # excpetion should be specified
now = datetime.now().isoformat()
print("{} WARNING item {} (try {}/10)".format(now, ref, try_nb))
try_nb += 1
sleep(60 * try_nb)
if i % 100 == 0:
now = datetime.now().isoformat()
print("{} STATUS: item {} / {}".format(now, i, len(boat_refs)))
The result is a python dict that can easily be serialized into a JSON.