Wikibase support by philippesaade-wmde · Pull Request #4 · wmde/WikidataTextifier

philippesaade-wmde · 2026-05-12T11:49:35Z

Including support for textifying Wikibase entities:

Added an API parameter for specifying the Wikibase Action API URL.
Removed TTL retrieval and normalization because it is not standardized across Wikibase instances.
Updated the label cache database to include the Wikibase Action API URL as a primary key, and added a migration function.

exowanderer

I added a lot of questions and suggestions. Nothing looks problematic so far, but I have not completed my review -- only 4 out of 13 files

exowanderer · 2026-05-22T12:25:08Z

-            "text": self.text,
-            "lang": self.lang
-        }
+        return {"text": self.text, "lang": self.lang}


Code reviewability and maintainability standards prefer the previous version:

return { "text": self.text, "lang": self.lang }

This was changed by Ruff Lint's formatting

exowanderer · 2026-05-22T12:26:30Z

-            self.latitude is not None
-            and self.longitude is not None
-        )
+        return self.latitude is not None and self.longitude is not None


This could go either way, but I prefer the maintainabilitiy of the previous version

return ( self.latitude is not None and self.longitude is not None )

This was changed by Ruff Lint's formatting

exowanderer · 2026-05-22T12:30:43Z

-            and self.label is not None
-            and str(self.label) != ""
-        )
+        return bool(self.id) and self.label is not None and str(self.label) != ""


I think we should use a temp variable here for clarity

id_and_label_check = bool(self.id) and self.label is not None and str(self.label) return id_and_label_check != ""

exowanderer · 2026-05-22T12:32:11Z

-            bool(self.property)
-            and any(bool(v) for v in self.values)
-        )
+        return bool(self.property) and any(bool(v) for v in self.values)


For maintainability and readability, I prefer the previous version

return ( bool(self.property) and any(bool(v) for v in self.values) )

This was changed by Ruff Lint's formatting

exowanderer · 2026-05-22T12:35:59Z

@@ -302,8 +289,8 @@ class WikidataClaimValue:
    value: Optional[


This is a little difficult to read. Maybe this could be better:

Optional[ Union[ WikidataEntity, WikidataQuantity, WikidataTime, WikidataCoordinates, WikidataText, WikidataMonolingualText ] ] = None

This was also changed by Ruff Lint's formatting

exowanderer · 2026-05-22T14:10:01Z

@@ -36,10 +37,11 @@


 class WikidataLabel(Base):


Should this be generalised to WikibaseLabel(Base)?

I made this change and renamed everything with "Wikidata" in it to "Wikibase".
I additionally added aliases to keep support for Wikidata classes and files, otherwise external repositories importing Wikidata would break (such as the vector database pipeline)

exowanderer · 2026-05-22T14:11:17Z

        """Create tables if they do not already exist."""
        try:
            Base.metadata.create_all(engine)
+            WikidataLabel._migrate_labels_table_for_wikibase()


If we want to generalise the class name, then we would need to change this to WikibaseLabel as well

I made this change and renamed everything with "Wikidata" in it to "Wikibase".
I additionally added aliases to keep support for Wikidata classes and files, otherwise external repositories importing Wikidata would break (such as the vector database pipeline)

exowanderer · 2026-05-22T14:23:33Z


    @staticmethod
-    def add_bulk_labels(data):
+    def _migrate_labels_table_for_wikibase():


Is this completely new?

If so, what problem does it solve or feature does it enable?

The database structure has been changed to include the wikibase action api as a new column and primary key. This function runs at start and migrates the old database to the new one by filling this column with Wikidata's action API.

exowanderer · 2026-05-22T14:39:58Z

+                    """
+                    SELECT COUNT(*)
+                    FROM information_schema.COLUMNS
+                    WHERE TABLE_SCHEMA = :schema_name


Can this be an F-String?

f"""
SELECT COUNT(*)
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = {DB_NAME} # <---- HERE
AND TABLE_NAME = 'labels'
AND COLUMN_NAME = 'wikibase_url'
"""

Bound parameters are safer than f-string because they handle SQL injection and syntax breakage.

exowanderer · 2026-05-22T14:43:11Z

+                        SELECT COLUMN_NAME
+                        FROM information_schema.KEY_COLUMN_USAGE
+                        WHERE TABLE_SCHEMA = :schema_name
+                          AND TABLE_NAME = 'labels'


f"""
SELECT COLUMN_NAME
FROM information_schema.KEY_COLUMN_USAGE
WHERE TABLE_SCHEMA = {DB_NAME} <--- HERE
AND TABLE_NAME = 'labels'
AND CONSTRAINT_NAME = 'PRIMARY'
ORDER BY ORDINAL_POSITION
"""

Bound parameters are safer than f-string because they handle SQL injection and syntax breakage.

exowanderer

Completed review of another 5 files -- the smaller ones

exowanderer · 2026-05-22T14:50:17Z

-    "-k", "uvicorn.workers.UvicornWorker", \
-    "-w", "1", \
-    "-b", "0.0.0.0:5000"]
+CMD ["uv", "run", "gunicorn", "main:app", "-k", "uvicorn.workers.UvicornWorker", "-w", "2", "--bind", "0.0.0.0:5000", "--timeout", "30", "--graceful-timeout", "15", "--max-requests", "1000", "--max-requests-jitter", "100", "--keep-alive", "5", "--access-logfile", "-", "--error-logfile", "-"]


nit: Please add a trailing space after the las line

exowanderer · 2026-05-22T14:53:00Z


+## Future Plan
+
+- Replace Action API with GraphQL once Wikibase GraphQL is available for Wikibases:


We may never "replace" the ActionAPI, but we will supplement some functionality with GraphQL in addition.

…with additional aliases for Wikidata

exowanderer · 2026-05-23T14:08:08Z

+                    text(
+                        """
+                        SELECT COLUMN_NAME
+                        FROM information_schema.KEY_COLUMN_USAGE


Is the DB schema stored in this repo as well?

The DB schema can be found under the WikidataLabel class

exowanderer · 2026-05-23T14:09:45Z

+            ]
+
+            if primary_key_cols == ["id"]:
+                connection.execute(


I assume this only happens when a new DB is first instantiated.

I think we should log that on the server and possibly flag it during deployment

exowanderer · 2026-05-23T14:16:02Z

        missing_ids = set(ids) - set(labels.keys())
        if missing_ids:
-            missing_labels = WikidataLabel._get_labels_wdapi(missing_ids)
+            missing_labels = WikidataLabel._get_labels_wdapi(missing_ids, wb_url=normalized_wb_url)


Suggested change

missing_labels = WikidataLabel._get_labels_wdapi(missing_ids, wb_url=normalized_wb_url)

missing_labels = WikidataLabel._get_labels_wdapi(

missing_ids,

wb_url=normalized_wb_url

)

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:16:36Z

            # Cache labels
            WikidataLabel.add_bulk_labels(
-                [{"id": entity_id, "labels": entity_labels} for entity_id, entity_labels in missing_labels.items()]
+                [{"id": entity_id, "labels": entity_labels} for entity_id, entity_labels in missing_labels.items()],


Suggested change

[{"id": entity_id, "labels": entity_labels} for entity_id, entity_labels in missing_labels.items()],

[

{"id": entity_id, "labels": entity_labels}

for entity_id, entity_labels in missing_labels.items()

],

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:17:25Z

            dict[str, dict]: Mapping of each ID to compressed labels.
        """
-        entities_data = get_wikidata_json_by_ids(ids, props="labels")
+        entities_data = get_wikidata_json_by_ids(ids, action_api_url=wb_url, props="labels")


Suggested change

entities_data = get_wikidata_json_by_ids(ids, action_api_url=wb_url, props="labels")

entities_data = get_wikidata_json_by_ids(

ids,

action_api_url=wb_url,

props="labels"

)

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:24:15Z

+        calls.append((list(ids), wb_url))
+        return {"Q42": {"en": "Douglas Adams"}}
+
+    monkeypatch.setattr(WikidataLabel, "get_bulk_labels", staticmethod(fake_get_bulk_labels))


Suggested change

monkeypatch.setattr(WikidataLabel, "get_bulk_labels", staticmethod(fake_get_bulk_labels))

monkeypatch.setattr(

WikidataLabel,

"get_bulk_labels",

staticmethod(fake_get_bulk_labels)

)

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:24:58Z

+
+    monkeypatch.setattr(WikidataLabel, "get_bulk_labels", staticmethod(fake_get_bulk_labels))
+
+    factory = LazyLabelFactory(lang="en", fallback_lang="en", wb_url="https://example.wikibase.local/")


Suggested change

factory = LazyLabelFactory(lang="en", fallback_lang="en", wb_url="https://example.wikibase.local/")

factory = LazyLabelFactory(

lang="en",

fallback_lang="en",

wb_url="https://example.wikibase.local/"

)

exowanderer · 2026-05-23T14:27:55Z


        qids = [q.strip() for q in id.split(",")]
-        label_factory = LazyLabelFactory(lang=lang, fallback_lang=fallback_lang)
+        label_factory = LazyLabelFactory(lang=lang, fallback_lang=fallback_lang, wb_url=action_api_url)


Suggested change

label_factory = LazyLabelFactory(lang=lang, fallback_lang=fallback_lang, wb_url=action_api_url)

label_factory = LazyLabelFactory(

lang=lang,

fallback_lang=fallback_lang,

wb_url=action_api_url

)

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:28:50Z

-                entity_id=qids[0],
-                ttl_text=entity_data,
+        try:
+            entity_data = utils.get_wikidata_json_by_ids(qids, action_api_url=action_api_url)


Suggested change

entity_data = utils.get_wikidata_json_by_ids(qids, action_api_url=action_api_url)

entity_data = utils.get_wikidata_json_by_ids(

qids,

action_api_url=action_api_url

)

Ruff Lint's formatter will revert this back to one line

exowanderer · 2026-05-23T14:36:30Z

There are a lot of changes that should have been single commits.

Could you please comment on each deletion section and addition section, "what changed and why"?

philippesaade-wmde · 2026-05-26T12:18:38Z

 from fastapi.middleware.cors import CORSMiddleware

 from src import utils
-from src.Normalizer import JSONNormalizer, TTLNormalizer


TTLNormalizer has been removed to support Wikibases.
TTLNormalizer is only capable of parsing Wikidata entities while JSONNormalizer supports all Wikibases with an Action API endpoint.

philippesaade-wmde · 2026-05-26T12:20:07Z

-
-            entity_data = TTLNormalizer(
-                entity_id=qids[0],
-                ttl_text=entity_data,


In the previous implementation, TTLNormalizer was used to parse requents of one Wikidata entity. This has been removed and replaced by the JSONNormalizer to support Wikibase entities as well.

philippesaade-wmde · 2026-05-26T12:21:46Z

-                if entity
-                else None
-                for qid, entity in entity_data.items()
-            }


All the removed lines here are also based on the decision to remove TTLNormalizer and only use the JSONNormalizer

philippesaade-wmde added 7 commits April 8, 2026 11:44

Adding support for Wikibase: Specify Wikibase URL in API

d1007a4

Update doctrings and README for Wikibase support

0ba1c22

Add user agent in as an env variable

0b51b61

Replace Wikibase URL with Action API URL

67fb1bf

Mention future plan for GraphQL

6e336f4

Merge remote-tracking branch 'origin/main' into wb_support

a79e365

Format with Ruff

804e7c6

exowanderer reviewed May 22, 2026

View reviewed changes

philippesaade-wmde added 2 commits May 25, 2026 19:02

Addressing comments in PR, and renaming Wikidata classes to Wikibase …

9eaf866

…with additional aliases for Wikidata

Run Ruff Lint format

0befd7c

exowanderer reviewed May 26, 2026

View reviewed changes

philippesaade-wmde commented May 26, 2026

View reviewed changes

		@@ -302,8 +289,8 @@ class WikidataClaimValue:
		value: Optional[


		## Future Plan

		- Replace Action API with GraphQL once Wikibase GraphQL is available for Wikibases:


		monkeypatch.setattr(WikidataLabel, "get_bulk_labels", staticmethod(fake_get_bulk_labels))

		factory = LazyLabelFactory(lang="en", fallback_lang="en", wb_url="https://example.wikibase.local/")

Conversation

philippesaade-wmde commented May 12, 2026

Uh oh!

exowanderer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

exowanderer left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment