Pywikibot-commits April 2022

pywikibot-commits@lists.wikimedia.org

1 participants
126 discussions

[Gerrit] ...core[master]: [FEAT] Add support for Lexicographical data

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/461372 ) Change subject: [FEAT] Add support for Lexicographical data ...................................................................... [FEAT] Add support for Lexicographical data Bug: T189321 Change-Id: I808dab5c3c4ba8169423e9796bac24f39cabac52 --- M pywikibot/__init__.py M pywikibot/data/api.py M pywikibot/page/__init__.py M pywikibot/page/_collections.py M pywikibot/page/_wikibase.py M pywikibot/site/_datasite.py 6 files changed, 603 insertions(+), 4 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/__init__.py b/pywikibot/__init__.py index 4456faa..348cfa7 100644 --- a/pywikibot/__init__.py +++ b/pywikibot/__init__.py @@ -79,7 +79,8 @@ '__version__', 'Bot', 'calledModuleName', 'Category', 'Claim', 'Coordinate', 'critical', 'CurrentPageBot', 'debug', 'error', 'exception', 'FilePage', 'handle_args', - 'html2unicode', 'input', 'input_choice', 'input_yn', 'ItemPage', 'Link', + 'html2unicode', 'input', 'input_choice', 'input_yn', 'ItemPage', + 'LexemeForm', 'LexemePage', 'LexemeSense', 'Link', 'log', 'MediaInfo', 'output', 'Page', 'PropertyPage', 'showDiff', 'show_help', 'Site', 'SiteLink', 'stdout', 'Timestamp', 'translate', 'ui', 'url2unicode', 'User', 'warning', 'WbGeoShape', 'WbMonolingualText', @@ -1222,6 +1223,9 @@ Claim, FilePage, ItemPage, + LexemeForm, + LexemePage, + LexemeSense, Link, MediaInfo, Page, diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index c04e093..4a89f13 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -1031,6 +1031,8 @@ 'wbremovequalifiers', 'wbremovereferences', 'wbsetaliases', 'wbsetclaim', 'wbsetclaimvalue', 'wbsetdescription', 'wbsetlabel', 'wbsetqualifier', 'wbsetreference', 'wbsetsitelink', + 'wbladdform', 'wbleditformelements', 'wblmergelexemes', + 'wblremoveform', } # Client side verification that the request is being performed # by a logged in user, and warn if it isn't a config username. diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py index 2caa740..7470180 100644 --- a/pywikibot/page/__init__.py +++ b/pywikibot/page/__init__.py @@ -24,6 +24,9 @@ PropertyPage, WikibaseEntity, WikibasePage, + LexemePage, + LexemeForm, + LexemeSense, ) from pywikibot.site import BaseSite as _BaseSite from pywikibot.tools import deprecated, issue_deprecation_warning @@ -41,6 +44,9 @@ 'User', 'WikibasePage', 'ItemPage', + 'LexemePage', + 'LexemeForm', + 'LexemeSense', 'PropertyPage', 'Property', 'Claim', diff --git a/pywikibot/page/_collections.py b/pywikibot/page/_collections.py index 078a5cf..84ecfad 100644 --- a/pywikibot/page/_collections.py +++ b/pywikibot/page/_collections.py @@ -5,7 +5,7 @@ # Distributed under the terms of the MIT license. # from collections import defaultdict -from collections.abc import MutableMapping +from collections.abc import MutableMapping, MutableSequence from typing import Optional import pywikibot @@ -17,6 +17,7 @@ 'ClaimCollection', 'LanguageDict', 'SiteLinkCollection', + 'SubEntityCollection', ) @@ -476,3 +477,101 @@ for dbname in to_nuke: del data[dbname] return data + + +class SubEntityCollection(MutableSequence): + + """Ordered collection of sub-entities indexed by their ids.""" + + def __init__(self, repo, data=None): + """ + Initializer. + + :param repo: Wikibase site + :type repo: pywikibot.site.DataSite + :param data: iterable of LexemeSubEntity + :type data: iterable + """ + super().__init__() + self.repo = repo + self._data = [] + self._by_key = {} + if data: + self.extend(data) + + def _validate_isinstance(self, obj): + if not isinstance(obj, self.type_class): + raise TypeError( + '{} should only hold instances of {}, ' + 'instance of {} was provided' + .format(self.__class__.__name__, + self.type_class.__name__, + obj.__class__.__name__)) + + def __getitem__(self, index): + if isinstance(index, str): + try: + index = self._by_key[index] + except KeyError as e: + raise ValueError('No entity with id {} was found' + .format(index)) from e + return self._data[index] + + def __setitem__(self, index, value): + raise NotImplementedError + + def __delitem__(self, index): + if isinstance(index, str): + try: + index = self._by_key[index] + except KeyError as e: + raise ValueError('No entity with id {} was found' + .format(index)) from e + obj = self._data[index] + del self._data[index] + del self._by_key[obj.id] + + def __len__(self): + return len(self._data) + + def insert(self, index, obj): + """Insert a sub-entity to the collection.""" + self._validate_isinstance(obj) + self._data.insert(index, obj) + self._by_key[obj.id] = index + + @classmethod + def new_empty(cls, repo): + """Construct a new empty SubEntityCollection.""" + return cls(repo) + + @classmethod + def fromJSON(cls, data, repo): + """Construct a new SubEntityCollection from JSON.""" + this = cls(repo) + for entity in data: + this.append(cls.type_class.fromJSON(repo, entity)) + return this + + @classmethod + def normalizeData(cls, data: list) -> dict: + """ + Helper function to expand data into the Wikibase API structure. + + :param data: Data to normalize + :type data: list + + :return: the altered dict from parameter data. + """ + raise NotImplementedError # todo + + def toJSON(self, diffto: Optional[dict] = None) -> dict: + """ + Create JSON suitable for Wikibase API. + + When diffto is provided, JSON representing differences + to the provided data is created. + + :param diffto: JSON containing entity data + """ + raise NotImplementedError # todo diff --git a/pywikibot/page/_wikibase.py b/pywikibot/page/_wikibase.py index f90577e..b584121 100644 --- a/pywikibot/page/_wikibase.py +++ b/pywikibot/page/_wikibase.py @@ -40,6 +40,7 @@ ClaimCollection, LanguageDict, SiteLinkCollection, + SubEntityCollection, ) from pywikibot.page._decorators import allow_asynchronous from pywikibot.page._pages import BasePage, FilePage @@ -1854,3 +1855,373 @@ 'value': self._formatValue(), 'type': self.value_types.get(self.type, self.type) } + + +class LexemePage(WikibasePage): + + """Wikibase entity of type 'lexeme'.""" + + _cache_attrs = WikibasePage._cache_attrs + ( + 'lemmas', 'language', 'lexicalCategory', 'forms', 'senses', + ) + entity_type = 'lexeme' + title_pattern = r'L[1-9]\d*' + DATA_ATTRIBUTES = { + 'lemmas': LanguageDict, + 'claims': ClaimCollection, + # added when defined + # 'forms': LexemeFormCollection, + # 'senses': LexemeSenseCollection, + } + + def __init__(self, site, title=None) -> None: + """ + Initializer. + + :param site: data repository + :type site: pywikibot.site.DataSite + :param title: identifier of lexeme, "L###", + -1 or None for an empty lexeme. + :type title: str or None + """ + # Special case for empty lexeme. + if title is None or title == '-1': + super().__init__(site, '-1', entity_type='lexeme') + assert self.id == '-1' + return + + # we don't want empty titles + if not title: + raise InvalidTitleError("Lexeme's title cannot be empty") + + super().__init__(site, title, entity_type='lexeme') + assert self.id == self._link.title + + def get_data_for_new_entity(self): + """Return data required for creation of a new lexeme.""" + raise NotImplementedError # todo + + def toJSON(self, diffto: Optional[dict] = None) -> dict: + """ + Create JSON suitable for Wikibase API. + + When diffto is provided, JSON representing differences + to the provided data is created. + + :param diffto: JSON containing entity data + """ + data = super().toJSON(diffto=diffto) + + for prop in ('language', 'lexicalCategory'): + value = getattr(self, prop, None) + if not value: + continue + if not diffto or diffto.get(prop) != value.getID(): + data[prop] = value.getID() + + return data + + def get(self, force=False, get_redirect=False, *args, **kwargs): + """ + Fetch all lexeme data, and cache it. + + :param force: override caching + :type force: bool + :param get_redirect: return the lexeme content, do not follow the + redirect, do not raise an exception. + :type get_redirect: bool + :raise NotImplementedError: a value in args or kwargs + :note: dicts returned by this method are references to content + of this entity and their modifying may indirectly cause + unwanted change to the live content + """ + data = super().get(force, *args, **kwargs) + + if self.isRedirectPage() and not get_redirect: + raise IsRedirectPageError(self) + + # language + self.language = None + if 'language' in self._content: + self.language = ItemPage(self.site, self._content['language']) + + # lexicalCategory + self.lexicalCategory = None + if 'lexicalCategory' in self._content: + self.lexicalCategory = ItemPage( + self.site, self._content['lexicalCategory']) + + data['language'] = self.language + data['lexicalCategory'] = self.lexicalCategory + + return data + + @classmethod + def _normalizeData(cls, data: dict) -> dict: + """ + Helper function to expand data into the Wikibase API structure. + + :param data: The dict to normalize + :return: the altered dict from parameter data. + """ + new_data = WikibasePage._normalizeData(data) + for prop in ('language', 'lexicalCategory'): + value = new_data.get(prop) + if value: + if isinstance(value, ItemPage): + new_data[prop] = value.getID() + else: + new_data[prop] = value + return new_data + + @allow_asynchronous + def add_form(self, form, **kwargs): + """ + Add a form to the lexeme. + + :param form: The form to add + :type form: Form + :keyword bot: Whether to flag as bot (if possible) + :type bot: bool + :keyword asynchronous: if True, launch a separate thread to add form + asynchronously + :type asynchronous: bool + :keyword callback: a callable object that will be called after the + claim has been added. It must take two arguments: + (1) a LexemePage object, and + (2) an exception instance, which will be None if the entity was + saved successfully. This is intended for use by bots that need to + keep track of which saves were successful. + :type callback: callable + """ + if form.on_lexeme is not None: + raise ValueError('The provided LexemeForm instance is already ' + 'used in an entity') + data = self.repo.add_form(self, form, **kwargs) + form.id = data['form']['id'] + form.on_lexeme = self + form._content = data['form'] + form.get() + self.forms.append(form) + self.latest_revision_id = data['lastrevid'] + + def remove_form(self, form, **kwargs) -> None: + """ + Remove a form from the lexeme. + + :param form: The form to remove + :type form: pywikibot.LexemeForm + """ + data = self.repo.remove_form(form, **kwargs) + form.on_lexeme.latest_revision_id = data['lastrevid'] + form.on_lexeme.forms.remove(form) + form.on_lexeme = None + form.id = '-1' + + # todo: senses + + def mergeInto(self, lexeme, **kwargs): + """ + Merge the lexeme into another lexeme. + + :param lexeme: The lexeme to merge into + :type lexeme: LexemePage + """ + data = self.repo.mergeLexemes(from_lexeme=self, to_lexeme=lexeme, + **kwargs) + if not data.get('success', 0): + return + self.latest_revision_id = data['from']['lastrevid'] + lexeme.latest_revision_id = data['to']['lastrevid'] + if data.get('redirected', 0): + self._isredir = True + self._redirtarget = lexeme + + def isRedirectPage(self): + """Return True if lexeme is redirect, False if not or not existing.""" + if hasattr(self, '_content') and not hasattr(self, '_isredir'): + self._isredir = self.id != self._content.get('id', self.id) + return self._isredir + return super().isRedirectPage() + + +class LexemeSubEntity(WikibaseEntity): + + """Common super class for LexemeForm and LexemeSense.""" + + def __init__(self, repo, id_=None) -> None: + """Initializer.""" + super().__init__(repo, id_) + self._on_lexeme = None + + @classmethod + def fromJSON(cls, repo, data): + new = cls(repo, data['id']) + new._content = data + return new + + def toJSON(self, diffto=None) -> dict: + data = super().toJSON(diffto) + if self.id != '-1': + data['id'] = self.id + return data + + @property + def on_lexeme(self) -> LexemePage: + if self._on_lexeme is None: + lexeme_id = self.id.partition('-')[0] + self._on_lexeme = LexemePage(self.repo, lexeme_id) + return self._on_lexeme + + @on_lexeme.setter + def on_lexeme(self, lexeme): + self._on_lexeme = lexeme + + @on_lexeme.deleter + def on_lexeme(self): + self._on_lexeme = None + + @allow_asynchronous + def addClaim(self, claim, **kwargs): + """ + Add a claim to the form. + + :param claim: The claim to add + :type claim: Claim + :keyword bot: Whether to flag as bot (if possible) + :type bot: bool + :keyword asynchronous: if True, launch a separate thread to add claim + asynchronously + :type asynchronous: bool + :keyword callback: a callable object that will be called after the + claim has been added. It must take two arguments: (1) a Form + object, and (2) an exception instance, which will be None if the + form was saved successfully. This is intended for use by bots that + need to keep track of which saves were successful. + :type callback: callable + """ + self.repo.addClaim(self, claim, **kwargs) + claim.on_item = self + + def removeClaims(self, claims, **kwargs) -> None: + """ + Remove the claims from the form. + + :param claims: list of claims to be removed + :type claims: list or pywikibot.Claim + """ + # this check allows single claims to be removed by pushing them into a + # list of length one. + if isinstance(claims, pywikibot.Claim): + claims = [claims] + data = self.repo.removeClaims(claims, **kwargs) + for claim in claims: + claim.on_item.latest_revision_id = data['pageinfo']['lastrevid'] + claim.on_item = None + claim.snak = None + + +class LexemeForm(LexemeSubEntity): + + """Wikibase lexeme form.""" + + entity_type = 'form' + title_pattern = LexemePage.title_pattern + r'-F[1-9]\d*' + DATA_ATTRIBUTES = { + 'representations': LanguageDict, + 'claims': ClaimCollection, + } + + def toJSON(self, diffto: Optional[dict] = None) -> dict: + """Create dict suitable for the MediaWiki API.""" + data = super().toJSON(diffto=diffto) + + key = 'grammaticalFeatures' + if getattr(self, key, None): + # could also avoid if no change wrt. diffto + data[key] = [value.getID() for value in self.grammaticalFeatures] + + return data + + @classmethod + def _normalizeData(cls, data): + new_data = LexemeSubEntity._normalizeData(data) + if 'grammaticalFeatures' in data: + value = [] + for feat in data['grammaticalFeatures']: + if isinstance(feat, ItemPage): + value.append(feat.getID()) + else: + value.append(feat) + new_data['grammaticalFeatures'] = value + return new_data + + def get(self, force: bool = False) -> dict: + """ + Fetch all form data, and cache it. + + :param force: override caching + :note: dicts returned by this method are references to content + of this entity and their modifying may indirectly cause + unwanted change to the live content + """ + data = super().get(force=force) + + # grammaticalFeatures + self.grammaticalFeatures = set() + for value in self._content.get('grammaticalFeatures', []): + self.grammaticalFeatures.add(ItemPage(self.repo, value)) + + data['grammaticalFeatures'] = self.grammaticalFeatures + + return data + + def edit_elements(self, data: dict, **kwargs) -> None: + """ + Update form elements. + + :param data: Data to be saved + """ + if self.id == '-1': + # Update only locally + if 'representations' in data: + self.representations = LanguageDict(data['representations']) + + if 'grammaticalFeatures' in data: + self.grammaticalFeatures = set() + for value in data['grammaticalFeatures']: + if not isinstance(value, ItemPage): + value = ItemPage(self.repo, value) + self.grammaticalFeatures.add(value) + else: + data = self._normalizeData(data) + updates = self.repo.edit_form_elements(self, data, **kwargs) + self._content = updates['form'] + + +class LexemeSense(LexemeSubEntity): + + """Wikibase lexeme sense.""" + + entity_type = 'sense' + title_pattern = LexemePage.title_pattern + r'-S[1-9]\d*' + DATA_ATTRIBUTES = { + 'glosses': LanguageDict, + 'claims': ClaimCollection, + } + + +class LexemeFormCollection(SubEntityCollection): + + type_class = LexemeForm + + +class LexemeSenseCollection(SubEntityCollection): + + type_class = LexemeSense + + +LexemePage.DATA_ATTRIBUTES.update({ + 'forms': LexemeFormCollection, + 'senses': LexemeSenseCollection, +}) diff --git a/pywikibot/site/_datasite.py b/pywikibot/site/_datasite.py index 7662092..b729f25 100644 --- a/pywikibot/site/_datasite.py +++ b/pywikibot/site/_datasite.py @@ -21,7 +21,7 @@ NoWikibaseEntityError, ) from pywikibot.site._apisite import APISite -from pywikibot.site._decorators import need_right, need_version +from pywikibot.site._decorators import need_extension, need_right, need_version from pywikibot.tools import itergroup, merge_unique_dicts, remove_last_args @@ -42,6 +42,9 @@ 'item': pywikibot.ItemPage, 'property': pywikibot.PropertyPage, 'mediainfo': pywikibot.MediaInfo, + 'lexeme': pywikibot.LexemePage, + 'form': pywikibot.LexemeForm, + 'sense': pywikibot.LexemeSense, } def _cache_entity_namespaces(self) -> None: @@ -69,7 +72,8 @@ if entity_type in self._entity_namespaces: return self._entity_namespaces[entity_type] raise EntityTypeUnknownError( - '{!r} does not support entity type "{}"' + '{!r} does not support entity type "{}" ' + "or it doesn't have its own namespace" .format(self, entity_type)) @property @@ -626,6 +630,35 @@ req = self.simple_request(**params) return req.submit() + @need_right('item-merge') + @need_extension('WikibaseLexeme') + def mergeLexemes(self, from_lexeme, to_lexeme, summary=None, *, + bot: bool = True) -> dict: + """ + Merge two lexemes together. + + :param from_lexeme: Lexeme to merge from + :type from_lexeme: pywikibot.LexemePage + :param to_lexeme: Lexeme to merge into + :type to_lexeme: pywikibot.LexemePage + :param summary: Edit summary + :type summary: str + :keyword bot: Whether to mark the edit as a bot edit + :return: dict API output + """ + params = { + 'action': 'wblmergelexemes', + 'source': from_lexeme.getID(), + 'target': to_lexeme.getID(), + 'token': self.tokens['edit'], + 'summary': summary, + } + if bot: + params['bot'] = 1 + req = self._simple_request(**params) + data = req.submit() + return data + @need_right('item-redirect') def set_redirect_target(self, from_item, to_item, bot: bool = True): """ @@ -823,3 +856,87 @@ See self._wbset_action() for parameters """ return self._wbset_action(itemdef, 'wbsetsitelink', sitelink, **kwargs) + + @need_right('edit') + @need_extension('WikibaseLexeme') + def add_form(self, lexeme, form, *, bot: bool = True, + baserevid=None) -> dict: + """ + Add a form. + + :param lexeme: Lexeme to modify + :type lexeme: pywikibot.LexemePage + :param form: Form to be added + :type form: pywikibot.LexemeForm + :keyword bot: Whether to mark the edit as a bot edit + :keyword baserevid: Base revision id override, used to detect + conflicts. + :type baserevid: long + """ + params = { + 'action': 'wbladdform', + 'lexemeId': lexeme.getID(), + 'data': json.dumps(form.toJSON()), + 'bot': bot, + 'token': self.tokens['edit'], + } + if baserevid: + params['baserevid'] = baserevid + req = self._simple_request(**params) + data = req.submit() + return data + + @need_right('edit') + @need_extension('WikibaseLexeme') + def remove_form(self, form, *, bot: bool = True, baserevid=None) -> dict: + """ + Remove a form. + + :param form: Form to be removed + :type form: pywikibot.LexemeForm + :keyword bot: Whether to mark the edit as a bot edit + :keyword baserevid: Base revision id override, used to detect + conflicts. + :type baserevid: long + """ + params = { + 'action': 'wblremoveform', + 'id': form.getID(), + 'bot': bot, + 'token': self.tokens['edit'], + } + if baserevid: + params['baserevid'] = baserevid + req = self._simple_request(**params) + data = req.submit() + return data + + @need_right('edit') + @need_extension('WikibaseLexeme') + def edit_form_elements(self, form, data, *, bot: bool = True, + baserevid=None) -> dict: + """ + Edit lexeme form elements. + + :param form: Form + :type form: pywikibot.LexemeForm + :param data: data updates + :type data: dict + :keyword bot: Whether to mark the edit as a bot edit + :keyword baserevid: Base revision id override, used to detect + conflicts. + :type baserevid: long + :return: New form data + """ + params = { + 'action': 'wbleditformelements', + 'formId': form.getID(), + 'data': json.dumps(data), + 'bot': bot, + 'token': self.tokens['edit'], + } + if baserevid: + params['baserevid'] = baserevid + req = self._simple_request(**params) + data = req.submit() + return data -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/461372 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I808dab5c3c4ba8169423e9796bac24f39cabac52 Gerrit-Change-Number: 461372 Gerrit-PatchSet: 21 Gerrit-Owner: Rua <ruawhitepaw(a)gmail.com> Gerrit-Assignee: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-CC: Lokal Profil <andre.costa(a)wikimedia.se> Gerrit-CC: Welcome, new contributor! <ssethi(a)wikimedia.org> Gerrit-MessageType: merged

2 years, 1 month

[Gerrit] ...core[master]: [IMPR] enable all parameters of APISite.imageusage() with FilePage.us...

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/775948 ) Change subject: [IMPR] enable all parameters of APISite.imageusage() with FilePage.usingPages() ...................................................................... [IMPR] enable all parameters of APISite.imageusage() with FilePage.usingPages() Change-Id: Ifda1a4ba119a9f7904f30c48468359dda4e2a288 --- M pywikibot/page/_pages.py M pywikibot/site/_generators.py 2 files changed, 19 insertions(+), 9 deletions(-) Approvals: Matěj Suchánek: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page/_pages.py b/pywikibot/page/_pages.py index f095424..3eea4ad 100644 --- a/pywikibot/page/_pages.py +++ b/pywikibot/page/_pages.py @@ -2473,14 +2473,19 @@ '| {{int:filehist-dimensions}} || {{int:filehist-comment}}\n' '|-\n%s\n|}\n' % '\n|-\n'.join(lines)) - def usingPages(self, total: Optional[int] = None, content: bool = False): + def usingPages(self, **kwargs): """Yield Pages on which the file is displayed. - :param total: iterate no more than this number of pages in total - :param content: if True, load the current content of each iterated page - (default False) + For parameters refer + :meth:`APISite.imageusage() + <pywikibot.site._generators.GeneratorsMixin.imageusage>` + + .. versionchanged:: 7.2 + all parameters from :meth:`APISite.imageusage() + <pywikibot.site._generators.GeneratorsMixin.imageusage>` + are available. """ - return self.site.imageusage(self, total=total, content=content) + return self.site.imageusage(self, **kwargs) @property def file_is_used(self) -> bool: diff --git a/pywikibot/site/_generators.py b/pywikibot/site/_generators.py index bf0b494..d6ecd73 100644 --- a/pywikibot/site/_generators.py +++ b/pywikibot/site/_generators.py @@ -1186,21 +1186,26 @@ namespaces=namespaces, total=total, g_content=content) - def imageusage(self, image, namespaces=None, filterredir=None, - total=None, content: bool = False): + def imageusage(self, image: 'pywikibot.FilePage', *, + namespaces=None, + filterredir: Optional[bool] = None, + total: Optional[int] = None, + content: bool = False): """Iterate Pages that contain links to the given FilePage. - :see: https://www.mediawiki.org/wiki/API:Imageusage + .. seealso:: https://www.mediawiki.org/wiki/API:Imageusage + .. versionchanged:: 7.2 + all parameters except `image` are keyword only. :param image: the image to search for (FilePage need not exist on the wiki) - :type image: pywikibot.FilePage :param namespaces: If present, only iterate pages in these namespaces :type namespaces: iterable of str or Namespace key, or a single instance of those types. May be a '|' separated list of namespace identifiers. :param filterredir: if True, only yield redirects; if False (and not None), only yield non-redirects (default: yield both) + :param total: iterate no more than this number of pages in total :param content: if True, load the current content of each iterated page (default False) :raises KeyError: a namespace identifier was not resolved -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/775948 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Ifda1a4ba119a9f7904f30c48468359dda4e2a288 Gerrit-Change-Number: 775948 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

2 years, 1 month

[Gerrit] ...core[master]: [fix] Don't raise NoPageError with file_is_shared

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/775922 ) Change subject: [fix] Don't raise NoPageError with file_is_shared ...................................................................... [fix] Don't raise NoPageError with file_is_shared Instead raising NoPageError, FilePage.file_is_shared returns False if a file neither exists locally nor on shared image repository (commons) Bug: T305182 Change-Id: I4d20bbc599619c74285232db31f3e1f4d05aa78e --- M pywikibot/page/_pages.py M tests/file_tests.py 2 files changed, 15 insertions(+), 8 deletions(-) Approvals: Matěj Suchánek: Looks good to me, but someone else must approve Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/page/_pages.py b/pywikibot/page/_pages.py index f095424..e9476f9 100644 --- a/pywikibot/page/_pages.py +++ b/pywikibot/page/_pages.py @@ -2446,17 +2446,27 @@ return self.latest_file_info.thumburl def file_is_shared(self) -> bool: - """Check if the file is stored on any known shared repository.""" + """Check if the file is stored on any known shared repository. + + .. versionchanged:: 7.0 + return False if file does not exist on shared image repository + instead raising NoPageError. + """ # as of now, the only known repositories are commons and wikitravel # TODO: put the URLs to family file if not self.site.has_image_repository: return False + try: + info = self.latest_file_info + except NoPageError: + return False + if 'wikitravel_shared' in self.site.shared_image_repository(): - return self.latest_file_info.url.startswith( - 'https://wikitravel.org/upload/shared/') + return info.url.startswith('https://wikitravel.org/upload/shared/') + # default to commons - return self.latest_file_info.url.startswith( + return info.url.startswith( 'https://upload.wikimedia.org/wikipedia/commons/') def getFileVersionHistoryTable(self): diff --git a/tests/file_tests.py b/tests/file_tests.py index ef81ba4..8aed05a 100755 --- a/tests/file_tests.py +++ b/tests/file_tests.py @@ -91,13 +91,10 @@ self.assertFalse(commons_file.exists()) self.assertFalse(enwp_file.file_is_shared()) + self.assertFalse(commons_file.file_is_shared()) page_doesnt_exist_exc_regex = re.escape( "Page [[commons:{}]] doesn't exist.".format(title)) - with self.assertRaisesRegex( - NoPageError, - page_doesnt_exist_exc_regex): - commons_file.file_is_shared() with self.assertRaisesRegex( NoPageError, -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/775922 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: I4d20bbc599619c74285232db31f3e1f4d05aa78e Gerrit-Change-Number: 775922 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

2 years, 1 month

[Gerrit] ...core[master]: [IMPR] Use itertools.zip_longest to find the most important image

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/773930 ) Change subject: [IMPR] Use itertools.zip_longest to find the most important image ...................................................................... [IMPR] Use itertools.zip_longest to find the most important image - important_image uses usingPages() to find the most important image but retrieving all pages need a long time. Now we collect all usingPages from all images in list_given parallel and stop if the generator produces only one image and the other tuple entries are False. - make important_image a static method - test it Change-Id: Idd8354469d444b72357a5a55b111cc9810fd6715 --- M scripts/checkimages.py M tests/checkimages_tests.py 2 files changed, 37 insertions(+), 14 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/checkimages.py b/scripts/checkimages.py index 4ec829b..af18f95 100755 --- a/scripts/checkimages.py +++ b/scripts/checkimages.py @@ -84,6 +84,8 @@ import collections import re import time + +from itertools import zip_longest from typing import Generator import pywikibot @@ -803,26 +805,33 @@ for element in self.load(page_hidden_text): self.hiddentemplates.add(pywikibot.Page(self.site, element)) - def important_image(self, list_given) -> pywikibot.FilePage: + @staticmethod + def important_image( + list_given: List[Tuple[float, pywikibot.FilePage]] + ) -> pywikibot.FilePage: """ Get tuples of image and time, return the most used or oldest image. + .. versionchanged: 7.2 + itertools.zip_longest is used to stop `usingPages` as soon as + possible. + :param list_given: a list of tuples which hold seconds and FilePage - :type list_given: list :return: the most used or oldest image """ # find the most used image - inx_found = None # index of found image - max_usage = 0 # hold max amount of using pages - for num, element in enumerate(list_given): - image = element[1] - image_used = len(list(image.usingPages())) - if image_used > max_usage: - max_usage = image_used - inx_found = num + images = [image for _, image in list_given] + iterables = [image.usingPages() for image in images] + curr_images = [] + for values in zip_longest(*iterables, fillvalue=False): + curr_images = values + # bool(FilePage) is True because it is an object subclass + if sum(bool(image) for image in values) <= 1: + break - if inx_found is not None: - return list_given[inx_found][1] + for inx, image in enumerate(curr_images): + if image is not False: + return images[inx] # find the oldest image _, image = max(list_given, key=lambda element: element[0]) diff --git a/tests/checkimages_tests.py b/tests/checkimages_tests.py index b0da988..2666052 100755 --- a/tests/checkimages_tests.py +++ b/tests/checkimages_tests.py @@ -1,14 +1,16 @@ #!/usr/bin/python3 """Unit tests for checkimages script.""" # -# (C) Pywikibot team, 2015-2021 +# (C) Pywikibot team, 2015-2022 # # Distributed under the terms of the MIT license. # import unittest +from pywikibot import FilePage + from scripts import checkimages -from tests.aspects import TestCase +from tests.aspects import DefaultSiteTestCase, TestCase class TestSettings(TestCase): @@ -29,5 +31,17 @@ self.assertEqual(item1[1], 'a deprecated template') +class TestMethods(DefaultSiteTestCase): + + """Test methods of CheckImagesBot.""" + + def test_important_image(self): + """Test important_image method.""" + filenames = ('Example.jpg', 'Demo.jpg') + images = [(0.0, FilePage(self.site, name)) for name in filenames] + self.assertEqual(checkimages.CheckImagesBot.important_image(images), + FilePage(self.site, 'Example.jpg')) + + if __name__ == '__main__': # pragma: no cover unittest.main() -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/773930 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Idd8354469d444b72357a5a55b111cc9810fd6715 Gerrit-Change-Number: 773930 Gerrit-PatchSet: 4 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

2 years, 1 month

[Gerrit] ...core[master]: [doc] Update documentation

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/776044 ) Change subject: [doc] Update documentation ...................................................................... [doc] Update documentation Change-Id: Ia1869d09dc21e9d431ce04b6184cc4552378e64f --- M scripts/archive/makecat.py M scripts/archive/match_images.py M scripts/archive/piper.py 3 files changed, 3 insertions(+), 3 deletions(-) Approvals: Meno25: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/archive/makecat.py b/scripts/archive/makecat.py index aed65ec..95abffc 100755 --- a/scripts/archive/makecat.py +++ b/scripts/archive/makecat.py @@ -44,7 +44,7 @@ * [r]educe - reduce menu bar """ -# (C) Pywikibot team, 2004-2020 +# (C) Pywikibot team, 2004-2022 # # Distributed under the terms of the MIT license. # diff --git a/scripts/archive/match_images.py b/scripts/archive/match_images.py index 9b252e0..1f2bafc 100755 --- a/scripts/archive/match_images.py +++ b/scripts/archive/match_images.py @@ -22,7 +22,7 @@ Expect the code to change a lot! """ # -# (c) Pywikibot team, 2009-2020 +# (c) Pywikibot team, 2009-2022 # # Distributed under the terms of the MIT license. # diff --git a/scripts/archive/piper.py b/scripts/archive/piper.py index 5918448..453c1da 100755 --- a/scripts/archive/piper.py +++ b/scripts/archive/piper.py @@ -30,7 +30,7 @@ &params; """ # -# (C) Pywikibot team, 2008-2020 +# (C) Pywikibot team, 2008-2022 # # Distributed under the terms of the MIT license. # -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/776044 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Ia1869d09dc21e9d431ce04b6184cc4552378e64f Gerrit-Change-Number: 776044 Gerrit-PatchSet: 2 Gerrit-Owner: Meno25 <meno25mail(a)gmail.com> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: Meno25 <meno25mail(a)gmail.com> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

2 years, 1 month

[Gerrit] ...core[master]: [bugfix] Improve image regex

by jenkins-bot (Code Review)

jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/776158 ) Change subject: [bugfix] Improve image regex ...................................................................... [bugfix] Improve image regex - respect lowercased namespace with image.py - make image regex less greedy Bug: T305226 Bug: T305227 Change-Id: Id5ba075c4aff030195fe73546103647958d0fe9d --- M scripts/image.py 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: D3r1ck01: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/image.py b/scripts/image.py index a40300d..a52a208 100755 --- a/scripts/image.py +++ b/scripts/image.py @@ -91,7 +91,7 @@ escaped = re.sub('\\\\[_ ]', '[_ ]', escaped) if not self.opt.loose or not self.new_image: image_regex = re.compile( - r'\[\[ *(?:{})\s*:\s*{} *(?P<parameters>\|[^\n]+|) *\]\]' + r'\[\[ *(?i:{})\s*:\s*{} *(?P<parameters>\|[^\n]+?|) *\]\]' .format('|'.join(namespace), escaped)) else: image_regex = re.compile(r'' + escaped) -- To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/776158 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-Change-Id: Id5ba075c4aff030195fe73546103647958d0fe9d Gerrit-Change-Number: 776158 Gerrit-PatchSet: 3 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: MarcoAurelio <maurelio(a)toolforge.org> Gerrit-Reviewer: jenkins-bot Gerrit-MessageType: merged

2 years, 1 month

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits April 2022