Pywikibot-commits March 2020

pywikibot-commits@lists.wikimedia.org

1 participants
124 discussions

[Gerrit] pywikibot/core[master]: [doc] Update HISTORY.rst

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583692 ) Change subject: [doc] Update HISTORY.rst ...................................................................... [doc] Update HISTORY.rst Change-Id: I9be41269204df6f3ba8e3d6243804ee3a2a171d8 --- M HISTORY.rst 1 file changed, 6 insertions(+), 0 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/HISTORY.rst b/HISTORY.rst index a5c3dc4..d17e43b 100644 --- a/HISTORY.rst +++ b/HISTORY.rst @@ -4,6 +4,12 @@ Current release --------------- +* Refactor data attributes of Wikibase entities (T233406) +* Functions dealing with stars list are desupported and may be removed +* Use path's stem of script filename within pwb.py wrapper (T248372) +* Disfunctional cgi_interface.py was removed (T248292, T248250, T193978) +* Fix logout on MW < 1.24 (T214009) +* Fixed TypeError in getFileVersionHistoryTable method (T248266) * Outdated secure connection overrides were removed (T247668) * Check for all modules which are needed by a script within pwb.py wrapper * Check for all modules which are mandatory within pwb.py wrapper script -- To view, visit https://gerrit.wikimedia.org/r/583692 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9be41269204df6f3ba8e3d6243804ee3a2a171d8 Gerrit-Change-Number: 583692 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [L10N] Update languages_by_size

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583695 ) Change subject: [L10N] Update languages_by_size ...................................................................... [L10N] Update languages_by_size Change-Id: I9b5e4042e6960d17b93f50a2db4579807f11d6b7 --- M pywikibot/families/wikibooks_family.py M pywikibot/families/wikipedia_family.py M pywikibot/families/wikisource_family.py 3 files changed, 17 insertions(+), 17 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/families/wikibooks_family.py b/pywikibot/families/wikibooks_family.py index b07d7a4..c0553b2 100644 --- a/pywikibot/families/wikibooks_family.py +++ b/pywikibot/families/wikibooks_family.py @@ -34,7 +34,7 @@ languages_by_size = [ 'en', 'hu', 'de', 'fr', 'pt', 'ja', 'it', 'nl', 'es', 'pl', 'he', 'id', - 'fi', 'sq', 'zh', 'az', 'fa', 'vi', 'ca', 'ru', 'da', 'ko', 'th', 'sv', + 'fi', 'sq', 'zh', 'az', 'fa', 'vi', 'ca', 'ru', 'da', 'ko', 'sv', 'th', 'gl', 'sr', 'cs', 'ba', 'hr', 'ar', 'no', 'tr', 'sa', 'ta', 'hi', 'uk', 'eo', 'sk', 'is', 'ro', 'bn', 'si', 'mk', 'bg', 'ka', 'ms', 'tt', 'lt', 'el', 'li', 'ur', 'sl', 'km', 'tl', 'la', 'et', 'kk', 'be', 'ia', 'ml', diff --git a/pywikibot/families/wikipedia_family.py b/pywikibot/families/wikipedia_family.py index 5587fe4..b3ff069 100644 --- a/pywikibot/families/wikipedia_family.py +++ b/pywikibot/families/wikipedia_family.py @@ -32,26 +32,26 @@ languages_by_size = [ 'en', 'ceb', 'sv', 'de', 'fr', 'nl', 'ru', 'it', 'es', 'pl', 'war', 'vi', 'ja', 'zh', 'ar', 'pt', 'uk', 'fa', 'ca', 'sr', 'no', 'id', 'ko', - 'fi', 'hu', 'sh', 'cs', 'ro', 'eu', 'tr', 'ms', 'zh-min-nan', 'eo', - 'hy', 'he', 'bg', 'da', 'ce', 'sk', 'kk', 'min', 'hr', 'arz', 'et', - 'azb', 'lt', 'be', 'el', 'sl', 'gl', 'simple', 'az', 'ur', 'nn', 'hi', - 'th', 'ka', 'uz', 'la', 'ta', 'vo', 'cy', 'mk', 'lv', 'tg', 'ast', + 'fi', 'hu', 'sh', 'cs', 'ro', 'eu', 'tr', 'zh-min-nan', 'ms', 'eo', + 'hy', 'he', 'bg', 'da', 'ce', 'sk', 'kk', 'min', 'azb', 'arz', 'hr', + 'et', 'lt', 'be', 'el', 'sl', 'gl', 'simple', 'az', 'ur', 'nn', 'hi', + 'th', 'ka', 'uz', 'la', 'ta', 'vo', 'cy', 'mk', 'ast', 'lv', 'tg', 'mg', 'af', 'tt', 'oc', 'bn', 'bs', 'ky', 'sq', 'zh-yue', 'new', 'tl', - 'te', 'be-tarask', 'ml', 'br', 'pms', 'nds', 'su', 'ht', 'lb', 'jv', - 'sw', 'sco', 'mr', 'pnb', 'ga', 'szl', 'ba', 'is', 'my', 'fy', 'cv', + 'te', 'be-tarask', 'ml', 'br', 'pms', 'nds', 'su', 'ht', 'lb', 'sw', + 'jv', 'sco', 'mr', 'pnb', 'ga', 'szl', 'ba', 'is', 'my', 'fy', 'cv', 'lmo', 'an', 'ne', 'pa', 'yo', 'bar', 'wuu', 'io', 'gu', 'als', 'ku', - 'scn', 'kn', 'ckb', 'bpy', 'ia', 'qu', 'vec', 'mn', 'bat-smg', 'wa', + 'scn', 'kn', 'ckb', 'bpy', 'ia', 'vec', 'qu', 'mn', 'bat-smg', 'wa', 'or', 'si', 'cdo', 'gd', 'yi', 'am', 'nap', 'ilo', 'bug', 'xmf', 'hsb', - 'mai', 'diq', 'map-bms', 'fo', 'mzn', 'sd', 'nv', 'li', 'eml', 'sah', - 'os', 'sa', 'ps', 'frr', 'ace', 'mrj', 'zh-classical', 'mhr', 'hif', - 'bcl', 'hak', 'roa-tara', 'pam', 'nso', 'km', 'hyw', 'se', 'rue', 'mi', + 'mai', 'diq', 'map-bms', 'fo', 'sd', 'mzn', 'nv', 'li', 'eml', 'sah', + 'os', 'sa', 'ps', 'frr', 'ace', 'mrj', 'zh-classical', 'bcl', 'mhr', + 'hak', 'hif', 'roa-tara', 'pam', 'nso', 'km', 'hyw', 'se', 'rue', 'mi', 'crh', 'vls', 'bh', 'shn', 'nah', 'nds-nl', 'sc', 'gan', 'vep', 'as', 'ab', 'glk', 'myv', 'bo', 'co', 'so', 'tk', 'fiu-vro', 'lrc', 'kv', 'csb', 'sn', 'gv', 'udm', 'ie', 'zea', 'ha', 'pcd', 'ay', 'kab', 'nrm', - 'ug', 'lez', 'stq', 'kw', 'haw', 'mwl', 'lfn', 'gn', 'gom', 'lij', - 'rm', 'lo', 'gor', 'lad', 'frp', 'koi', 'mt', 'fur', 'olo', 'dsb', - 'dty', 'ang', 'ext', 'ln', 'cbk-zam', 'dv', 'bjn', 'ksh', 'gag', 'pfl', - 'sat', 'pi', 'pag', 'av', 'ban', 'bxr', 'tyv', 'xal', 'krc', 'za', + 'ug', 'lez', 'stq', 'kw', 'haw', 'gor', 'lfn', 'mwl', 'gn', 'gom', + 'lij', 'rm', 'lo', 'lad', 'frp', 'koi', 'mt', 'fur', 'olo', 'dsb', + 'dty', 'ang', 'ext', 'ln', 'cbk-zam', 'dv', 'bjn', 'ksh', 'gag', 'sat', + 'pfl', 'pi', 'pag', 'ban', 'av', 'bxr', 'tyv', 'xal', 'krc', 'za', 'pap', 'kaa', 'pdc', 'rw', 'szy', 'to', 'arc', 'nov', 'kl', 'jam', 'tpi', 'kbp', 'kbd', 'tet', 'ig', 'zu', 'wo', 'ki', 'na', 'tcy', 'inh', 'jbo', 'roa-rup', 'bi', 'lbe', 'kg', 'ty', 'mdf', 'lg', 'atj', 'srn', diff --git a/pywikibot/families/wikisource_family.py b/pywikibot/families/wikisource_family.py index 42dd9d6..5fa8380 100644 --- a/pywikibot/families/wikisource_family.py +++ b/pywikibot/families/wikisource_family.py @@ -29,9 +29,9 @@ languages_by_size = [ 'pl', 'en', 'ru', 'de', 'fr', 'zh', 'he', 'it', 'es', 'ar', 'cs', 'pt', - 'mul', 'gu', 'fa', 'uk', 'hu', 'ml', 'sv', 'sr', 'ko', 'bn', 'sa', + 'mul', 'gu', 'uk', 'fa', 'hu', 'ml', 'sv', 'sr', 'ko', 'bn', 'sa', 'hy', 'sl', 'te', 'el', 'ro', 'fi', 'ta', 'ja', 'nap', 'vi', 'az', - 'ca', 'br', 'th', 'nl', 'kn', 'hr', 'la', 'no', 'is', 'eo', 'vec', + 'br', 'ca', 'th', 'nl', 'kn', 'hr', 'la', 'no', 'is', 'eo', 'vec', 'tr', 'pms', 'et', 'be', 'da', 'mk', 'id', 'yi', 'bg', 'mr', 'li', 'as', 'hi', 'or', 'lt', 'eu', 'pa', 'gl', 'bs', 'sah', 'sk', 'cy', 'zh-min-nan', 'fo', -- To view, visit https://gerrit.wikimedia.org/r/583695 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9b5e4042e6960d17b93f50a2db4579807f11d6b7 Gerrit-Change-Number: 583695 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: Revert "[IMPR] Don't trust token from NeedToken response"

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583707 ) Change subject: Revert "[IMPR] Don't trust token from NeedToken response" ...................................................................... Revert "[IMPR] Don't trust token from NeedToken response" Fails login on MW < 1.27. Needs to be wrapped with try-except This reverts commit 70abc6550ec9b554ff539b689a8a9d767edd8f1d. Change-Id: I53597e4bd70692473c0bf633362f5c50187bb970 --- M pywikibot/data/api.py 1 file changed, 2 insertions(+), 32 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index 4a31c7f..d65c260 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -3143,8 +3143,7 @@ return '' elif login_result['login']['result'] == 'NeedToken': # Kept for backwards compatibility - # Generate a new token if invalid, see T224712 - token = self.get_login_token(login_result['login']['token']) + token = login_result['login']['token'] login_request['lgtoken'] = token continue elif login_result['login']['result'] == 'Throttled': @@ -3160,21 +3159,17 @@ """Ignore data; cookies are set by threadedhttp module.""" http.cookie_jar.save() - def get_login_token(self, old_token=None): + def get_login_token(self): """Fetch login token from action=query&meta=tokens. Requires MediaWiki >= 1.27. - @param old_token: old token to try - @type old_token: str @return: login token @rtype: str """ if self.site.mw_version < '1.27': raise NotImplementedError('The method get_login_token() requires ' 'at least MediaWiki version 1.27.') - if old_token and self.check_login_token(old_token) == 'valid': - return old_token login_token_request = self.site._request( use_get=False, parameters={'action': 'query', 'meta': 'tokens', 'type': 'login'}, @@ -3182,31 +3177,6 @@ login_token_result = login_token_request.submit() return login_token_result['query']['tokens'].get('logintoken') - def check_login_token(self, token): - """Check login token from action=checktoken. - - Requires MediaWiki >= 1.27. - - @param token: token to check - @tyoe token: str - @return: token validity status - @rtype: str - """ - if self.site.mw_version < '1.27': - raise NotImplementedError('The method check_login_token() ' - 'requires at least MediaWiki ' - 'version 1.27.') - check_token_request = self.site._request( - use_get=False, - parameters={ - 'action': 'checktoken', - 'type': 'login', - 'token': token, - }, - ) - check_token_result = check_token_request.submit() - return check_token_result['checktoken'].get('result') - def encode_url(query): """ -- To view, visit https://gerrit.wikimedia.org/r/583707 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I53597e4bd70692473c0bf633362f5c50187bb970 Gerrit-Change-Number: 583707 Gerrit-PatchSet: 1 Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: D3r1ck <alangiderick(a)gmail.com> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: Huji <huji.huji(a)gmail.com> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <Ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm(a)member.fsf.org> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [cleanup] Move several bot_choice options to bot_choice.py

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/511729 ) Change subject: [cleanup] Move several bot_choice options to bot_choice.py ...................................................................... [cleanup] Move several bot_choice options to bot_choice.py - Move Choice, StaticChoice, LinkChoice, AlwaysChoice from bot.py to bot_choice.py. - Also add them to the interface Change-Id: If74ad55fab0d0072222945756a10dd84e88ae35a --- M pywikibot/bot.py M pywikibot/bot_choice.py 2 files changed, 101 insertions(+), 100 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/bot.py b/pywikibot/bot.py index 954d07f..353b784 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -64,7 +64,7 @@ 'ContextOption', 'ListOption', 'ShowingListOption', 'MultipleChoiceList', 'ShowingMultipleChoiceList', 'OutputProxyOption', 'HighlightContextOption', 'ChoiceException', 'UnhandledAnswer', - 'Choice', 'AlwaysChoice', + 'Choice', 'StaticChoice', 'LinkChoice', 'AlwaysChoice', 'QuitKeyboardInterrupt', 'InteractiveReplace', 'calledModuleName', 'handle_args', 'handleArgs', @@ -104,6 +104,7 @@ ListOption, ShowingListOption, MultipleChoiceList, ShowingMultipleChoiceList, OutputProxyOption, HighlightContextOption, ChoiceException, QuitKeyboardInterrupt, + Choice, StaticChoice, LinkChoice, AlwaysChoice ) from pywikibot.logging import ( CRITICAL, ERROR, INFO, WARNING, @@ -585,105 +586,6 @@ force=force) -class Choice(StandardOption): - - """A simple choice consisting of a option, shortcut and handler.""" - - def __init__(self, option, shortcut, replacer): - """Initializer.""" - super(Choice, self).__init__(option, shortcut) - self._replacer = replacer - - @property - def replacer(self): - """The replacer.""" - return self._replacer - - def handle(self): - """Handle this choice. Must be implemented.""" - raise NotImplementedError() - - def handle_link(self): - """The current link will be handled by this choice.""" - return False - - -class StaticChoice(Choice): - - """A static choice which just returns the given value.""" - - def __init__(self, option, shortcut, result): - """Create instance with replacer set to None.""" - super(StaticChoice, self).__init__(option, shortcut, None) - self._result = result - - def handle(self): - """Return the predefined value.""" - return self._result - - -class LinkChoice(Choice): - - """A choice returning a mix of the link new and current link.""" - - def __init__(self, option, shortcut, replacer, replace_section, - replace_label): - """Initializer.""" - super(LinkChoice, self).__init__(option, shortcut, replacer) - self._section = replace_section - self._label = replace_label - - def handle(self): - """Handle by either applying the new section or label.""" - kwargs = {} - if self._section: - kwargs['section'] = self.replacer._new.section - else: - kwargs['section'] = self.replacer.current_link.section - if self._label: - if self.replacer._new.anchor is None: - kwargs['label'] = self.replacer._new.canonical_title() - if self.replacer._new.section: - kwargs['label'] += '#' + self.replacer._new.section - else: - kwargs['label'] = self.replacer._new.anchor - else: - if self.replacer.current_link.anchor is None: - kwargs['label'] = self.replacer.current_groups['title'] - if self.replacer.current_groups['section']: - kwargs['label'] += '#' + \ - self.replacer.current_groups['section'] - else: - kwargs['label'] = self.replacer.current_link.anchor - return pywikibot.Link.create_separated( - self.replacer._new.canonical_title(), self.replacer._new.site, - **kwargs) - - -class AlwaysChoice(Choice): - - """Add an option to always apply the default.""" - - def __init__(self, replacer, option='always', shortcut='a'): - """Initializer.""" - super(AlwaysChoice, self).__init__(option, shortcut, replacer) - self.always = False - - def handle(self): - """Handle the custom shortcut.""" - self.always = True - return self.answer - - def handle_link(self): - """Directly return answer whether it's applying it always.""" - return self.always - - @property - def answer(self): - """Get the actual default answer instructing the replacement.""" - return self.replacer.handle_answer(self.replacer._default) - - class InteractiveReplace(object): """ diff --git a/pywikibot/bot_choice.py b/pywikibot/bot_choice.py index 74a519b..a512f69 100755 --- a/pywikibot/bot_choice.py +++ b/pywikibot/bot_choice.py @@ -225,6 +225,105 @@ pywikibot.output(self.text[start_context:end_context]) +class Choice(StandardOption): + + """A simple choice consisting of a option, shortcut and handler.""" + + def __init__(self, option, shortcut, replacer): + """Initializer.""" + super(Choice, self).__init__(option, shortcut) + self._replacer = replacer + + @property + def replacer(self): + """The replacer.""" + return self._replacer + + def handle(self): + """Handle this choice. Must be implemented.""" + raise NotImplementedError() + + def handle_link(self): + """The current link will be handled by this choice.""" + return False + + +class StaticChoice(Choice): + + """A static choice which just returns the given value.""" + + def __init__(self, option, shortcut, result): + """Create instance with replacer set to None.""" + super(StaticChoice, self).__init__(option, shortcut, None) + self._result = result + + def handle(self): + """Return the predefined value.""" + return self._result + + +class LinkChoice(Choice): + + """A choice returning a mix of the link new and current link.""" + + def __init__(self, option, shortcut, replacer, replace_section, + replace_label): + """Initializer.""" + super(LinkChoice, self).__init__(option, shortcut, replacer) + self._section = replace_section + self._label = replace_label + + def handle(self): + """Handle by either applying the new section or label.""" + kwargs = {} + if self._section: + kwargs['section'] = self.replacer._new.section + else: + kwargs['section'] = self.replacer.current_link.section + if self._label: + if self.replacer._new.anchor is None: + kwargs['label'] = self.replacer._new.canonical_title() + if self.replacer._new.section: + kwargs['label'] += '#' + self.replacer._new.section + else: + kwargs['label'] = self.replacer._new.anchor + else: + if self.replacer.current_link.anchor is None: + kwargs['label'] = self.replacer.current_groups['title'] + if self.replacer.current_groups['section']: + kwargs['label'] += '#' + \ + self.replacer.current_groups['section'] + else: + kwargs['label'] = self.replacer.current_link.anchor + return pywikibot.Link.create_separated( + self.replacer._new.canonical_title(), self.replacer._new.site, + **kwargs) + + +class AlwaysChoice(Choice): + + """Add an option to always apply the default.""" + + def __init__(self, replacer, option='always', shortcut='a'): + """Initializer.""" + super(AlwaysChoice, self).__init__(option, shortcut, replacer) + self.always = False + + def handle(self): + """Handle the custom shortcut.""" + self.always = True + return self.answer + + def handle_link(self): + """Directly return answer whether it's applying it always.""" + return self.always + + @property + def answer(self): + """Get the actual default answer instructing the replacement.""" + return self.replacer.handle_answer(self.replacer._default) + + class IntegerOption(Option): """An option allowing a range of integers.""" -- To view, visit https://gerrit.wikimedia.org/r/511729 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If74ad55fab0d0072222945756a10dd84e88ae35a Gerrit-Change-Number: 511729 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [FIX] Blacklisted user can't get description from flinfo

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/577607 ) Change subject: [FIX] Blacklisted user can't get description from flinfo ...................................................................... [FIX] Blacklisted user can't get description from flinfo see https://tools.wmflabs.org/redpanda/flinfo/flinfo.php?id=47526502491&repo=fl… Change-Id: I436498ae9d841708c58191c863c6fb4566f1f9f2 --- M scripts/flickrripper.py 1 file changed, 5 insertions(+), 0 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/scripts/flickrripper.py b/scripts/flickrripper.py index 7de0d27..e7d4e70 100755 --- a/scripts/flickrripper.py +++ b/scripts/flickrripper.py @@ -320,6 +320,11 @@ else: filename = getFilename(photoInfo, photo_url=photoUrl) flinfoDescription = getFlinfoDescription(photo_id) + if 'Blacklisted user' in flinfoDescription: + pywikibot.warning('Blacklisted user found:\n' + + flinfoDescription) + return 0 + photoDescription = buildDescription(flinfoDescription, flickrreview, reviewer, override, addCategory, -- To view, visit https://gerrit.wikimedia.org/r/577607 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I436498ae9d841708c58191c863c6fb4566f1f9f2 Gerrit-Change-Number: 577607 Gerrit-PatchSet: 14 Gerrit-Owner: Shizhao <shizhao(a)gmail.com> Gerrit-Reviewer: Hashar <hashar(a)free.fr> Gerrit-Reviewer: Maven-release-user <maven-release-user(a)wikimedia.org> Gerrit-Reviewer: Shizhao <shizhao(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75) Gerrit-CC: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz>

4 years, 1 month

[Gerrit] pywikibot/core[master]: [IMPR] Don't trust token from NeedToken response

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/575038 ) Change subject: [IMPR] Don't trust token from NeedToken response ...................................................................... [IMPR] Don't trust token from NeedToken response Bug: T224712 Change-Id: I415d9c0d7cc3768e5780ced55ce0a75035cd6615 --- M pywikibot/data/api.py 1 file changed, 32 insertions(+), 2 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/data/api.py b/pywikibot/data/api.py index d65c260..4a31c7f 100644 --- a/pywikibot/data/api.py +++ b/pywikibot/data/api.py @@ -3143,7 +3143,8 @@ return '' elif login_result['login']['result'] == 'NeedToken': # Kept for backwards compatibility - token = login_result['login']['token'] + # Generate a new token if invalid, see T224712 + token = self.get_login_token(login_result['login']['token']) login_request['lgtoken'] = token continue elif login_result['login']['result'] == 'Throttled': @@ -3159,17 +3160,21 @@ """Ignore data; cookies are set by threadedhttp module.""" http.cookie_jar.save() - def get_login_token(self): + def get_login_token(self, old_token=None): """Fetch login token from action=query&meta=tokens. Requires MediaWiki >= 1.27. + @param old_token: old token to try + @type old_token: str @return: login token @rtype: str """ if self.site.mw_version < '1.27': raise NotImplementedError('The method get_login_token() requires ' 'at least MediaWiki version 1.27.') + if old_token and self.check_login_token(old_token) == 'valid': + return old_token login_token_request = self.site._request( use_get=False, parameters={'action': 'query', 'meta': 'tokens', 'type': 'login'}, @@ -3177,6 +3182,31 @@ login_token_result = login_token_request.submit() return login_token_result['query']['tokens'].get('logintoken') + def check_login_token(self, token): + """Check login token from action=checktoken. + + Requires MediaWiki >= 1.27. + + @param token: token to check + @tyoe token: str + @return: token validity status + @rtype: str + """ + if self.site.mw_version < '1.27': + raise NotImplementedError('The method check_login_token() ' + 'requires at least MediaWiki ' + 'version 1.27.') + check_token_request = self.site._request( + use_get=False, + parameters={ + 'action': 'checktoken', + 'type': 'login', + 'token': token, + }, + ) + check_token_result = check_token_request.submit() + return check_token_result['checktoken'].get('result') + def encode_url(query): """ -- To view, visit https://gerrit.wikimedia.org/r/575038 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I415d9c0d7cc3768e5780ced55ce0a75035cd6615 Gerrit-Change-Number: 575038 Gerrit-PatchSet: 6 Gerrit-Owner: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: D3r1ck <alangiderick(a)gmail.com> Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com> Gerrit-Reviewer: Dalba <dalba.wiki(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Framawiki <framawiki(a)tools.wmflabs.org> Gerrit-Reviewer: Huji <huji.huji(a)gmail.com> Gerrit-Reviewer: JJMC89 <JJMC89.Wikimedia(a)gmail.com> Gerrit-Reviewer: Ladsgroup <Ladsgroup(a)gmail.com> Gerrit-Reviewer: Legoktm <legoktm(a)member.fsf.org> Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com> Gerrit-Reviewer: XZise <CommodoreFabianus(a)gmx.de> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: Zhuyifei1999 <zhuyifei1999(a)gmail.com> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [FEAT] Refactor data attributes of Wikibase entities

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/526380 ) Change subject: [FEAT] Refactor data attributes of Wikibase entities ...................................................................... [FEAT] Refactor data attributes of Wikibase entities Split entity data by keys to "collections" to reduce data duplication and make it easier to support new entity types. SiteLinkCollection was the first step in this direction. Bug: T233406 Change-Id: Icbcc728c7e6f52eea27ec694e9bc7b158e127860 --- M pywikibot/bot.py M pywikibot/page/__init__.py M tests/wikibase_tests.py 3 files changed, 615 insertions(+), 354 deletions(-) Approvals: Xqt: Looks good to me, approved jenkins-bot: Verified diff --git a/pywikibot/bot.py b/pywikibot/bot.py index afafcc5..daf16d2 100644 --- a/pywikibot/bot.py +++ b/pywikibot/bot.py @@ -2017,14 +2017,14 @@ 'property ID (e.g. P123) of it:' % property_name).upper() - def user_edit_entity(self, item, data=None, + def user_edit_entity(self, entity, data=None, ignore_save_related_errors=None, ignore_server_errors=None, **kwargs): """ Edit entity with data provided, with user confirmation as required. - @param item: page to be edited - @type item: pywikibot.page.ItemPage + @param entity: page to be edited + @type entity: WikibasePage @param data: data to be saved, or None if the diff should be created automatically @param ignore_save_related_errors: Ignore save related errors and @@ -2048,10 +2048,9 @@ show_diff = kwargs.pop('show_diff', True) if show_diff: if data is None: - diff = item.toJSON(diffto=( - item._content if hasattr(item, '_content') else None)) + diff = entity.toJSON(diffto=getattr(entity, '_content', None)) else: - diff = pywikibot.page.WikibasePage._normalizeData(data) + diff = entity._normalizeData(data) pywikibot.output(json.dumps(diff, indent=4, sort_keys=True)) if 'summary' in kwargs: @@ -2060,7 +2059,7 @@ # TODO PageSaveRelatedErrors should be actually raised in editEntity # (bug T86083) return self._save_page( - item, item.editEntity, data, + entity, entity.editEntity, data, ignore_save_related_errors=ignore_save_related_errors, ignore_server_errors=ignore_server_errors, **kwargs) diff --git a/pywikibot/page/__init__.py b/pywikibot/page/__init__.py index f8b4418..43f7465 100644 --- a/pywikibot/page/__init__.py +++ b/pywikibot/page/__init__.py @@ -26,6 +26,10 @@ import unicodedata from collections import Counter, defaultdict, namedtuple, OrderedDict +try: + from collections.abc import MutableMapping +except ImportError: + from collections import MutableMapping from itertools import chain from warnings import warn @@ -3707,6 +3711,384 @@ return self.isRegistered() and 'bot' not in self.groups() +class LanguageDict(MutableMapping): + + """ + A structure holding language data for a Wikibase entity. + + Language data are mappings from a language to a string. It can be + labels, descriptions and others. + """ + + def __init__(self, data=None): + super(LanguageDict, self).__init__() + self._data = {} + if data: + self.update(data) + + @classmethod + def fromJSON(cls, data, repo=None): + this = cls({key: value['value'] for key, value in data.items()}) + return this + + def __getitem__(self, key): + key = self.normalizeKey(key) + return self._data[key] + + def __setitem__(self, key, value): + key = self.normalizeKey(key) + self._data[key] = value + + def __delitem__(self, key): + key = self.normalizeKey(key) + del self._data[key] + + def __iter__(self): + return iter(self._data) + + def __len__(self): + return len(self._data) + + def __contains__(self, key): + key = self.normalizeKey(key) + return key in self._data + + @staticmethod + def normalizeKey(key): + """ + Helper function to replace site objects with their language codes. + + @rtype: str + """ + if isinstance(key, pywikibot.site.BaseSite): + key = key.lang + return key + + @classmethod + def normalizeData(cls, data): + norm_data = {} + for key, value in data.items(): + if isinstance(value, UnicodeType): + norm_data[key] = {'language': key, 'value': value} + else: + norm_data[key] = value + return norm_data + + def toJSON(self, diffto=None): + data = {} + if diffto: + for key in diffto: + if key not in self: + data[key] = {'language': key, 'value': ''} + elif self[key] != diffto[key]['value']: + data[key] = {'language': key, 'value': self[key]} + for key in self: + if key not in diffto: + data[key] = {'language': key, 'value': self[key]} + else: + for key in self: + data[key] = {'language': key, 'value': self[key]} + return data + + +class AliasesDict(MutableMapping): + + """ + A structure holding aliases for a Wikibase entity. + + It is a mapping from a language to a list of strings. + """ + + def __init__(self, data=None): + super(AliasesDict, self).__init__() + self._data = {} + if data: + self.update(data) + + @classmethod + def fromJSON(cls, data, repo=None): + this = cls() + for key, value in data.items(): + this[key] = [val['value'] for val in value] + return this + + def __getitem__(self, key): + key = LanguageDict.normalizeKey(key) + return self._data[key] + + def __setitem__(self, key, value): + key = LanguageDict.normalizeKey(key) + self._data[key] = value + + def __delitem__(self, key): + key = LanguageDict.normalizeKey(key) + del self._data[key] + + def __iter__(self): + return iter(self._data) + + def __len__(self): + return len(self._data) + + def __contains__(self, key): + key = LanguageDict.normalizeKey(key) + return key in self._data + + @classmethod + def normalizeData(cls, data): + norm_data = {} + for key, values in data.items(): + if isinstance(values, list): + strings = [] + for value in values: + if isinstance(value, UnicodeType): + strings.append({'language': key, 'value': value}) + else: + strings.append(value) + norm_data[key] = strings + return norm_data + + def toJSON(self, diffto=None): + data = {} + if diffto: + for lang, strings in diffto.items(): + if len(self.get(lang, [])) > 0: + if tuple(sorted(val['value'] for val in strings)) != tuple( + sorted(self[lang])): + data[lang] = [{'language': lang, 'value': i} + for i in self[lang]] + else: + data[lang] = [ + {'language': lang, 'value': i['value'], 'remove': ''} + for i in strings] + else: + for lang, values in self.items(): + data[lang] = [{'language': lang, 'value': i} for i in values] + return data + + +class ClaimCollection(MutableMapping): + """A structure holding claims for a Wikibase entity.""" + + def __init__(self, repo): + super(ClaimCollection, self).__init__() + self.repo = repo + self._data = {} + + @classmethod + def fromJSON(cls, data, repo): + this = cls(repo) + for key, claims in data.items(): + this[key] = [Claim.fromJSON(repo, claim) for claim in claims] + return this + + def __getitem__(self, key): + return self._data[key] + + def __setitem__(self, key, value): + self._data[key] = value + + def __delitem__(self, key): + del self._data[key] + + def __iter__(self): + return iter(self._data) + + def __len__(self): + return len(self._data) + + def __contains__(self, key): + return key in self._data + + @classmethod + def normalizeData(cls, data): + # no normalization here, should there be? + return data + + def toJSON(self, diffto=None): + claims = {} + for prop in self: + if len(self[prop]) > 0: + claims[prop] = [claim.toJSON() for claim in self[prop]] + + if diffto: + temp = defaultdict(list) + props_add = set(claims.keys()) + props_orig = set(diffto.keys()) + for prop in (props_orig | props_add): + if prop not in props_orig: + temp[prop].extend(claims[prop]) + continue + if prop not in props_add: + temp[prop].extend( + {'id': claim['id'], 'remove': ''} + for claim in diffto[prop] if 'id' in claim) + continue + + claim_ids = set() + claim_map = { + json['id']: json for json in diffto[prop] + if 'id' in json} + for claim, json in zip(self[prop], claims[prop]): + if 'id' in json: + claim_ids.add(json['id']) + if json['id'] in claim_map: + other = Claim.fromJSON( + self.repo, claim_map[json['id']]) + if claim.same_as(other, ignore_rank=False, + ignore_refs=False): + continue + temp[prop].append(json) + + for claim in diffto[prop]: + if 'id' in claim and claim['id'] not in claim_ids: + temp[prop].append({'id': claim['id'], 'remove': ''}) + + claims = temp + + return claims + + +class SiteLinkCollection(MutableMapping): + """A structure holding SiteLinks for a Wikibase item.""" + + def __init__(self, repo, data=None): + """ + Initializer. + + @param repo: the Wikibase site on which badges are defined + @type repo: pywikibot.site.DataSite + """ + super(SiteLinkCollection, self).__init__() + self.repo = repo + self._data = {} + if data: + self.update(data) + + @staticmethod + def getdbName(site): + """ + Helper function to obtain a dbName for a Site. + + @param site: The site to look up. + @type site: pywikibot.site.BaseSite or str + """ + if isinstance(site, pywikibot.site.BaseSite): + return site.dbName() + return site + + def __getitem__(self, key): + """ + Get the SiteLink with the given key. + + @param key: site key as Site instance or db key + @type key: pywikibot.Site or str + @rtype: SiteLink + """ + key = self.getdbName(key) + return self._data[key] + + def __setitem__(self, key, val): + """ + Set the SiteLink for a given key. + + @param key: site key as Site instance or db key + @type key: pywikibot.Site or str + @param val: page name as a string or JSON containing SiteLink data + @type val: dict or str + @rtype: SiteLink + """ + if isinstance(val, UnicodeType): + val = SiteLink(val, key) + else: + val = SiteLink.fromJSON(val, self.repo) + key = self.getdbName(key) + self._data[key] = val + + def __delitem__(self, key): + key = self.getdbName(key) + del self._data[key] + + def __iter__(self): + return iter(self._data) + + def __len__(self): + return len(self._data) + + def __contains__(self, key): + key = self.getdbName(key) + return key in self._data + + @classmethod + def fromJSON(cls, data, repo): + """Construct a new SiteLinkCollection from JSON.""" + return cls(repo, data) + + @classmethod + def normalizeData(cls, data): + """ + Helper function to expand data into the Wikibase API structure. + + @param data: Data to normalize + @type data: list + + @return: the altered dict from parameter data. + @rtype: dict + """ + norm_data = {} + for obj in data: + if isinstance(obj, Page): + db_name = obj.site.dbName() + norm_data[db_name] = {'site': db_name, 'title': obj.title()} + else: + # TODO: Do some verification here + db_name = obj['site'] + norm_data[db_name] = obj + return norm_data + + def toJSON(self, diffto=None): + """ + Create JSON suitable for Wikibase API. + + When diffto is provided, JSON representing differences + to the provided data is created. + + @param diffto: JSON containing entity data + @type diffto: dict + + @rtype: dict + """ + data = {dbname: sitelink.toJSON() + for (dbname, sitelink) in self.items()} + if diffto: + to_nuke = [] + for dbname, sitelink in data.items(): + if dbname in diffto: + diffto_link = diffto[dbname] + if diffto_link.get('title') == sitelink.get('title'): + # compare badges + tmp_badges = [] + diffto_badges = diffto_link.get('badges', []) + badges = sitelink.get('badges', []) + for badge in set(diffto_badges) - set(badges): + tmp_badges.append('') + for badge in set(badges) - set(diffto_badges): + tmp_badges.append(badge) + if tmp_badges: + data[dbname]['badges'] = tmp_badges + else: + to_nuke.append(dbname) + # find removed sitelinks + for dbname in (set(diffto.keys()) - set(self.keys())): + badges = [''] * len(diffto[dbname].get('badges', [])) + data[dbname] = {'site': dbname, 'title': ''} + if badges: + data[dbname]['badges'] = badges + for dbname in to_nuke: + del data[dbname] + return data + + class WikibaseEntity(object): """ @@ -3714,8 +4096,22 @@ Each entity is identified by a data repository it belongs to and an identifier. + + @cvar DATA_ATTRIBUTES: dictionary which maps data attributes (eg. 'labels', + 'claims') to appropriate collection classes (eg. LanguageDict, + ClaimsCollection) + @type DATA_ATTRIBUTES: dict + + @cvar entity_type: entity type identifier + @type entity_type: str + + @cvar title_pattern: regular expression which matches all possible + entity ids for this entity type + @type title_pattern: str """ + DATA_ATTRIBUTES = {} + def __init__(self, repo, id_=None): """ Initializer. @@ -3810,7 +4206,18 @@ @rtype: dict """ - return {} + data = {} + for key in self.DATA_ATTRIBUTES: + attr = getattr(self, key, None) + if attr is None: + continue + if diffto: + value = attr.toJSON(diffto=diffto.get(key)) + else: + value = attr.toJSON() + if value: + data[key] = value + return data @classmethod def _normalizeData(cls, data): @@ -3823,7 +4230,11 @@ @return: the altered dict from parameter data. @rtype: dict """ - return {} + norm_data = {} + for key, attr in cls.DATA_ATTRIBUTES.items(): + if key in data: + norm_data[key] = attr.normalizeData(data[key]) + return norm_data def exists(self): """ @@ -3865,7 +4276,13 @@ self._content = content if 'missing' in self._content: raise pywikibot.NoWikibaseEntity(self) - return {} + + data = {} + for key, cls in self.DATA_ATTRIBUTES.items(): + value = cls.fromJSON(self._content.get(key, {}), self.repo) + setattr(self, key, value) + data[key] = value + return data def editEntity(self, data=None, **kwargs): """ @@ -3903,14 +4320,12 @@ class WikibasePage(BasePage, WikibaseEntity): """ - The base page for the Wikibase extension. + Mixin base class for Wikibase entities which are also pages (eg. items). There should be no need to instantiate this directly. """ - _cache_attrs = BasePage._cache_attrs + ( - '_content', 'labels', 'descriptions', 'aliases', 'claims', - ) + _cache_attrs = BasePage._cache_attrs + ('_content', ) def __init__(self, site, title='', **kwargs): """ @@ -4110,137 +4525,6 @@ if 'pageid' in self._content: self._pageid = self._content['pageid'] - # labels - self.labels = {} - for lang in self._content.get('labels', {}): - if 'removed' not in self._content['labels'][lang]: # T56767 - self.labels[lang] = self._content['labels'][lang]['value'] - - # descriptions - self.descriptions = {} - for lang in self._content.get('descriptions', {}): - self.descriptions[lang] = self._content[ - 'descriptions'][lang]['value'] - - # aliases - self.aliases = {} - for lang in self._content.get('aliases', {}): - self.aliases[lang] = [] - for value in self._content['aliases'][lang]: - self.aliases[lang].append(value['value']) - - # claims - self.claims = {} - for pid in self._content.get('claims', {}): - self.claims[pid] = [] - for claim in self._content['claims'][pid]: - c = Claim.fromJSON(self.repo, claim) - c.on_item = self - self.claims[pid].append(c) - - data['labels'] = self.labels - data['descriptions'] = self.descriptions - data['aliases'] = self.aliases - data['claims'] = self.claims - return data - - def _diff_to(self, type_key, key_name, value_name, diffto, data): - assert type_key not in data, 'Key type must be defined in data' - source = self._normalizeLanguages(getattr(self, type_key)).copy() - diffto = {} if not diffto else diffto.get(type_key, {}) - new = set(source.keys()) - for key in diffto: - if key in new: - if source[key] == diffto[key][value_name]: - del source[key] - else: - source[key] = '' - for key, value in source.items(): - source[key] = {key_name: key, value_name: value} - if source: - data[type_key] = source - - def toJSON(self, diffto=None): - """ - Create JSON suitable for Wikibase API. - - When diffto is provided, JSON representing differences - to the provided data is created. - - @param diffto: JSON containing entity data - @type diffto: dict - - @rtype: dict - """ - data = {} - self._diff_to('labels', 'language', 'value', diffto, data) - - self._diff_to('descriptions', 'language', 'value', diffto, data) - - aliases = self._normalizeLanguages(self.aliases).copy() - if diffto: - for lang, strings in diffto.get('aliases', {}).items(): - if len(aliases.get(lang, [])) > 0: - if tuple(sorted(val['value'] for val in strings)) != tuple( - sorted(aliases[lang])): - aliases[lang] = [{'language': lang, 'value': i} - for i in aliases[lang]] - else: - del aliases[lang] - else: - aliases[lang] = [ - {'language': lang, 'value': i['value'], 'remove': ''} - for i in strings] - else: - for lang, values in aliases.items(): - aliases[lang] = [{'language': lang, 'value': i} - for i in values] - - if aliases: - data['aliases'] = aliases - - claims = {} - for prop in self.claims: - if len(self.claims[prop]) > 0: - claims[prop] = [claim.toJSON() for claim in self.claims[prop]] - - if diffto and 'claims' in diffto: - temp = defaultdict(list) - props_add = set(claims.keys()) - props_orig = set(diffto['claims'].keys()) - for prop in (props_orig | props_add): - if prop not in props_orig: - temp[prop].extend(claims[prop]) - continue - if prop not in props_add: - temp[prop].extend( - {'id': claim['id'], 'remove': ''} - for claim in diffto['claims'][prop] if 'id' in claim) - continue - - claim_ids = set() - claim_map = { - json['id']: json for json in diffto['claims'][prop] - if 'id' in json} - for claim, json in zip(self.claims[prop], claims[prop]): - if 'id' in json: - claim_ids.add(json['id']) - if json['id'] in claim_map: - other = Claim.fromJSON( - self.repo, claim_map[json['id']]) - if claim.same_as(other, ignore_rank=False, - ignore_refs=False): - continue - temp[prop].append(json) - - for claim in diffto['claims'][prop]: - if 'id' in claim and claim['id'] not in claim_ids: - temp[prop].append({'id': claim['id'], 'remove': ''}) - - claims = temp - - if claims: - data['claims'] = claims return data @property @@ -4262,68 +4546,6 @@ def latest_revision_id(self): self.clear_cache() - @staticmethod - def _normalizeLanguages(data): - """ - Helper function to replace site objects with their language codes. - - @param data: The dict to normalize. - @type data: dict - - @return: the altered dict from parameter data. - @rtype: dict - """ - for key in list(data): - if isinstance(key, pywikibot.site.BaseSite): - data[key.lang] = data[key] - del data[key] - return data - - @classmethod - def _normalizeData(cls, data): - """ - Helper function to expand data into the Wikibase API structure. - - @param data: The dict to normalize - @type data: dict - - @return: the altered dict from parameter data. - @rtype: dict - """ - WikibaseEntity._normalizeData(data) - for prop in ('labels', 'descriptions'): - if prop not in data: - continue - data[prop] = cls._normalizeLanguages(data[prop]) - for key, value in data[prop].items(): - if isinstance(value, UnicodeType): - data[prop][key] = {'language': key, 'value': value} - - if 'aliases' in data: - data['aliases'] = cls._normalizeLanguages(data['aliases']) - for key, values in data['aliases'].items(): - if isinstance(values, list): - strings = [] - for value in values: - if isinstance(value, UnicodeType): - strings.append({'language': key, 'value': value}) - else: - strings.append(value) - data['aliases'][key] = strings - - return data - - def getdbName(self, site): - """ - Helper function to obtain a dbName for a Site. - - @param site: The site to look up. - @type site: Site - """ - if isinstance(site, pywikibot.site.BaseSite): - return site.dbName() - return site - @allow_asynchronous def editEntity(self, data=None, **kwargs): """ @@ -4462,9 +4684,17 @@ been looked up, the item is then defined by the qid. """ - _cache_attrs = WikibasePage._cache_attrs + ('sitelinks',) + _cache_attrs = WikibasePage._cache_attrs + ( + 'labels', 'descriptions', 'aliases', 'claims', 'sitelinks') entity_type = 'item' title_pattern = r'Q[1-9]\d*' + DATA_ATTRIBUTES = { + 'labels': LanguageDict, + 'descriptions': LanguageDict, + 'aliases': AliasesDict, + 'claims': ClaimCollection, + 'sitelinks': SiteLinkCollection, + } def __init__(self, site, title=None, ns=None): """ @@ -4472,7 +4702,7 @@ @param site: data repository @type site: pywikibot.site.DataSite - @param title: id number of item, "Q###", + @param title: identifier of item, "Q###", -1 or None for an empty item. @type title: str @type ns: namespace @@ -4689,12 +4919,6 @@ if self.isRedirectPage() and not get_redirect: raise pywikibot.IsRedirectPage(self) - # sitelinks - self.sitelinks = SiteLinkCollection(self.site) - for dbname in self._content.get('sitelinks', {}): - self.sitelinks[dbname] = self._content['sitelinks'][dbname] - - data['sitelinks'] = self.sitelinks return data def getRedirectTarget(self): @@ -4707,61 +4931,6 @@ (self, target, cmodel)) return self.__class__(target.site, target.title(), target.namespace()) - def toJSON(self, diffto=None): - """ - Create JSON suitable for Wikibase API. - - When diffto is provided, JSON representing differences - to the provided data is created. - - @param diffto: JSON containing entity data - @type diffto: dict - - @rtype: dict - """ - data = super(ItemPage, self).toJSON(diffto=diffto) - - sitelinks = {dbname: sitelink.toJSON() - for (dbname, sitelink) in self.sitelinks.items()} - - if diffto and 'sitelinks' in diffto: - to_nuke = [] - diffto_sitelinks = diffto['sitelinks'] - - for dbname, sitelink in sitelinks.items(): - if dbname in diffto_sitelinks: - diffto_link = diffto_sitelinks[dbname] - if diffto_link.get('title') == sitelink.get('title'): - # compare badges - tmp_badges = [] - diffto_badges = diffto_link.get('badges', []) - badges = sitelink.get('badges', []) - for badge in set(diffto_badges) - set(badges): - tmp_badges.append('') - for badge in set(badges) - set(diffto_badges): - tmp_badges.append(badge) - - if tmp_badges: - sitelinks[dbname]['badges'] = tmp_badges - else: - to_nuke.append(dbname) - - # find removed sitelinks - for dbname in (set(diffto_sitelinks.keys()) - - set(sitelinks.keys())): - badges = [''] * len(diffto_sitelinks[dbname].get('badges', [])) - sitelinks[dbname] = {'site': dbname, 'title': ''} - if badges: - sitelinks[dbname]['badges'] = badges - - for dbname in to_nuke: - del sitelinks[dbname] - - if sitelinks: - data['sitelinks'] = sitelinks - - return data - def iterlinks(self, family=None): """ Iterate through all the sitelinks. @@ -4796,11 +4965,10 @@ """ if force or not hasattr(self, '_content'): self.get(force=force) - dbname = self.getdbName(site) - if dbname not in self.sitelinks: + if site not in self.sitelinks: raise pywikibot.NoPage(self) else: - return self.sitelinks[dbname].canonical_title() + return self.sitelinks[site].canonical_title() def setSitelink(self, sitelink, **kwargs): """ @@ -4828,7 +4996,7 @@ """ data = [] for site in sites: - site = self.getdbName(site) + site = SiteLinkCollection.getdbName(site) data.append({'site': site, 'title': ''}) self.setSitelinks(data, **kwargs) @@ -4840,17 +5008,7 @@ list can either be a Page object, or a dict with a value for 'site' and 'title'. """ - data = {} - # todo: move to _normalizeData - for obj in sitelinks: - if isinstance(obj, Page): - db_name = self.getdbName(obj.site) - data[db_name] = {'site': db_name, 'title': obj.title()} - else: - # TODO: Do some verification here - db_name = obj['site'] - data[db_name] = obj - data = {'sitelinks': data} + data = {'sitelinks': sitelinks} self.editEntity(data, **kwargs) def mergeInto(self, item, **kwargs): @@ -5022,9 +5180,16 @@ PropertyPage(DataSite, datatype='url') """ - _cache_attrs = WikibasePage._cache_attrs + ('_type',) + _cache_attrs = WikibasePage._cache_attrs + ( + '_type', 'labels', 'descriptions', 'aliases', 'claims') entity_type = 'property' title_pattern = r'P[1-9]\d*' + DATA_ATTRIBUTES = { + 'labels': LanguageDict, + 'descriptions': LanguageDict, + 'aliases': AliasesDict, + 'claims': ClaimCollection, + } def __init__(self, source, title=None, datatype=None): """ @@ -5032,7 +5197,7 @@ @param source: data repository property is on @type source: pywikibot.site.DataSite - @param title: page name of property, like "P##", + @param title: identifier of property, like "P##", "-1" or None for an empty property. @type title: str @param datatype: Datatype for a new property. @@ -6608,50 +6773,6 @@ return json -class SiteLinkCollection(dict): - """A structure holding SiteLinks for a Wikibase item.""" - - def __init__(self, repo, *args): - """ - Initializer. - - @param repo: the Wikibase site on which badges are defined - @type repo: pywikibot.site.DataSite - """ - super(SiteLinkCollection, self).__init__(*args) - self.repo = repo - - def __getitem__(self, key): - """ - Get the SiteLink with the given key. - - @param key: site key as Site instance or db key - @type key: pywikibot.page.Site or str - @rtype: pywikibot.page.SiteLink - """ - if isinstance(key, pywikibot.site.BaseSite): - key = key.dbName() - return super(SiteLinkCollection, self).__getitem__(key) - - def __setitem__(self, key, val): - """ - Set the SiteLink for a given key. - - @param key: site key as Site instance or db key - @type key: pywikibot.Site or str - @param val: page name as a string or JSON containing SiteLink data - @type val: dict or str - @rtype: pywikibot.page.SiteLink - """ - if isinstance(val, UnicodeType): - val = SiteLink(val, key) - else: - val = SiteLink.fromJSON(val, self.repo) - if isinstance(key, pywikibot.site.BaseSite): - key = key.dbName() - return super(SiteLinkCollection, self).__setitem__(key, val) - - # Utility functions for parsing page titles # This regular expression will match any decimal and hexadecimal entity and diff --git a/tests/wikibase_tests.py b/tests/wikibase_tests.py index 768c42b..f3669a1 100644 --- a/tests/wikibase_tests.py +++ b/tests/wikibase_tests.py @@ -15,7 +15,9 @@ import pywikibot from pywikibot import pagegenerators -from pywikibot.page import WikibasePage, ItemPage, PropertyPage, Page +from pywikibot.page import ( + WikibasePage, ItemPage, PropertyPage, Page, LanguageDict, AliasesDict, +) from pywikibot.site import Namespace, NamespacesDict from pywikibot.tools import MediaWikiVersion, suppress_warnings @@ -1760,13 +1762,9 @@ self.assertLength(wvlinks, 2) -class TestWriteNormalizeLang(TestCase): +class TestLanguageDict(TestCase): - """Test cases for routines that normalize languages in a dict. - - Exercises WikibasePage._normalizeLanguages with data that is - not normalized and data which is already normalized. - """ + """Test cases covering LanguageDict methods.""" family = 'wikipedia' code = 'en' @@ -1775,29 +1773,170 @@ def setUp(self): """Setup tests.""" - super(TestWriteNormalizeLang, self).setUp() + super(TestLanguageDict, self).setUp() self.site = self.get_site() self.lang_out = {'en': 'foo'} - def test_normalize_lang(self): - """Test _normalizeLanguages() method.""" - lang_in = {self.site: 'foo'} + def test_init(self): + """Test LanguageDict initializer.""" + ld = LanguageDict() + self.assertLength(ld, 0) + ld = LanguageDict(self.lang_out) + self.assertLength(ld, 1) - response = WikibasePage._normalizeLanguages(lang_in) - self.assertEqual(response, self.lang_out) + def test_setitem(self): + """Test LanguageDict.__setitem__ metamethod.""" + ld = LanguageDict(self.lang_out) + self.assertIn('en', ld) + ld[self.site] = 'bar' + self.assertIn('en', ld) - def test_normalized_lang(self): - """Test _normalizeData() method.""" - response = WikibasePage._normalizeData( - copy.deepcopy(self.lang_out)) - self.assertEqual(response, self.lang_out) + def test_getitem(self): + """Test LanguageDict.__getitem__ metamethod.""" + ld = LanguageDict(self.lang_out) + self.assertEqual(ld['en'], 'foo') + self.assertEqual(ld[self.site], 'foo') + self.assertIsNone(ld.get('de')) + + def test_delitem(self): + """Test LanguageDict.__delitem__ metamethod.""" + ld = LanguageDict(self.lang_out) + ld.pop(self.site) + self.assertNotIn('en', ld) + self.assertLength(ld, 0) + + def test_fromJSON(self): + """Test LanguageDict.fromJSON method.""" + ld = LanguageDict.fromJSON({'en': {'language': 'en', 'value': 'foo'}}) + self.assertIsInstance(ld, LanguageDict) + self.assertEqual(ld, LanguageDict(self.lang_out)) + + def test_toJSON(self): + """Test LanguageDict.toJSON method.""" + ld = LanguageDict() + self.assertEqual(ld.toJSON(), {}) + ld = LanguageDict(self.lang_out) + self.assertEqual( + ld.toJSON(), {'en': {'language': 'en', 'value': 'foo'}}) + + def test_toJSON_diffto(self): + """Test LanguageDict.toJSON method.""" + ld = LanguageDict({'de': 'foo'}) + diffto = { + 'de': {'language': 'de', 'value': 'bar'}, + 'en': {'language': 'en', 'value': 'foo'}} + self.assertEqual( + ld.toJSON(diffto=diffto), + {'de': {'language': 'de', 'value': 'foo'}, + 'en': {'language': 'en', 'value': ''}}) + + def test_normalizeData(self): + """Test LanguageDict.normalizeData method.""" + self.assertEqual( + LanguageDict.normalizeData(self.lang_out), + {'en': {'language': 'en', 'value': 'foo'}}) + + +class TestAliasesDict(TestCase): + + """Test cases covering AliasesDict methods.""" + + family = 'wikipedia' + code = 'en' + + dry = True + + def setUp(self): + """Setup tests.""" + super(TestAliasesDict, self).setUp() + self.site = self.get_site() + self.lang_out = {'en': ['foo', 'bar']} + + def test_init(self): + """Test AliasesDict initializer.""" + ad = AliasesDict() + self.assertLength(ad, 0) + ad = AliasesDict(self.lang_out) + self.assertLength(ad, 1) + + def test_setitem(self): + """Test AliasesDict.__setitem__ metamethod.""" + ad = AliasesDict(self.lang_out) + self.assertIn('en', ad) + ad[self.site] = ['baz'] + self.assertIn('en', ad) + + def test_getitem(self): + """Test AliasesDict.__getitem__ metamethod.""" + ad = AliasesDict(self.lang_out) + self.assertEqual(ad['en'], ['foo', 'bar']) + self.assertEqual(ad[self.site], ['foo', 'bar']) + self.assertIsNone(ad.get('de')) + + def test_delitem(self): + """Test AliasesDict.__delitem__ metamethod.""" + ad = AliasesDict(self.lang_out) + ad.pop(self.site) + self.assertNotIn('en', ad) + self.assertLength(ad, 0) + + def test_fromJSON(self): + """Test AliasesDict.fromJSON method.""" + ad = AliasesDict.fromJSON({'en': [ + {'language': 'en', 'value': 'foo'}, + {'language': 'en', 'value': 'bar'}]}) + self.assertIsInstance(ad, AliasesDict) + self.assertEqual(ad, AliasesDict(self.lang_out)) + + def test_toJSON(self): + """Test AliasesDict.toJSON method.""" + ad = AliasesDict() + self.assertEqual(ad.toJSON(), {}) + ad = AliasesDict(self.lang_out) + self.assertEqual(ad.toJSON(), {'en': [ + {'language': 'en', 'value': 'foo'}, + {'language': 'en', 'value': 'bar'}, + ]}) + + def test_toJSON_diffto(self): + """Test AliasesDict.toJSON method.""" + ad = AliasesDict(self.lang_out) + diffto = { + 'de': [ + {'language': 'de', 'value': 'foo'}, + {'language': 'de', 'value': 'bar'}, + ], + 'en': [ + {'language': 'en', 'value': 'foo'}, + {'language': 'en', 'value': 'baz'}, + ]} + self.assertEqual( + ad.toJSON(diffto=diffto), + {'de': [{'language': 'de', 'value': 'foo', 'remove': ''}, + {'language': 'de', 'value': 'bar', 'remove': ''}], + 'en': [{'language': 'en', 'value': 'foo'}, + {'language': 'en', 'value': 'bar'}]}) + + def test_normalizeData(self): + """Test AliasesDict.normalizeData method.""" + data_in = {'en': [ + {'language': 'en', 'value': 'foo'}, + 'bar', + {'language': 'en', 'value': 'baz', 'remove': ''}, + ]} + data_out = {'en': [ + {'language': 'en', 'value': 'foo'}, + {'language': 'en', 'value': 'bar'}, + {'language': 'en', 'value': 'baz', 'remove': ''}, + ]} + self.assertEqual(AliasesDict.normalizeData(data_in), data_out) class TestWriteNormalizeData(TestCase): """Test cases for routines that normalize data for writing to Wikidata. - Exercises WikibasePage._normalizeData with data that is not normalized + Exercises ItemPage._normalizeData with data that is not normalized and data which is already normalized. """ @@ -1807,29 +1946,31 @@ """Setup tests.""" super(TestWriteNormalizeData, self).setUp() self.data_out = { + 'labels': {'en': {'language': 'en', 'value': 'Foo'}}, + 'descriptions': {'en': {'language': 'en', 'value': 'Desc'}}, 'aliases': {'en': [ {'language': 'en', 'value': 'Bah'}, {'language': 'en', 'value': 'Bar', 'remove': ''}, ]}, - 'labels': {'en': {'language': 'en', 'value': 'Foo'}}, } def test_normalize_data(self): """Test _normalizeData() method.""" data_in = { + 'labels': {'en': 'Foo'}, + 'descriptions': {'en': 'Desc'}, 'aliases': {'en': [ 'Bah', {'language': 'en', 'value': 'Bar', 'remove': ''}, ]}, - 'labels': {'en': 'Foo'}, } - response = WikibasePage._normalizeData(data_in) + response = ItemPage._normalizeData(data_in) self.assertEqual(response, self.data_out) def test_normalized_data(self): """Test _normalizeData() method for normalized data.""" - response = WikibasePage._normalizeData( + response = ItemPage._normalizeData( copy.deepcopy(self.data_out)) self.assertEqual(response, self.data_out) -- To view, visit https://gerrit.wikimedia.org/r/526380 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Icbcc728c7e6f52eea27ec694e9bc7b158e127860 Gerrit-Change-Number: 526380 Gerrit-PatchSet: 18 Gerrit-Owner: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com> Gerrit-Reviewer: Lokal Profil <andre.costa(a)wikimedia.se> Gerrit-Reviewer: Matěj Suchánek <matejsuchanek97(a)gmail.com> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75) Gerrit-CC: Dvorapa <dvorapa(a)seznam.cz>

4 years, 1 month

[Gerrit] pywikibot/core[master]: [IMPR] Use find_namespace_packages with Python 3

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583564 ) Change subject: [IMPR] Use find_namespace_packages with Python 3 ...................................................................... [IMPR] Use find_namespace_packages with Python 3 Use find_namespace_packages in favour of find_packages with Python 3 Change-Id: I5897ccfd17d461c1fd334b722278d6dc51a2b619 --- M setup.py 1 file changed, 14 insertions(+), 4 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/setup.py b/setup.py index af8b022..049c2a8 100644 --- a/setup.py +++ b/setup.py @@ -11,7 +11,7 @@ import os import sys -from setuptools import find_packages, setup +from setuptools import setup PYTHON_VERSION = sys.version_info[:3] PY2 = (PYTHON_VERSION[0] == 2) @@ -210,6 +210,18 @@ return ''.join(desc) +def get_packages(name): + """Find framework packages.""" + if PY2: + from setuptools import find_packages + packages = [package for package in find_packages() + if package.startswith(name + '.')] + else: + from setuptools import find_namespace_packages + packages = find_namespace_packages(include=[name + '.*']) + return [str(name)] + packages + + def main(): """Setup entry point.""" name = 'pywikibot' @@ -224,9 +236,7 @@ maintainer='The Pywikibot team', maintainer_email='pywikibot(a)lists.wikimedia.org', license='MIT License', - packages=[str(name)] + [package - for package in find_packages() - if package.startswith('pywikibot.')], + packages=get_packages(name), python_requires='>=2.7.4, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*', install_requires=dependencies, extras_require=extra_deps, -- To view, visit https://gerrit.wikimedia.org/r/583564 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5897ccfd17d461c1fd334b722278d6dc51a2b619 Gerrit-Change-Number: 583564 Gerrit-PatchSet: 2 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [cleanup] Remove notes which are obsolete

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583349 ) Change subject: [cleanup] Remove notes which are obsolete ...................................................................... [cleanup] Remove notes which are obsolete Change-Id: I726f4cabc92501aba101a26fd5147f2bca348156 --- M setup.py 1 file changed, 0 insertions(+), 2 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/setup.py b/setup.py index af8b022..66651df 100644 --- a/setup.py +++ b/setup.py @@ -99,8 +99,6 @@ # ------- setup install_requires ------- # # packages which are mandatory -# version.package_version() uses pathlib which is a python 3 library. -# pathlib2 is required for python 2.7 dependencies = ['requests>=2.20.1,<2.22.0; python_version == "3.4"', 'requests>=2.20.1; python_version != "3.4"', 'enum34>=1.1.6,!=1.1.8; python_version < "3"', -- To view, visit https://gerrit.wikimedia.org/r/583349 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I726f4cabc92501aba101a26fd5147f2bca348156 Gerrit-Change-Number: 583349 Gerrit-PatchSet: 1 Gerrit-Owner: Xqt <info(a)gno.de> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

[Gerrit] pywikibot/core[master]: [IMPR] Prefer userscripts over scripts

by jenkins-bot (Code Review)

jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/583362 ) Change subject: [IMPR] Prefer userscripts over scripts ...................................................................... [IMPR] Prefer userscripts over scripts Add userscripts at first position in script_paths and update documentation in config2.py Bug: T248374 Change-Id: I1b37e1aa0033772ea450a6c1fbbfa7d73252d6a1 --- M pwb.py M pywikibot/config2.py 2 files changed, 6 insertions(+), 6 deletions(-) Approvals: Dvorapa: Looks good to me, approved jenkins-bot: Verified diff --git a/pwb.py b/pwb.py index 3d1c8a5..d4dc86d 100755 --- a/pwb.py +++ b/pwb.py @@ -282,9 +282,9 @@ """Search for the filename in the given script paths.""" from pywikibot import config - script_paths = ['scripts', - 'scripts.maintenance', - 'scripts.userscripts'] + script_paths = ['scripts.userscripts', + 'scripts', + 'scripts.maintenance'] if config.user_script_paths: if isinstance(config.user_script_paths, list): diff --git a/pywikibot/config2.py b/pywikibot/config2.py index da754de..98c489c 100644 --- a/pywikibot/config2.py +++ b/pywikibot/config2.py @@ -558,9 +558,9 @@ # # The search order is # 1. user_script_paths in the given order -# 2. scripts -# 3. scripts/maintenance -# 4. scripts/archive +# 2. scripts/userscripts +# 3. scripts +# 4. scripts/maintenance # # sample: # user_script_paths = ['scripts.myscripts'] -- To view, visit https://gerrit.wikimedia.org/r/583362 To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings Gerrit-Project: pywikibot/core Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1b37e1aa0033772ea450a6c1fbbfa7d73252d6a1 Gerrit-Change-Number: 583362 Gerrit-PatchSet: 3 Gerrit-Owner: Mh-3110 <bipi3110(a)gmail.com> Gerrit-Reviewer: Dvorapa <dvorapa(a)seznam.cz> Gerrit-Reviewer: Xqt <info(a)gno.de> Gerrit-Reviewer: jenkins-bot (75)

4 years, 1 month

← Newer
1
2
3
4
5
6
7
...
13
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

Pywikibot-commits March 2020