jenkins-bot has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/772400 )
Change subject: [bugfix] Solve UnicodeDecodeError in reflinks.py
......................................................................
[bugfix] Solve UnicodeDecodeError in reflinks.py
- add a new staticmethod 'charset' to extract a charset
- try to extract a charset from meta content
- try any charset including site.encodings to decode the meta content
Bug: T304288
Change-Id: I3d9b534197fedb4f0a4fa1c0980be5d9369e319e
---
M scripts/reflinks.py
1 file changed, 30 insertions(+), 11 deletions(-)
Approvals:
Rubin: Looks good to me, but someone else must approve
Xqt: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/reflinks.py b/scripts/reflinks.py
index 6e7aba8..b15f2e5 100755
--- a/scripts/reflinks.py
+++ b/scripts/reflinks.py
@@ -42,7 +42,7 @@
¶ms;
"""
-# (C) Pywikibot team, 2008-2021
+# (C) Pywikibot team, 2008-2022
#
# Distributed under the terms of the MIT license.
#
@@ -58,10 +58,11 @@
from functools import partial
from http import HTTPStatus
from textwrap import shorten
+from typing import Optional
import pywikibot
from pywikibot import comms, config, i18n, pagegenerators, textlib
-from pywikibot.backports import removeprefix
+from pywikibot.backports import Match, removeprefix
from pywikibot.bot import (
ConfigParserBot,
ExistingPageBot,
@@ -553,6 +554,19 @@
return True
return super().skip_page(page)
+ @staticmethod
+ def charset(enc: Match) -> Optional[str]:
+ """Find an encoding type."""
+ if enc:
+ # Use encoding if found. Else use chardet apparent encoding
+ encoding = enc.group('enc').strip('"\' ').lower()
+ naked = re.sub(r'[ _\-]', '', encoding)
+ # Convert to python correct encoding names
+ if naked == 'xeucjp':
+ encoding = 'euc_jp'
+ return encoding
+ return None
+
def treat(self, page) -> None:
"""Process one page."""
# Load the page's text from the wiki
@@ -659,21 +673,26 @@
if content_type:
# use charset from http header
s = self.CHARSET.search(content_type)
+
if meta_content:
- tag = meta_content.group().decode()
+ tag = None
+ encoding = self.charset(s)
+ encodings = [encoding] if encoding else []
+ encodings += list(page.site.encodings())
+ for enc in encodings:
+ with suppress(UnicodeDecodeError):
+ tag = meta_content.group().decode(enc)
+ break
+
# Prefer the contentType from the HTTP header :
- if not content_type:
+ if not content_type and tag:
content_type = tag
if not s:
# use charset from html
s = self.CHARSET.search(tag)
- if s:
- # Use encoding if found. Else use chardet apparent encoding
- encoding = s.group('enc').strip('"\' ').lower()
- naked = re.sub(r'[ _\-]', '', encoding)
- # Convert to python correct encoding names
- if naked == 'xeucjp':
- encoding = 'euc_jp'
+
+ encoding = self.charset(s)
+ if encoding:
r.encoding = encoding
if not content_type:
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/772400
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: I3d9b534197fedb4f0a4fa1c0980be5d9369e319e
Gerrit-Change-Number: 772400
Gerrit-PatchSet: 1
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: D3r1ck01 <xsavitar.wiki(a)aol.com>
Gerrit-Reviewer: Rubin <rubin.happy(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged
Xqt has submitted this change. ( https://gerrit.wikimedia.org/r/c/pywikibot/core/+/772441 )
Change subject: [IMPR] Add links to Python Enhancement Proposals in documentations
......................................................................
[IMPR] Add links to Python Enhancement Proposals in documentations
Change-Id: Ic9572c680de359848494eefe74303b147e21af34
---
M HISTORY.rst
M pywikibot/pagegenerators.py
2 files changed, 5 insertions(+), 6 deletions(-)
Approvals:
jenkins-bot: Verified
Xqt: Looks good to me, approved
diff --git a/HISTORY.rst b/HISTORY.rst
index b1c3e5f..d578d5c 100644
--- a/HISTORY.rst
+++ b/HISTORY.rst
@@ -280,9 +280,9 @@
* pagenenerators.handleArg() method was renamed to handle_arg() (:phab:`T271437`)
* CategoryGenerator, FileGenerator, ImageGenerator and ReferringPageGenerator pagegenerator functions were removed
* Family.ignore_certificate_error() method was removed in favour of verify_SSL_certificate (:phab:`T265205`)
-* tools.is_IP was renamed to is_ip_address due to PEP8
+* tools.is_IP was renamed to is_ip_address due to :pep:`8`
* config2.py was renamed to config.py
-* Exceptions were renamed having a suffix "Error" due to PEP8 (:phab:`T280227`)
+* Exceptions were renamed having a suffix "Error" due to :pep:`8` (:phab:`T280227`)
6.1.0
@@ -339,7 +339,7 @@
~~~~~~~~~~~~~~~~
* interwiki_graph module was removed (:phab:`T223826`)
-* Require setuptools >= 20.2 due to PEP 440
+* Require setuptools >= 20.2 due to :pep:`440`
* Support of MediaWiki < 1.23 has been dropped (:phab:`T268979`)
* APISite.loadimageinfo will no longer return any content
* Return requests.Response with http.request() instead of plain text (:phab:`T265206`)
@@ -995,7 +995,7 @@
* BaseBot has new methods setup and teardown
* UploadBot got a filename prefix parameter (:phab:`T170123`)
* cosmetic_changes is able to remove empty sections (:phab:`T140570`)
-* Pywikibot is following PEP 396 versioning
+* Pywikibot is following :pep:`396` versioning
* pagegenerators AllpagesPageGenerator, CombinedPageGenerator, UnconnectedPageGenerator are deprecated
* Some DayPageGenerator parameters has been renamed
* unicodedata2, httpbin and Flask dependency was removed (:phab:`T102461`, :phab:`T108068`, :phab:`T178864`, :phab:`T193383`)
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index f8f1b26..0926580 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -1,8 +1,7 @@
"""
This module offers a wide variety of page generators.
-A page generator is an object that is iterable (see
-`PEP 255 <https://legacy.python.org/dev/peps/pep-0255/>`_) and
+A page generator is an object that is iterable (see :pep:`255`) and
that yields page objects on which other scripts can then work.
Pagegenerators.py cannot be run as script. For testing purposes listpages.py
--
To view, visit https://gerrit.wikimedia.org/r/c/pywikibot/core/+/772441
To unsubscribe, or for help writing mail filters, visit https://gerrit.wikimedia.org/r/settings
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Change-Id: Ic9572c680de359848494eefe74303b147e21af34
Gerrit-Change-Number: 772441
Gerrit-PatchSet: 2
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot
Gerrit-MessageType: merged