jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/357574 )
Change subject: [bugfix] Adjust _treat_counter
......................................................................
[bugfix] Adjust _treat_counter
redirect.py always reads one page on top of the -total number. The reason is
that Bot.run increases the _treat_counter after the treat(page) was done but
RedirectBot uses the old value after action() was done. Fix this behavior.
Bug: T167254
Change-Id: Ia5a0fa722a26a0614508804f2842c38ee4f7a509
---
M scripts/redirect.py
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Merlijn van Deen: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/redirect.py b/scripts/redirect.py
index 9bd6e7a..c44ae5f 100755
--- a/scripts/redirect.py
+++ b/scripts/redirect.py
@@ -735,11 +735,11 @@
def treat(self, page):
"""Treat a single page."""
- self.action_treat(page)
if self._treat_counter >= self.getOption('total'):
pywikibot.output('\nNumber of pages reached the total limit. '
'Script terminated.')
self.quit()
+ self.action_treat(page)
def main(*args):
--
To view, visit https://gerrit.wikimedia.org/r/357574
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia5a0fa722a26a0614508804f2842c38ee4f7a509
Gerrit-PatchSet: 1
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/288871 )
Change subject: Use bs4 for imageharvest, not BeautifulSoup v3
......................................................................
Use bs4 for imageharvest, not BeautifulSoup v3
Change necessary scripts in scripts/ and tests/ to ensure that
imageharvest uses BeautifulSoup v4, not v3
Bug: T115428
Change-Id: Icfd98d357126623f9431797188a52fcbcfe40dbc
---
M scripts/imageharvest.py
M tests/script_tests.py
2 files changed, 19 insertions(+), 8 deletions(-)
Approvals:
jenkins-bot: Verified
Xqt: Looks good to me, approved
diff --git a/scripts/imageharvest.py b/scripts/imageharvest.py
index 300676d..ddbf06a 100644
--- a/scripts/imageharvest.py
+++ b/scripts/imageharvest.py
@@ -5,7 +5,7 @@
It takes a URL as an argument and finds all images (and other files specified
by the extensions in 'fileformats') that URL is referring to, asking whether to
upload them. If further arguments are given, they are considered to be the text
-that is common to the descriptions.
+that is common to the descriptions. BeautifulSoup is needed only in this case.
A second use is to get a number of images that have URLs only differing in
numbers. To do this, use the command line option "-pattern", and give the URL
@@ -24,10 +24,13 @@
from __future__ import absolute_import, unicode_literals
__version__ = '$Id$'
-#
+
import os
-import BeautifulSoup
+try:
+ from bs4 import BeautifulSoup
+except ImportError as e:
+ BeautifulSoup = e
import pywikibot
@@ -45,11 +48,15 @@
def get_imagelinks(url):
"""Given a URL, get all images linked to by the page at that URL."""
+ # Check if BeautifulSoup is imported.
+ if isinstance(BeautifulSoup, ImportError):
+ raise BeautifulSoup
+
links = []
uo = URLopener()
- file = uo.open(url)
- soup = BeautifulSoup.BeautifulSoup(file.read())
- file.close()
+ with uo.open(url) as f:
+ soup = BeautifulSoup(f.read())
+
if not shown:
tagname = "a"
elif shown == "just":
diff --git a/tests/script_tests.py b/tests/script_tests.py
index 9a20c8c..b305a2b 100644
--- a/tests/script_tests.py
+++ b/tests/script_tests.py
@@ -40,9 +40,8 @@
'imagecopy_self': [TK_IMPORT],
'script_wui': ['crontab', 'lua'],
# Note: package 'lunatic-python' provides module 'lua'
-
'flickrripper': ['flickrapi'],
- 'imageharvest': ['BeautifulSoup'],
+ 'imageharvest': ['beautifulsoup4'],
'match_images': ['PIL.ImageTk'],
'states_redirect': ['pycountry'],
'patrol': ['mwparserfromhell'],
@@ -391,6 +390,11 @@
net = False
_expected_failures = failed_dep_script_list
+ # -help supported not explicitly
+ try:
+ _expected_failures.remove('imageharvest')
+ except ValueError:
+ pass
_allowed_failures = []
_argument = 'help'
--
To view, visit https://gerrit.wikimedia.org/r/288871
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Icfd98d357126623f9431797188a52fcbcfe40dbc
Gerrit-PatchSet: 13
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Darthbhyrava <hbhyrava(a)gmail.com>
Gerrit-Reviewer: Darthbhyrava <hbhyrava(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: MtDu <justin.d128(a)gmail.com>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/346164 )
Change subject: [IMPR] Implement EventStreams
......................................................................
[IMPR] Implement EventStreams
eventstreams.py:
- EventStreams class
- setup the url from site.family and streamtype if it is not given
- setup timeout from config value
- set total number with set_maximum_items
- setup any filter with register_filter method
- streamfilter filters the events to be handled
- site_rc_listener method for pagegenerators.py with fallback to rcstream.py
family.py:
- rcstream_path and eventstream_path added
rcstream.py:
- use rcstream_path from family file instead of a default path
- print a deprecation warning when site_rc_listener is used
family.py:
- additional paths are provided for eventstreams and rcstreams
pagegenerators.py:
- use the new eventstream site_rc_listener
requirements.txt:
- bind sseclient instead of socketIO-client
pagegenerators_tests.py:
- test live recent changes with EventStreams
eventstreams_tests.py:
- test suite added
setup.py
- add core library dependency
Bug: T158943
Change-Id: I4e48784cc3a30c22cdb4882dbbebf0e5a68ff8c2
---
M pywikibot/README.rst
A pywikibot/comms/eventstreams.py
M pywikibot/comms/rcstream.py
M pywikibot/family.py
M pywikibot/pagegenerators.py
M requirements.txt
M setup.py
A tests/eventstreams_tests.py
M tests/pagegenerators_tests.py
9 files changed, 525 insertions(+), 16 deletions(-)
Approvals:
Merlijn van Deen: Looks good to me, approved
jenkins-bot: Verified
diff --git a/pywikibot/README.rst b/pywikibot/README.rst
index ad05394..702f1a1 100644
--- a/pywikibot/README.rst
+++ b/pywikibot/README.rst
@@ -106,9 +106,11 @@
+---------------------------+-------------------------------------------------------+
| comms | Communication layer. |
+===========================+=======================================================+
+ | eventstreams.py | rcstream client for server sent events |
+ +---------------------------+-------------------------------------------------------+
| http.py | Basic HTTP access interface |
+---------------------------+-------------------------------------------------------+
- | rcstream.py | SocketIO-based rcstream client |
+ | rcstream.py | SocketIO-based rcstream client (deprecated) |
+---------------------------+-------------------------------------------------------+
| threadedhttp.py | Httplib2 threaded cookie layer extending httplib2 |
+---------------------------+-------------------------------------------------------+
diff --git a/pywikibot/comms/eventstreams.py b/pywikibot/comms/eventstreams.py
new file mode 100644
index 0000000..bd80ffe
--- /dev/null
+++ b/pywikibot/comms/eventstreams.py
@@ -0,0 +1,267 @@
+# -*- coding: utf-8 -*-
+"""
+Server-Sent Events client.
+
+This file is part of the Pywikibot framework.
+
+This module requires sseclient to be installed:
+ pip install sseclient
+"""
+#
+# (C) xqt, 2017
+# (C) Pywikibot team, 2017
+#
+# Distributed under the terms of the MIT license.
+#
+from __future__ import absolute_import, unicode_literals
+
+import json
+import socket
+
+from requests.packages.urllib3.exceptions import ProtocolError
+
+try:
+ from sseclient import SSEClient as EventSource
+except ImportError as e:
+ EventSource = e
+
+from pywikibot import config, debug, Site, warning
+from pywikibot.tools import StringTypes
+
+_logger = 'pywikibot.eventstreams'
+
+
+class EventStreams(object):
+
+ """Basic EventStreams iterator class for Server-Sent Events (SSE) protocol.
+
+ It provides access to arbitrary streams of data including recent changes.
+ It replaces rcstream.py implementation.
+ """
+
+ def __init__(self, **kwargs):
+ """Constructor.
+
+ @keyword site: a project site object. Used when no url is given
+ @type site: APISite
+ @keyword stream: event stream type. Used when no url is given.
+ @type stream: str
+ @keyword timeout: a timeout value indication how long to wait to send
+ data before giving up
+ @type timeout: int, float or a tuple of two values of int or float
+ @keyword url: an url retrieving events from. Will be set up to a
+ default url using _site.family settings and streamtype
+ @type url: str
+ @param kwargs: keyword arguments passed to SSEClient and requests lib
+ @type kwargs: dict
+ @raises ImportError: sseclient is not installed
+ """
+ if isinstance(EventSource, Exception):
+ raise ImportError('sseclient is required for EventStreams;\n'
+ 'install it with "pip install sseclient"\n')
+ self.filter = {'all': [], 'any': [], 'none': []}
+ self._total = None
+ self._site = kwargs.pop('site', Site())
+ self._stream = kwargs.pop('stream', None)
+ self._url = kwargs.get('url') or self.url
+ kwargs.setdefault('url', self._url)
+ kwargs.setdefault('timeout', config.socket_timeout)
+ self.sse_kwargs = kwargs
+
+ @property
+ def url(self):
+ """Get the EventStream's url.
+
+ @raises NotImplementedError: streamtype is not specified
+ """
+ if not hasattr(self, '_url'):
+ if self._stream is None:
+ raise NotImplementedError(
+ 'No stream specified for class {0}'
+ .format(self.__class__.__name__))
+ self._url = ('{0}{1}/{2}'.format(self._site.rcstream_host(),
+ self._site.eventstreams_path(),
+ self._stream))
+ return self._url
+
+ def set_maximum_items(self, value):
+ """
+ Set the maximum number of items to be retrieved from the stream.
+
+ If not called, most queries will continue as long as there is
+ more data to be retrieved from the stream.
+
+ @param value: The value of maximum number of items to be retrieved
+ in total to set.
+ @type value: int
+ """
+ if value is not None:
+ self._total = int(value)
+ debug('{0}: Set limit (maximum_items) to {1}.'
+ .format(self.__class__.__name__, self._total), _logger)
+
+ def register_filter(self, *args, **kwargs):
+ """Register a filter.
+
+ Filter types
+ ============
+ There are 3 types of filter: 'all', 'any' and 'none'.
+ The filter type must be given with the keyword argument 'ftype'
+ (see below). If no 'ftype' keyword argument is given, 'all' is
+ assumed as default.
+
+ You may register multiple filters for each type of filter.
+ The behaviour of filter type is as follows::
+
+ - B{'none'}: Skip if the any filter matches. Otherwise check 'all'.
+ - B{'all'}: Skip if not all filter matches. Otherwise check 'any':
+ - B{'any'}: Skip if no given filter matches. Otherwise pass.
+
+ Filter functions
+ ================
+ Filter may be specified as external function methods given as
+ positional argument like::
+
+ def foo(data):
+ return True
+
+ register_filter(foo, ftype='any')
+
+ The data dict from event is passed to the external filter function as
+ a parameter and that method must handle it in a proper way and return
+ C{True} if the filter matches and C{False} otherwise.
+
+ Filter keys and values
+ ======================
+ Another method to register a filter is to pass pairs of keys and values
+ as keyword arguments to this method. The key must be a key of the event
+ data dict and the value must be any value or an iterable of values the
+ C{data['key']} may match or be part of it. Samples::
+
+ register_filter(server_name='de.wikipedia.org') # 1
+ register_filter(type=('edit', 'log')) # 2
+ register_filter(ftype='none', bot=True) # 3
+
+ Explanation for the result of the filter function:
+ 1. C{return data['sever_name'] == 'de.wikipedia.org'}
+ 2. C{return data['type'] in ('edit', 'log')}
+ 3. C{return data['bot'] is True}
+
+ @keyword ftype: The filter type, one of 'all', 'any', 'none'.
+ Default value is 'all'
+ @type ftype: str
+ @param args: You may pass your own filter functions here.
+ Every function should be able to handle the data dict from events.
+ @type args: callable
+ @param kwargs: Any key returned by event data with a event data value
+ for this given key.
+ @type kwargs: str, list, tuple or other sequence
+ @raise TypeError: A given args parameter is not a callable.
+ """
+ ftype = kwargs.pop('ftype', 'all') # set default ftype value
+
+ # register an external filter function
+ for func in args:
+ if callable(func):
+ self.filter[ftype].append(func)
+ else:
+ raise TypeError('{0} is not a callable'.format(func))
+
+ # register pairs of keys and items as a filter function
+ for key, value in kwargs.items():
+ # append function for singletons
+ if value in (True, False, None):
+ self.filter[ftype].append(lambda e: key in e and
+ e[key] is value)
+ # append function for a single value
+ elif isinstance(value, StringTypes):
+ self.filter[ftype].append(lambda e: key in e and
+ e[key] == value)
+ # append function for an iterable as value
+ else:
+ self.filter[ftype].append(lambda e: key in e and
+ e[key] in value)
+
+ def streamfilter(self, data):
+ """Filter function for eventstreams.
+
+ See the description of register_filter() how it works.
+
+ @param data: event data dict used by filter functions
+ @type data: dict
+ """
+ if any(function(data) for function in self.filter['none']):
+ return False
+ if not all(function(data) for function in self.filter['all']):
+ return False
+ if not self.filter['any']:
+ return True
+ return any(function(data) for function in self.filter['any'])
+
+ def __iter__(self):
+ """Iterator."""
+ n = 0
+ event = None
+ while self._total is None or n < self._total:
+ if not hasattr(self, 'source'):
+ self.source = EventSource(**self.sse_kwargs)
+ try:
+ event = next(self.source)
+ except (ProtocolError, socket.error) as e:
+ warning('Connection error: {0}.\n'
+ 'Try to re-establish connection.'.format(e))
+ del self.source
+ if event is not None:
+ self.sse_kwargs['last_id'] = event.id
+ continue
+ if event.event == 'message' and event.data:
+ try:
+ element = json.loads(event.data)
+ except ValueError as e:
+ warning('Could not load json data from\n{0}\n{1}'
+ .format(event, e))
+ else:
+ if self.streamfilter(element):
+ n += 1
+ yield element
+ elif event.event == 'message' and not event.data:
+ warning('Empty message found.')
+ elif event.event == 'error':
+ warning('Encountered error: {0}'.format(event.data))
+ else:
+ warning('Unknown event {0} occured.'.format(event.event))
+ else:
+ debug('{0}: Stopped iterating due to '
+ 'exceeding item limit.'
+ .format(self.__class__.__name__), _logger)
+ del self.source
+
+
+def site_rc_listener(site, total=None):
+ """Yield changes received from EventStream.
+
+ @param site: the Pywikibot.Site object to yield live recent changes for
+ @type site: Pywikibot.BaseSite
+ @param total: the maximum number of changes to return
+ @type total: int
+
+ @return: pywikibot.comms.eventstream.rc_listener configured for given site
+ """
+ if isinstance(EventSource, Exception):
+ warning('sseclient is required for EventStreams;\n'
+ 'install it with "pip install sseclient"\n')
+ # fallback to old rcstream method
+ # NOTE: this will be deprecated soon
+ from pywikibot.comms.rcstream import rc_listener
+ return rc_listener(
+ wikihost=site.hostname(),
+ rchost=site.rcstream_host(),
+ rcport=site.rcstream_port(),
+ rcpath=site.rcstream_path(),
+ total=total,
+ )
+
+ stream = EventStreams(stream='recentchange', site=site)
+ stream.set_maximum_items(total)
+ stream.register_filter(server_name=site.hostname())
+ return stream
diff --git a/pywikibot/comms/rcstream.py b/pywikibot/comms/rcstream.py
index bb9ed7b..a235f90 100644
--- a/pywikibot/comms/rcstream.py
+++ b/pywikibot/comms/rcstream.py
@@ -9,7 +9,7 @@
"""
#
# (C) 2014 Merlijn van Deen
-# (C) Pywikibot team, 2014-2016
+# (C) Pywikibot team, 2014-2017
#
# Distributed under the terms of the MIT license.
#
@@ -32,6 +32,7 @@
socketIO_client = e
from pywikibot.bot import debug, warning
+from pywikibot.tools import deprecated
_logger = 'pywikibot.rcstream'
@@ -205,6 +206,7 @@
yield element
+@deprecated('eventstreams.site_rc_listener')
def site_rc_listener(site, total=None):
"""Yield changes received from RCstream.
@@ -219,5 +221,6 @@
wikihost=site.hostname(),
rchost=site.rcstream_host(),
rcport=site.rcstream_port(),
+ rcpath=site.rcstream_path(),
total=total,
)
diff --git a/pywikibot/family.py b/pywikibot/family.py
index c24e0f5..45431a6 100644
--- a/pywikibot/family.py
+++ b/pywikibot/family.py
@@ -1149,7 +1149,16 @@
def rcstream_host(self, code):
"""Hostname for RCStream."""
+ raise NotImplementedError(
+ 'This family does support neither RCStream nor EventStreams')
+
+ def rcstream_path(self, code):
+ """Return path for RCStream."""
raise NotImplementedError("This family does not support RCStream")
+
+ def eventstreams_path(self, code):
+ """Return path for EventStreams."""
+ raise NotImplementedError('This family does not support EventStreams')
@deprecated_args(name='title')
def get_address(self, code, title):
@@ -1643,6 +1652,14 @@
"""Return 443 as the RCStream port number."""
return 443
+ def rcstream_path(self, code):
+ """Return path for RCStream."""
+ return '/rc'
+
+ def eventstreams_path(self, code):
+ """Return path for EventStreams."""
+ return '/v2/stream'
+
class WikimediaOrgFamily(SingleSiteFamily, WikimediaFamily):
diff --git a/pywikibot/pagegenerators.py b/pywikibot/pagegenerators.py
index c4be421..6dd1290 100644
--- a/pywikibot/pagegenerators.py
+++ b/pywikibot/pagegenerators.py
@@ -2381,7 +2381,7 @@
if site is None:
site = pywikibot.Site()
- from pywikibot.comms.rcstream import site_rc_listener
+ from pywikibot.comms.eventstreams import site_rc_listener
for entry in site_rc_listener(site, total=total):
# The title in a log entry may have been suppressed
diff --git a/requirements.txt b/requirements.txt
index d17dc42..0838f60 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -55,7 +55,7 @@
# core pagegenerators
google >= 1.7
-socketIO-client<0.6.1
+sseclient
# scripts/script_wui.py:
crontab
diff --git a/setup.py b/setup.py
index bf0055e..cb74bca 100644
--- a/setup.py
+++ b/setup.py
@@ -54,6 +54,7 @@
extra_deps = {
# Core library dependencies
+ 'eventstreams': ['sseclient'],
'isbn': ['python-stdnum'],
'Graphviz': ['pydot>=1.0.28'],
'Google': ['google>=1.7'],
diff --git a/tests/eventstreams_tests.py b/tests/eventstreams_tests.py
new file mode 100644
index 0000000..8e440bc
--- /dev/null
+++ b/tests/eventstreams_tests.py
@@ -0,0 +1,213 @@
+# -*- coding: utf-8 -*-
+"""Tests for the eventstreams module."""
+#
+# (C) Pywikibot team, 2017
+#
+# Distributed under the terms of the MIT license.
+#
+from __future__ import absolute_import, unicode_literals
+
+from types import FunctionType
+
+import mock
+
+from pywikibot.comms.eventstreams import EventStreams
+from pywikibot import config
+from pywikibot.family import WikimediaFamily
+
+from tests.aspects import unittest, TestCase, DefaultSiteTestCase
+
+
+(a)mock.patch('pywikibot.comms.eventstreams.EventSource', new=mock.MagicMock())
+class TestEventStreamsUrlTests(TestCase):
+
+ """Url tests for eventstreams module."""
+
+ sites = {
+ 'de.wp': {
+ 'family': 'wikipedia',
+ 'code': 'de',
+ 'hostname': 'de.wikipedia.org',
+ },
+ 'en.wq': {
+ 'family': 'wikiquote',
+ 'code': 'en',
+ 'hostname': 'en.wikiquote.org',
+ },
+ }
+
+ def test_url_parameter(self, key):
+ """Test EventStreams with given url."""
+ e = EventStreams(url=self.sites[key]['hostname'])
+ self.assertEqual(e._url, self.sites[key]['hostname'])
+ self.assertEqual(e._url, e.url)
+ self.assertEqual(e._url, e.sse_kwargs.get('url'))
+ self.assertIsNone(e._total)
+ self.assertIsNone(e._stream)
+
+ def test_url_from_site(self, key):
+ """Test EventStreams with url from site."""
+ site = self.get_site(key)
+ stream = 'recentchanges'
+ e = EventStreams(site=site, stream=stream)
+ self.assertEqual(
+ e._url, 'https://stream.wikimedia.org/v2/stream/' + stream)
+ self.assertEqual(e._url, e.url)
+ self.assertEqual(e._url, e.sse_kwargs.get('url'))
+ self.assertIsNone(e._total)
+ self.assertEqual(e._stream, stream)
+
+
+(a)mock.patch('pywikibot.comms.eventstreams.EventSource', new=mock.MagicMock())
+class TestEventStreamsStreamTests(DefaultSiteTestCase):
+
+ """Stream tests for eventstreams module."""
+
+ def test_url_with_stream(self):
+ """Test EventStreams with url from default site."""
+ site = self.get_site()
+ fam = site.family
+ if not isinstance(fam, WikimediaFamily):
+ self.skipTest(
+ "Family '{0}' of site '{1}' is not a WikimediaFamily."
+ .format(fam, site))
+
+ stream = 'recentchanges'
+ e = EventStreams(stream=stream)
+ self.assertEqual(
+ e._url, 'https://stream.wikimedia.org/v2/stream/' + stream)
+ self.assertEqual(e._url, e.url)
+ self.assertEqual(e._url, e.sse_kwargs.get('url'))
+ self.assertIsNone(e._total)
+ self.assertEqual(e._stream, stream)
+
+ def test_url_missing_stream(self):
+ """Test EventStreams with url from site with missing stream."""
+ with self.assertRaises(NotImplementedError):
+ EventStreams()
+
+
+class TestEventStreamsSettingTests(TestCase):
+
+ """Setting tests for eventstreams module."""
+
+ dry = True
+
+ def setUp(self):
+ """Set up unit test."""
+ super(TestEventStreamsSettingTests, self).setUp()
+ with mock.patch('pywikibot.comms.eventstreams.EventSource'):
+ self.es = EventStreams(url='dummy url')
+
+ def test_maximum_items(self):
+ """Test EventStreams total value."""
+ total = 4711
+ self.es.set_maximum_items(total)
+ self.assertEqual(self.es._total, total)
+
+ def test_timeout_setting(self):
+ """Test EventStreams timeout value."""
+ self.assertEqual(self.es.sse_kwargs.get('timeout'),
+ config.socket_timeout)
+
+ def test_filter_function_settings(self):
+ """Test EventStreams filter function settings."""
+ def foo():
+ """Dummy function."""
+ return True
+
+ self.es.register_filter(foo)
+ self.assertEqual(self.es.filter['all'][0], foo)
+ self.assertEqual(self.es.filter['any'], [])
+ self.assertEqual(self.es.filter['none'], [])
+
+ self.es.register_filter(foo, ftype='none')
+ self.assertEqual(self.es.filter['all'][0], foo)
+ self.assertEqual(self.es.filter['any'], [])
+ self.assertEqual(self.es.filter['none'][0], foo)
+
+ self.es.register_filter(foo, ftype='any')
+ self.assertEqual(self.es.filter['all'][0], foo)
+ self.assertEqual(self.es.filter['any'][0], foo)
+ self.assertEqual(self.es.filter['none'][0], foo)
+
+ def test_filter_function_settings_fail(self):
+ """Test EventStreams failing filter function settings."""
+ with self.assertRaises(TypeError):
+ self.es.register_filter('test')
+
+ def test_filter_settings(self):
+ """Test EventStreams filter settings."""
+ self.es.register_filter(foo='bar')
+ self.assertIsInstance(self.es.filter['all'][0], FunctionType)
+ self.es.register_filter(bar='baz')
+ self.assertEqual(len(self.es.filter['all']), 2)
+
+
+class TestEventStreamsFilterTests(TestCase):
+
+ """Filter tests for eventstreams module."""
+
+ dry = True
+
+ data = {'foo': True, 'bar': 'baz'}
+
+ def setUp(self):
+ """Set up unit test."""
+ super(TestEventStreamsFilterTests, self).setUp()
+ with mock.patch('pywikibot.comms.eventstreams.EventSource'):
+ self.es = EventStreams(url='dummy url')
+
+ def test_filter_function_all(self):
+ """Test EventStreams filter all function."""
+ self.es.register_filter(lambda x: True)
+ self.assertTrue(self.es.streamfilter(self.data))
+ self.es.register_filter(lambda x: False)
+ self.assertFalse(self.es.streamfilter(self.data))
+
+ def test_filter_function_any(self):
+ """Test EventStreams filter any function."""
+ self.es.register_filter(lambda x: True, ftype='any')
+ self.assertTrue(self.es.streamfilter(self.data))
+ self.es.register_filter(lambda x: False, ftype='any')
+ self.assertTrue(self.es.streamfilter(self.data))
+
+ def test_filter_function_none(self):
+ """Test EventStreams filter none function."""
+ self.es.register_filter(lambda x: False, ftype='none')
+ self.assertTrue(self.es.streamfilter(self.data))
+ self.es.register_filter(lambda x: True, ftype='none')
+ self.assertFalse(self.es.streamfilter(self.data))
+
+ def _test_filter(self, none_type, all_type, any_type, result):
+ """Test a single fixed filter."""
+ self.es.filter = {'all': [], 'any': [], 'none': []}
+ self.es.register_filter(lambda x: none_type, ftype='none')
+ self.es.register_filter(lambda x: all_type, ftype='all')
+ if any_type is not None:
+ self.es.register_filter(lambda x: any_type, ftype='any')
+ self.assertEqual(self.es.streamfilter(self.data), result,
+ 'Test EventStreams filter mixed function failed for\n'
+ "'none': {0}, 'all': {1}, 'any': {2}\n"
+ '(expected {3}, given {4})'
+ .format(none_type, all_type, any_type,
+ result, not result))
+
+ def test_filter_mixed_function(self):
+ """Test EventStreams filter mixed function."""
+ for none_type in (False, True):
+ for all_type in (False, True):
+ for any_type in (False, True, None):
+ if none_type is False and all_type is True and (
+ any_type is None or any_type is True):
+ result = True
+ else:
+ result = False
+ self._test_filter(none_type, all_type, any_type, result)
+
+
+if __name__ == '__main__': # pragma: no cover
+ try:
+ unittest.main()
+ except SystemExit:
+ pass
diff --git a/tests/pagegenerators_tests.py b/tests/pagegenerators_tests.py
index 6eaca83..3081298 100755
--- a/tests/pagegenerators_tests.py
+++ b/tests/pagegenerators_tests.py
@@ -26,6 +26,8 @@
CategorizedPageGenerator
)
+from pywikibot.tools import has_module
+
from tests import join_data_path
from tests.aspects import (
unittest,
@@ -1299,27 +1301,31 @@
)
-class LiveRCPageGeneratorTestCase(RecentChangesTestCase):
+class EventStreamsPageGeneratorTestCase(RecentChangesTestCase):
"""Test case for Live Recent Changes pagegenerator."""
@classmethod
def setUpClass(cls):
"""Setup test class."""
- super(LiveRCPageGeneratorTestCase, cls).setUpClass()
- try:
- import socketIO_client
- except ImportError:
- raise unittest.SkipTest('socketIO_client not available')
-
- if LooseVersion(socketIO_client.__version__) >= LooseVersion('0.6.1'):
- raise unittest.SkipTest(
- 'socketIO_client %s not supported by Wikimedia-Stream'
- % socketIO_client.__version__)
+ super(EventStreamsPageGeneratorTestCase, cls).setUpClass()
+ cls.client = 'sseclient'
+ if not has_module(cls.client):
+ cls.client = 'socketIO_client'
+ try:
+ import socketIO_client
+ except ImportError:
+ raise unittest.SkipTest(
+ 'Neither sseclient nor socketIO_client is available')
+ if LooseVersion(
+ socketIO_client.__version__) >= LooseVersion('0.6.1'):
+ raise unittest.SkipTest(
+ 'socketIO_client %s not supported by Wikimedia-Stream'
+ % socketIO_client.__version__)
def test_RC_pagegenerator_result(self):
"""Test RC pagegenerator."""
- lgr = logging.getLogger('socketIO_client')
+ lgr = logging.getLogger(self.client)
lgr.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
--
To view, visit https://gerrit.wikimedia.org/r/346164
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I4e48784cc3a30c22cdb4882dbbebf0e5a68ff8c2
Gerrit-PatchSet: 33
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: Eranroz <eranroz89(a)gmail.com>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Lokal Profil <lokal.profil(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Multichill <maarten(a)mdammers.nl>
Gerrit-Reviewer: Ottomata <aotto(a)wikimedia.org>
Gerrit-Reviewer: Xqt <info(a)gno.de>
Gerrit-Reviewer: jenkins-bot <>
jenkins-bot has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/358000 )
Change subject: Code cleanup
......................................................................
Code cleanup
- Remove deprecated method which is no longer used.
It is not needed to deprecate it for scripts which isn't derived anywhere.
Change-Id: I51c6492eef82af9f68b730d941797ad7f148d079
---
M scripts/checkimages.py
1 file changed, 1 insertion(+), 9 deletions(-)
Approvals:
Mpaa: Looks good to me, approved
jenkins-bot: Verified
diff --git a/scripts/checkimages.py b/scripts/checkimages.py
index 10f6599..7b22ade 100755
--- a/scripts/checkimages.py
+++ b/scripts/checkimages.py
@@ -102,7 +102,7 @@
from pywikibot.exceptions import ArgumentDeprecationWarning, NotEmailableError
from pywikibot.family import Family
-from pywikibot.tools import deprecated, issue_deprecation_warning, StringTypes
+from pywikibot.tools import issue_deprecation_warning
###############################################################################
# <--------------------------- Change only below! --------------------------->#
@@ -828,14 +828,6 @@
# find the oldest image
sec, image = max(listGiven, key=lambda element: element[0])
return image
-
- @deprecated('Page.revision_count()')
- def countEdits(self, pagename, userlist):
- """Function to count the edit of a user or a list of users in a page."""
- if isinstance(userlist, StringTypes):
- userlist = [userlist]
- page = pywikibot.Page(self.site, pagename)
- return page.revision_count(userlist)
def checkImageOnCommons(self):
"""Checking if the file is on commons."""
--
To view, visit https://gerrit.wikimedia.org/r/358000
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I51c6492eef82af9f68b730d941797ad7f148d079
Gerrit-PatchSet: 2
Gerrit-Project: pywikibot/core
Gerrit-Branch: master
Gerrit-Owner: Xqt <info(a)gno.de>
Gerrit-Reviewer: John Vandenberg <jayvdb(a)gmail.com>
Gerrit-Reviewer: Magul <tomasz.magulski(a)gmail.com>
Gerrit-Reviewer: Merlijn van Deen <valhallasw(a)arctus.nl>
Gerrit-Reviewer: Mpaa <mpaa.wiki(a)gmail.com>
Gerrit-Reviewer: jenkins-bot <>
Build Update for wikimedia/pywikibot-core
-------------------------------------
Build: #4108
Status: Broken
Duration: 2 hours, 52 minutes, and 57 seconds
Commit: 4d134df (master)
Author: xqt
Message: Revert "Full list of 'Category redirect' template aliases for rowiki:"
Redirected templates aren't necessary because they are retrieved via
family._get_cr_templates from api, see 5c382e9d
This reverts commit 4b84943ca66313f9a3ebe5f3ff745b322860622f.
Change-Id: I36b1312af28618453845fdcf124a1a9bcb7907cc
View the changeset: https://github.com/wikimedia/pywikibot-core/compare/bd625de1c256...4d134df0…
View the full build log and details: https://travis-ci.org/wikimedia/pywikibot-core/builds/241142457?utm_source=…
--
You can configure recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications