Hi,
On 3/28/23 14:07, Roy Smith wrote:
> Hmmm. What I'm doing requires
Page.expand_text(), which looks like it
> does a Page.get() followed by a Site.expand_text(), and it's the later
> which actually takes most of the time. That becomes an
> action=expandtemplates API call
> <https://www.mediawiki.org/w/api.php?action=help&modules=expandtemplates>,
which I don't see any way to batch.
Naively you can batch action=expandtemplates by putting unique markers
in between each fragment of wikitext if you know the template will emit
balanced wikitext (this part is key).
For example:
{{foo1}}
--mycoolbot123--
{{foo2}}
--mycoolbot123--
{{foo3}}
Then split the response on your marker and sync up with your list of
pages :)
I suppose it's also worth asking what you're using expand_text() for in
the first place, to see if there's a better way to do whatever it is you
want to :)
For real concurrency I'll refer you to the blog post I wrote last year
about this:
<https://blog.legoktm.com/2022/01/21/building-fast-wikipedia-bots-in-rust.html>.
Yes it's about Rust, but the same concepts can easily be replicated in
Python.
-- Kunal / Legoktm