My question is: why does the default behavior need to change? Wouldn't continuing with the default behavior allow people to continue using the "rawcontinue" behavior for as long as we want to support it—without making any changes?

---------------------------------------------------------------------- Message: 1 Date: Wed, 3 Jun 2015 19:08:24 +0300 From: Yuri Astrakhan &lt;yastrakhan(a)wikimedia.org&gt; To: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt; Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CALOOOkhzCUf14Tf= P0uQw7ZXAmwb9qAxs4kgvbJAvSsMQY82GA(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 MZ: >I feel that former MediaWiki Web API maintainers should actively pay attention to which mailing lists they're posting to. ;-) I doubt you intended to send this message to mediawiki-api-announce. Mz, I don't think I ever spent much time maintaining it :)) But yes, good point, reply all is evil at times :) MZ: re why minimalistic lib - for most apis out there, people tend to write "reference implementation" - code that explains to would-be lib authors how to use API. It sets up prefered usage patterns and guidelines. We never had that for the api. This resulted in a large number of API libs that vary in how they use api. Pywikibot is a powerful platform, but is too big for many usecases (as discussed in Lyon). Hence the need. SW: >Most operator are volunteers and don't have time to change the code every month because there is a change in the api. * I might agree about every month, but we are talking about a feature that has been out for 2 years... * Most API users imho would laugh at this statement. See the roadmap <https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap> which came as a result of analysing numerous bugs & wishes filed against the api, such as returning {} instead of [] for empty values, inconsistencies, the silly '*' value for text, and many other problems that have accumulated during the years. API might work for you, but it does not support multilingual error messaging, it overcomplicates the process of writing javascript clients, it does not easily cache. In short, lots of problems. * I wrote the original API as a volunteer (took about a year in 2005/6). I recently coded the continuation as a volunteer, and Brad has improved it as a WMF employee. I later became a full time WMF employee as well, but my project was Zero, not API. As a volunteer, over 2 years ago I wrote a huge API improvement roadmap <https://www.mediawiki.org/wiki/Requests_for_comment/API_roadmap>that Brad picked up and greatly extended, and is driving it forward with the amazing improvements. From what I know, Brad has now been officially moved into position of working on the API. In other words, that employee vs volunteer line is very fuzzy. No point of splitting it that way. TLDR; In short, we need to make improvements if we want to move forward in turning API into a platform, with flexible JavaScript-driven interfaces such as Visual Editor. To allow creative uses that go beyond AWB, that support complex interactions, and not just the way for bots to make edits. Unfortunately, that means that despite our best efforts, very infrequently, some bots must be updated. If the bot author cannot add a simple one line change "rawcontinue=1" once in two years because of their busy personal live, I don't think that person should be making automatic edits to Wikipedia - because sometimes bots make mistakes that require substantially more time involvement. I would not trust wikipedia edits to a bot runner under such circumstances. If the bot runner is not a programmer, they should get the latest version of their bot. If there is no up-to-date code because noone is maintaining it, it again should not be accessing wikipedia - we sometimes discover security bugs that require fixing, or because bot calls wiki too often, or other changes in content structure - e.g. introduction of WikiData for interwiki links required all interwiki bots to be updated. Wikipedia is a living, developing ecosystem, and it does require updates to all parties accessing it. People accessing wikipedia from the older browsers discover that they no longer can do everything they used to many years ago - because we now use newer browser features, and fallback into basic mode for older browsers. Please participate in the evolution of the project. On Wed, Jun 3, 2015 at 5:22 PM, Steinsplitter Wiki < steinsplitter-wiki(a)live.com&gt; wrote: perfectly. > > > > I feel that bot operators should actively pay attention to the technical mode technical who announcements, for hackathon https://www.mediawiki.org/wiki/Requests_for_comment/Minimalistic_MW_API_Cli… wrote: scale in ------------------------------ Message: 2 Date: Wed, 3 Jun 2015 12:34:12 -0400 From: "Brad Jorsch (Anomie)" &lt;bjorsch(a)wikimedia.org&gt; To: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt; Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSuKej9O0beGt6xco+c+j7uTa0BtZoeCAQs9Cz6XQ2gYwg(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 On Wed, Jun 3, 2015 at 6:49 AM, Steinsplitter Wiki < steinsplitter-wiki(a)live.com&gt; wrote: wondering We usually do. In this case, however, the advantages of changing the default for new API users seems to outweigh the disadvantages of a well-announced breaking change with a simple parameter to request backwards-compatible output. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation ------------------------------ Message: 3 Date: Wed, 3 Jun 2015 12:43:36 -0400 From: "Brad Jorsch (Anomie)" &lt;bjorsch(a)wikimedia.org&gt; To: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt; Cc: mediawiki-api-announce(a)lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CAEepRSvprWamR3y+TFfcLg6Jv4hkbtCqbzj6JxfSvZe4v8f4Ow(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 On Wed, Jun 3, 2015 at 7:29 AM, John Mark Vandenberg &lt;jayvdb(a)gmail.com&gt; wrote: I already have the list of *accounts* affected: there are 510 with between 1000 and 10000 hits. Of those, 454 do not contain "bot" (case insensitively), so they might be human users with user scripts, or AWB if that's not fixed (someone please check!), or the like. For comparison, in the over-10000 group there were 30 such that I filtered out. I'll want to check with Legal to make sure the additional release of account names is still compliant with the privacy policy (I'm almost but not entirely sure it would be ok). That's not impossible, but I wouldn't count on it. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation ------------------------------ Message: 4 Date: Wed, 3 Jun 2015 13:13:34 -0400 From: "Brad Jorsch (Anomie)" &lt;bjorsch(a)wikimedia.org&gt; To: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt; Cc: MediaWiki API announcements & discussion &lt;mediawiki-api(a)lists.wikimedia.org&gt; Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: <CAEepRSsfin=praEeth2qQQFtwS_SHgFDbZFpXEd= uqORvn4sJA(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 On Wed, Jun 3, 2015 at 10:04 AM, Brian Gerstle &lt;bgerstle(a)wikimedia.org&gt; wrote: making The decision to change the default came out of the same concerns that led to the improved action=help output and some of the other work I've been doing lately: We want to lower the barriers for using our API, which means that the default shouldn't be something user-hostile. The raw continuation is deceptively simple: it looks straightforward, but if you're using it with a generator, multiple prop modules, and meta or list modules, your client code has to know when to ignore the returned continuation for the generator, when to remove a module from prop and then when to re-add it, and when to remove the meta or list modules. I wouldn't be that surprised to learn that more people have it wrong than correct if their code supports using prop modules with generators. The new continuation actually is simple: you send the equivalent of array_merge( $originalParams, $continueParams ) and it just works. Yes, some of the same could be said for making format=json&formatversion=2 the default. In this case the formatversion=1 output is just annoying rather than actually hostile (although representing boolean true as a falsey string comes close), so at this time there's no plan to make that breaking change. -- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation ------------------------------ Message: 5 Date: Wed, 3 Jun 2015 20:31:42 +0300 From: Yuri Astrakhan &lt;yastrakhan(a)wikimedia.org&gt; To: "MediaWiki API announcements & discussion" &lt;mediawiki-api(a)lists.wikimedia.org&gt; Cc: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt; Subject: Re: [Wikitech-l] [Mediawiki-api] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CALOOOkjcnXoXCSMOQe5z58NR2iw24tf7qPcOkSXWT1qZLt30hQ(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 I would like to point out that it might be a good idea add &formatversion=1 for anyone who wants to lock the current formatting in place. On Wed, Jun 3, 2015 at 8:13 PM, Brad Jorsch (Anomie) < bjorsch(a)wikimedia.org&gt; wrote: making means then wouldn't format=json&formatversion=2 ------------------------------ Message: 6 Date: Wed, 3 Jun 2015 11:55:40 -0600 From: Brian Wolff &lt;bawolff(a)gmail.com&gt; To: Wikimedia developers &lt;wikitech-l(a)lists.wikimedia.org&gt;rg>, mediawiki-api(a)lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month Message-ID: < CA+oo+DUnkpwL_EsG7DmGQOq5OsSPJmH9FnL+eW_xbaOO6XFjdg(a)mail.gmail.com&gt; Content-Type: text/plain; charset=UTF-8 On 6/3/15, Steinsplitter Wiki &lt;steinsplitter-wiki(a)live.com&gt; wrote: keep perfectly. > > >> I feel that bot operators should actively pay attention to the technical >> aspects of the community and the mailing lists. > Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most My understanding is that most of the people who were using the original continuation, were using it wrong, causing subtle bugs in their script. Thus the existing implementation was wasting considerable amount of volunteer bot developer time. In the long run this change will hopefully reduce the total amount of time spent by volunteer bot makers chasing weird bugs, at the expense of some short term pain. Its always a challenge to balance backwards compatibility with fixing things that are causing problems. I think the API team is keenly aware of the frustrations that changes to the api cause, and try to make sure that intentional breakage only happens when the benefits truly outweigh the cons. -- Bawolff ------------------------------ _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l End of Wikitech-l Digest, Vol 143, Issue 8 ******************************************

Date: Tue, 9 Jun 2015 13:03:03 +1000 From: jayvdb(a)gmail.com To: wikitech-l(a)lists.wikimedia.org Subject: Re: [Wikitech-l] API BREAKING CHANGE: Default continuation mode for action=query will change at the end of this month On Tue, Jun 9, 2015 at 6:04 AM, Ilya Korniyko &lt;intracer(a)gmail.com&gt; wrote: On the pywikipedia side, it has been unofficially deprecated for a while, and the intention was to decommission compat as gracefully as possible, with Wikimania as the final killing ground. The pywikibot developers roughly hashed out a decommissioning plan at the Lyon hackathon, and worked with WMF staff to start doing impact analysis. https://phabricator.wikimedia.org/T101214 Then the MW API continuation breakage was announced post Lyon. Hmm. The continuation problem in pywikipedia / compat is not so much those 50 or so occurrences where continuation bugs appear to exist, but 1. testing those 50 or so occurrences 2. finding the other less obvious occurrences 3. scripts people have written using compat's query.GetData may also be broken, as it doesnt do continuation. With a lot of wasted effort, 1 & 2 might be resolved so that 'compat' code works, and someone could create a hack on query.GetData which adds continuation for scripts not yet adapted to do continuation. However the active pywikibot developers (approx. 5 people?) are focused on making core better , including about 10 complex patches under review that improve its query continuation algorithms, and making core more compatible with 'compat' to ease the pain of switching to core. If anyone submits patches for compat continuation bugs affecting them, they will be reviewed (usually by people familiar with compat) with the presumption that the patch author has tested it, and merged if there are no major problems. -- John Vandenberg _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Maybe someone with enough time and knowledge can fork compat and keep it alive...

On Wed, Jun 3, 2015 at 1:13 PM, Brad Jorsch (Anomie) < bjorsch(a)wikimedia.org&gt; wrote: While it is (a little bit) nicer for new developers, they'll just burned (along with all the other current API users) when you change the defaults. What I'm trying to say is, changing the default seems like more work for more people with very little benefit. This is why <https://developer.github.com/v3/> people <https://www.reddit.com/dev/api> version <https://stripe.com/docs/api#charge_object> APIs <https://developer.linkedin.com/docs/rest-api>. That's my point, why *ever* make it a breaking change? It's such a low cost to add a few characters to the URL to specify an API version. Our docs should tell developers what the current version is. -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

deprecation process* will take care of most things that actually need improvement, to the point where we can do per-module versioning on an ad hoc basis rather than trying to force it everywhere.

On 06/04/2015 09:45 AM, Brian Gerstle wrote: defaults. https://www.reddit.com/dev/api> I'd recommend reading <https://phabricator.wikimedia.org/T41592>, which contains a pretty good rationale of why we currently don't version the API. -- Legoktm _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1. If not already present, a request parameter will be added to specifically request the old behavior. 2. The change will be announced: - A message will be sent to the mediawiki-api-announce <http://lists.wikimedia.org/pipermail/mediawiki-api-announce/> mailing list. - Deprecation warnings will be output when neither the select-new-version nor the select-old-version flags are used. Logs will also be made. 3. *After a suitable timeframe, the new version will become the default.* 4. *Any flag to select the new version explicitly may at some point be removed, leading to "unrecognized parameter" warnings.*

I know about the phab ticket, but I'm glad you referenced it because I found this: (Anomie): I think feature flags to *select new behavior* and *a good IOW, why don't we continue w/ this feature flagging approach, which seems like a decent way to version APIs and prevent breaking backwards compatibility? On Thu, Jun 4, 2015 at 1:05 PM, Legoktm &lt;legoktm.wikipedia(a)gmail.com&gt; wrote: -- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle

Most operator are volunteers and don't have time to change the code every month because there is a change in the api. Because of this devs should keep the api backward-compatible. Also wondering why wee need this "new" api. The old one was imho perfectly. Was the new api coded by WMF or by volunteers? Sorry, i disagree. Bot operators are volunteers and not payed staffers. Most of them having a job and real live. -- Steinsplitter

On Wed, 03 Jun 2015 18:29:08 +0700, John Mark Vandenberg wrote: Should someone contact those bots on their talk page to notify the owners or do we hope everyone reads this mailing list?

Also, this change will affect not only bots but also every piece of software or tool that was written or published prior to the change and relies on the API to fetch data. So there are two issues with that: * The maintainer of the tool has no time/interest to compile and publish new releases of versions which were considered stable. This means the tool will die if the source code is not available or no one wants to take over. * Previously published programs that were considered stable and are executed after July 1st. The user may not be aware that this tool is no longer stable and requires an update. As far as I understood this API change: The results from the API will just be somewhat different and not produce an error. So the software will not crash but may produce weird behavior instead. Is there any solution to this?

On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg &lt;jayvdb(a)gmail.com&gt; wrote: Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?

On Wed, Jun 17, 2015 at 7:44 PM, John Mark Vandenberg &lt;jayvdb(a)gmail.com&gt; wrote: Hmm, this is actually an interesting idea - would it make sense to error on missing "continue" or "rawcontinue" for all action=query for about a month or two, so that everyone notices it right away and gets updated, and than resume with the new behavior?

1. Current, "change the default" approach: 1. All clients start receiving warnings as *extra payload data* (how does this even work for API's w/o a response payload?)

5. Server removes API 2 (becomes the new default) 6. Clients A, B, & C have to switch to API 1

Brad Jorsch (Anomie) wrote: I would think any request to api.php using format=none would lack a response payload? However I'm not totally sure that I'm reading and using the term response payload in the same way that you two might be.

I guess it comes down to is this: if we're going to continue supporting old behavior, they should be accessible via the same old requests. *This removes the need for existing clients to be updated in the first place*. If we eventually want to delete the old code keeping the old behavior separated from the new will make it clear & explicit what's being dropped and what to use instead. ... Lastly, changing the default behavior to make things sane for new developers is, IMO, a bad trade-off

because they'll eventually get tripped by us pulling the rug out from under their feet by *breaking backwards compatibility stable APIs*.

Those sorts of changes should be reserved for experimental or even beta APIs. Continuing queries seems like a stable—and pervasive—part of the API.

On Thu, Jun 18, 2015 at 9:26 AM, Brian Gerstle &lt;bgerstle(a)wikimedia.org&gt; wrote: That seems the crux of it. Because the MediaWiki API isn't versioned (and there are good reasons for it), the recommended "best practices" form of an API request evolves, something like api.php?continue=& formatversion=2 &utf8= &*your_actual_params*