I don't like it's cryptic nature. Someone looking at the headers sent to his browser would be very confused about what's the point of «X-MF-Mode: b». Instead something like this would be much more descriptive: X-Mobile-Mode: stable X-Mobile-Request: secondary But that also means sending more bytes through the wire :S

Thanks Ori, I was not aware of this D Sent from my iPhone On 2013-02-02, at 16:55, Ori Livneh <ori(a)wikimedia.org> wrote: Similar Constructs in Application Protocols _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

If you want to differentiate categories of API requests in logs, add descriptive noop query params to the requests. I.e &mfmode=2. Doing this in request headers and altering edge config is unnecessary and a bad design pattern. On the analytics side, if parsing query params seems challenging vs. having a fixed field to parse, deal. On Sunday, February 3, 2013, David Schoonover wrote:

Considering that the query component of a URI is meant to identify the resource whereas HTTP headers are meant to tell the server additional information about the request, I think a header approach is much more appropriate than a no-op query parameter. If the X- is removed, I'd have no problem with the addition of these headers, but what is the advantage of having two over one. Wouldn't a header like: MobileFrontend: 1/2 a/b/s work just as fine? *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com <javascript:;> On Sun, Feb 3, 2013 at 4:35 AM, Asher Feldman <afeldman@wikimedia.org<javascript:;> query all that unrelated this design challenging Thanks, be or Prefix > >> and > >> > Similar Constructs in Application Protocols > >> > > > >> > > > >> > > -- > >> > > Ori Livneh > >> > > > >> > > > >> > > > >> > > > >> > > _______________________________________________ > >> > > Wikitech-l mailing list > >> > > Wikitech-l(a)lists.wikimedia.org > >> > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > >> > > >> > _______________________________________________ > >> > Wikitech-l mailing list > >> > Wikitech-l(a)lists.wikimedia.org > >> > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > >

Remind me again why a production setup is logging every header of every request?

Also, if you are logging every header, then the amount of data added by a single extra header would be insignificant compared to the rest of the request. *--* *Tyler Romeo* Stevens Institute of Technology, Class of 2015 Major in Computer Science www.whizkidztech.com | tylerromeo(a)gmail.com <javascript:;> On Sun, Feb 3, 2013 at 5:12 AM, Asher Feldman <afeldman@wikimedia.org<javascript:;> in > the log stream that must be set on all non mobile requests ("\t-\t-") > equals gigabytes of unnecessarily log data every day (that we want > to save 100% of) for no good reason. Wanting to keep query params "pure" > isn't a good reason. > > On Sunday, February 3, 2013, Tyler Romeo wrote: > > > Considering that the query component of a URI is meant to identify the > > resource whereas HTTP headers are meant to tell the server additional > > information about the request, I think a header approach is much more > > appropriate than a no-op query parameter. > > > > If the X- is removed, I'd have no problem with the addition of these > > headers, but what is the advantage of having two over one. Wouldn't a > > header like: > > MobileFrontend: 1/2 a/b/s > > work just as fine? > > > > *--* > > *Tyler Romeo* > > Stevens Institute of Technology, Class of 2015 > > Major in Computer Science > > www.whizkidztech.com | tylerromeo(a)gmail.com <javascript:;><javascript:;> > > > > > > On Sun, Feb 3, 2013 at 4:35 AM, Asher Feldman <afeldman@wikimedia.org<javascript:;> > <javascript:;> > > > > > > > Regarding varnish cacheability of mobile API requests with a logging > > query > > > param - it would probably be worth making frontend varnishes strip out name add > > > > descriptive noop query params to the requests. I.e &mfmode=2. Doing > > this > > > in > > > > request headers and altering edge config is unnecessary and a bad > > design > > > > pattern. On the analytics side, if parsing query params seems > > challenging > > > > vs. having a fixed field to parse, deal. > > > > > > > > On Sunday, February 3, 2013, David Schoonover wrote: > > > > > > > >> Huh! News to me as well. I definitely agree with that decision. > > Thanks, > > > >> Ori! > > > >> > > > >> I've already written the Varnish code for setting X-MF-Mode so it > can > > be > > > >> captured by varnishncsa. Is there agreement to switch to > Mobile-Mode, > > or > > > >> at > > > >> least, MF-Mode? > > > >> > > > >> Looking especially to hear from Arthur and Matt. > > > >> > > > >> -- > > > >> David Schoonover > > > >> dsc(a)wikimedia.org > > > >> > > > >> > > > >> On Sat, Feb 2, 2013 at 2:16 PM, Diederik van Liere > > > >> &lt;dvanliere(a)wikimedia.org&gt;wrote;wrote: > > > >> > > > >> > Thanks Ori, I was not aware of this > > > >> > D > > > >> > > > > >> > Sent from my iPhone > > > >> > > > > >> > On 2013-02-02, at 16:55, Ori Livneh &lt;ori(a)wikimedia.org&gt; wrote: > > > >> > > > > >> > > > > > >> > > > > > >> > > On Saturday, February 2, 2013 at 1:36 PM, Platonides wrote: > > > >> > > > > > >> > >> I don't like it's cryptic nature. > > > >> > >> > > > >> > >> Someone looking at the headers sent to his browser would be > very > > > >> > >> confused about what's the point of «X-MF-Mode: b». > > > >> > >> > > > >> > >> Instead something like this would be much more descriptive: > > > >> > >> X-Mobile-Mode: stable > > > >> > >> X-Mobile-Request: secondary > > > >> > >> > > > >> > >> But that also means sending more bytes through the wire :S > > > >> > > Well, you can (and should) drop the 'X-' :-) > > > >> > > > > > >> > > See http://tools.ietf.org/html/rfc6648: Deprecating the "X-" > > Prefix > > > >> and > > > >> > Similar Constructs in Application Protocols > > > >> > > > > > >> > > > > > >> > > -- > > > >> > > Ori Livneh > > > >> > > > > > >> > > > > >

Asher, I understand your hesitation about using HTTP header fields, but there are a couple problems I'm seeing with using query string parameters. Perhaps you or others have some ideas how to get around these: * We should keep user-facing URLs canonical as much as possible (primarily for link sharing) ** If we keep user-facing URLs canonical, we could potentially add query string params via javascript, but that would only work on devices that support javascript/have javascript enabled (this might not be a huge deal as we are planning changes such that users that do not support jQuery will get a simplified version of the stable site)

link from Google or even just browsing to http://en.wikipedia.org)?

On Mon, Feb 4, 2013 at 5:30 PM, Brion Vibber &lt;brion(a)pobox.com&gt; wrote: a We also need to be able to differentiate between alpha/beta/stable versions of the mobile site, without having to parse the cookie header (I believe as a result of performance constraints around this? I think the analytics team had looked into this previously).

In the case of the cookie, the header would actually get set by the backend response (from Apache) and I believe Dave cooked up or was planning on cooking some magic to somehow make that information discernable when results are cached.

On Mon, Feb 4, 2013 at 5:21 PM, Asher Feldman &lt;afeldman(a)wikimedia.org > On Mon, Feb 4, 2013 at 4:59 PM, Arthur Richards &lt;arichards(a)wikimedia.org > >> In the case of the cookie, the header would actually get set by the >> backend >> response (from Apache) and I believe Dave cooked up or was planning on >> cooking some magic to somehow make that information discernable when >> results are cached. >> > > Opting into the mobile beta as it is currently implemented bypasses > varnish caching for all future mobile pageviews for the life of the > cookie. So this probably isn't quite right (at least the "when results are Thinking about this further.. So long as all beta optins bypass all caching and always have to hit an apache, it would be fine for mf to set a response header reflecting the version of the site the optin cookie triggers (but only if there's an optin, avoid setting on standard.) I'd just prefer this to be logged without adding a field to the entire udplog stream that will generally just be wasted space. Mobile already has one dedicated udplog field currently intended for zero carriers, wasted log space for nearly every request. Make it a key/value field that can contain multiple keys, i.e. "zc:orn;v:b1" (zero carrier = orange whatever, version = beta1) If by some chance mobile beta gets implemented in a way that doesn't kill frontend caching for its users (maybe solely via different js behavior based on the presence of the optin cookie?) the above won't be applicable anymore, so using the event log facility / pixel service to note beta usage becomes more appropriate. If beta usage is going to be driven upwards, I hope this approach is seriously considered. Mobile currently only has around a 58% edge cache hitrate as it is and it sounds like upcoming features will place significant new demands on the apaches and for memcached space. If a non cache busting beta site is doable, go for the logging method now that will later be compatible with it to avoid having to change processing methods. _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

key/value pairs in the http header that we are now using for the X-CS header. Diederik _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Wednesday, February 6, 2013, David Schoonover wrote: Nope. There will be a header denoting non-standard MobileFrontend views if the mobile team wants to leave the caching situation as is. It will be a response header set by mediawiki, not varnish. The header will have a unique name, it will not share the name of the zero carrier header. The udplog field that currently only ever contains carrier information on zero requests will become a key value field. Udplog fields are not named, they are positional. the "Bypasses varnish caching" != "bypassing varnish." I don't see any use of the later in this thread, but if there has been confusion, know that all m.wikipedia.org requests are served via varnish. Nope.. We're repurposing the fixed position udplog field, not the zero carrier code header. client-side time _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Just want to summarize and make sure I've got the right conclusions, as this thread has wandered a bit. *1. X-MF-Mode: Alpha/Beta Site Usage* * * We'll roll this into the X-CS header, which will now be KV-pairs (using normal URL encoding), and set by Varnish. This will avoid an explosion of cryptic headers for analytic purposes. Questions: - It seems there's some confusion around "bypassing Varnish". If I understand correctly, it's not that Varnish is ever bypassed, just that the upstream response is not cached if cookies are present. Is that right?

- Since we're repurposing X-CS, should we perhaps rename it to something more apt to address concerns about cryptic non-standard headers flying about?

*2. X-MF-Req: Primary vs Secondary API Requests* This header will be replaced with a query parameter set by the client-side JS code making the request. Analytics will parse it out at processing time and Do The Right Thing.

I'd like to propose to define *one* request header to be used for all analytics purposes. It can be key/value pairs, and be set client side where applicable.

I think the question of using a URL param vs a request header should mainly take into account whether the response varies on the value of the parameter. If the responses are otherwise identical, and the value is only used for analytics purposes, I would prefer to put that into the above header instead, as it will impair cacheability / cache size otherwise (even if those requests are currently not cacheable for other reasons). If the responses are actually different based on this parameter, I would prefer to have it in the URL where possible.

On Feb 9, 2013, at 11:21 PM, Asher Feldman <afeldman@wikimedia.org<javascript:;>> wrote: to What is the main motivation used here? Reducing article sizes/transfers at the expense of more latency?

On 11.02.2013, 22:11 Asher wrote: frontend Since the API doesn't require pretty URLs, we could simply append the current revision ID to the mobileview URLs. for action=mobileview caches pages only in full and serves only sections requested, so no changes in request patterns will result in increased memcached usage. -- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

I'm a bit worried that now we are asking why pages are lazy loaded rather than focusing on the fact that they currently __are doing this___ and how we can log these (if we want to discuss this further let's start another thread as I'm getting extremely confused doing so on this one). Lazy loading sections ################ For motivation behind moving MobileFrontend into the direction of lazy loading section content and subsequent pages can be found here [1], I just gave it a refresh as it was a little out of date. In summary the reason is to 1) make the app feel more responsive by simply loading content rather than reloading the entire interface 2) reducing the payload sent to a device. Session Tracking ################ Going back to the discussion of tracking mobile page views, it sounds like a header stating whether a page is being viewed in alpha, beta or stable works fine for standard page views. As for the situations where an entire page is loaded via the api it makes no difference to us to whether we 1) send the same header (set via javascript) or 2) add a query string parameter. The only advantage I can see of using a header is that an initial page load of the article San Francisco currently uses the same api url as a page load of the article San Francisco via javascript (e.g. I click a link to 'San Francisco' on the California article). In this new method they would use different urls (as the data sent is different). I'm not sure how that would effect caching. Let us know which method is preferred. From my perspective implementation of either is easy. [1] http://www.mediawiki.org/wiki/MobileFrontend/Dynamic_Sections On Mon, Feb 11, 2013 at 12:50 PM, Asher Feldman &lt;afeldman(a)wikimedia.org&gt; wrote: bytes lazy direction the wrote: individual needs memcached to -- Jon Robson http://jonrobson.me.uk @rakugojon _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

It does still seem to me that the data to determine secondary api requests should already be present in the existing log line. If the value of the page param in an action=mobileview api request matches the page in the referrer (perhaps with normalization), it's a secondary request as per case 1 below. Otherwise, it's a pageview as per case 2. Difficult or expensive to reconcile? Not when you're doing distributed log analysis via hadoop.

On Mon, Feb 11, 2013 at 7:11 PM, Arthur Richards &lt;arichards(a)wikimedia.org two will and once by http://en.m.wikipedia.org/w/api.php?format=json&action=mobileview&p… clicked This http://en.m.wikipedia.org/w/api.php?format=json&action=mobileview&p… API need afeldman(a)wikimedia.org&gt; with and the individual result _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

On Tuesday, February 12, 2013, Diederik van Liere wrote: If you can point me to some examples, I'll see if I can find any insights into the behavior.

Asher, I understand your hesitation about using HTTP header fields, but there are a couple problems I'm seeing with using query string parameters. Perhaps you or others have some ideas how to get around these: * We should keep user-facing URLs canonical as much as possible (primarily for link sharing) ** If we keep user-facing URLs canonical, we could potentially add query string params via javascript, but that would only work on devices that support javascript/have javascript enabled (this might not be a huge deal as we are planning changes such that users that do not support jQuery will get a simplified version of the stable site)

link from Google or even just browsing to http://en.wikipedia.org)?