Luca Passani, the co-creator and maintainer of the WURFL mobile device characteristics database, with input from other mobile developers has published what he calls a “Manifesto for Ethical Reformatting“. The manifesto outlines some simple rules for how transcoding can work to make non-mobile websites usable on any phone, without breaking the mobile web. This manifesto is a reation to the current transcoding implementations from Vodafone in the UK, Spain, and Portugal with Novarra, and Vodafone Ireland with Bytemobile and Sprint USA using Openwave – all of which are hurting usability by reformatting mobile sites that don’t need it and by removing and changing html headers with negative consequences for content delivery services.
The rules are:
- No USER AGENT Spoofing: under no circumstances should the original User-Agent string be removed, modified or moved to a different header.
- Preserve headers: under no circumstances should a transcoder modify or delete existing HTTP headers. The addition of extra
x-*headers is admissible. - Recognize Mobile-Specific MIME-types and Document Type Declarations (DTDs): documents served with the following MIME types:
application/xhtml+xml,
text/vnd.wap.wml
application/vnd.wap.xhtml+xml
and documents served with the XHTML-Basic 1.0 or XHTML-MP 1.1 DTD:
<!DOCTYPE html PUBLIC "-//OMA//DTD XHTML Mobile 1.2//EN" "http://www.openmobilealliance.org/tech/DTD/xhtml-mobile12.dtd">
<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.1//EN" "http://www.wapforum.org/DTD/xhtml-mobile11.dtd">
<!DOCTYPE html PUBLIC "-//WAPFORUM//DTD XHTML Mobile 1.0//EN" "http://www.wapforum.org/DTD/xhtml-mobile10.dtd">
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.1//EN" "http://www.w3.org/TR/xhtml-basic/xhtml-basic11.dtd">
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML Basic 1.0//EN" "http://www.w3.org/TR/xhtml-basic/xhtml-basic10.dtd">
MUST be considered mobile-optimized and no transcoding should be applied - Do Not Adapt pictures of mobile-optimized sites: mobile-optimized sites should not have their graphics adapted.
- 30Kb Limit: pages that have a total weight of 30 kilobytes or less MUST not be adapted.
- do not reformat pages that return the HTTP header:
Cache-Control: no-transformThis also applies to the case when the header is specified through the <meta> HTTP-Equiv tag - do not reformat pages which contain the following meta tag:
<link rel="alternate" type="text/html" media="handheld" href="[url]" title="[title]" />The proxy should redirect the user to the [url] contained in thehrefattribute. - Transcoders MUST not reformat sites with URLs matching one of the following patterns:
m.*
mobile.*
wap.*
*.mobi
pda.*
avantgo.*
iphone.*
wml.*
xhtml.*
*/mobile/
*/iphone/ - Whitelists are acceptable only as inclusion list to inform the reformatter of sites which would normally escape reformatting, but which are marked as reformattable through human intervention. Use of exclusion whitelists is not acceptable.
These seem like reasonable guidelines that both transcoding proxy vendors and mobile developers can live with. I hope that Novarra, Openwave (which has been reaching out to the development community and promises significant improvements in the next release) and Bytemobile will adopt thes guidelines. It’s in everyone’s best interest; web developers, vendors of content reformatting transcoders and users. For developers of mobile content and users it insures that the mobile web and mobile content services work as designed and for transcoder vendors it lets them concentrate on the worthwhile goal of mobilizing the full-web without the negative publicity that breaking mobile services creates.
If my prbolem was a Death Star, this article is a photon torpedo.
Vodafone Itlia decided against Novarra becase it was not MSP compliant
As a tech lead if the company who is #1 content seller in France:
there is a very very simple option and I don’t understand why nobody thought of it. No worry with fake useragents, strange patterns of URLs, page sizes of this kind of stuff. Mobile web developers are adults and code their pages in cHTML, WML or XHTML-MP.
So, any site that will sent one of theses DTDs in its declaration would be considered coded for mobiles. (anyway anybody trying to write a site for mobile in HTML would be silly, since it has much much more probability to fail on mid end handsets).
Any other site would be transcoded. That’s it.
We go from 10s of complex rules with exclusions to only one : if you want to code a mobile site, use a mobile markup. That’s it.
What do you think ?
Pingback: » Verizon Installs Novarra Transcoder - Mobilytics Has You Covered!
Sean, operators and reformatter vendors break the mobile web from one day to the next and what’s yours and W3C’s answer? a spec that tells developer how to change their previously working application, while crossing their fingers and hope that the spec W3C has created gets adopted.
Even before I go into the details of why W3C’s spec is way too weak, this approach is so tragically wrong even when it comes to the underlying spirit!
How can you demand that developers change their applications when someone else has broken them without justification?
No. W3C has failed to provide the answer. The legitimate response should be: get those transcoders out of the way now! only opt-in transcoders should be allowed.
Now, the Manifesto goes out of its way to establish rules which will, at least, allow those “enforced reformatters” and the ecosystem to co-exist. You should be thanking the developer community for showing the industry the way, and what’s your reaction? you call the manifesto “misguided”.
you are misguided.
Luca
Sean,
Thanks for you comment. I think we are all moving in the direction of a workable compromise. I agree that a confrontational tone is counter productive.
Regarding whitelisting based on domain name, sure it’s a bit of a kludge, but it will have the immediate effect of stopping the transcoding of sites that match the pattern where as waiting for every mobile site in the world to add a no-transform header will take years.
Google GWT, by the way, is a great service and I have no issue with the way it works currently. It’s perfectly acceptable for an opt-in service like GWT to send its own User Agent. I only opject to content reformatting proxies which unilaterly and withourt user consent inpose themself between browser and mobile site and operate in such a way as to break existing mobile sites and services.
Yes that was not Mr. Patterson, sorry, no reason I should have not used my full name.
I also have freely mixed my own opinion on the matter with a reference to W3C work, which isn’t the right thing to do, since I don’t claim to be summarizing or representing that output.
I don’t think anyone disagrees there is an actual problem, that some new mechanism must come into existence, or an old one properly supported; the question is what exactly it is.
I support the W3C note which points out we have some good mechanisms for enabling this negotiation already. lets a page tell any potential upstream transcoder about a mobile version for example. I understand people complain that by altering User-Agent, the site can’t know what mobile phone is on the other side; I am trying to point out that it also tells you you’re talking to a transcoder! and can easily make sure you tell the transcoder to buzz off with a redirect. But that’s me.
The “manifesto” largely agrees, but goes too far in some cases in my opinion, specifying heuristics based on domain name and markup type where “real” mechanisms can and should suffice. These are the parts I find misguided; if you’re going to be telling, say, our GWT guys how to build their server, might as well spell out how to do it right eh?
Yes this is all changes we need in the transcoders… so how do we accomplish that? We might as well cast this as the struggle of the oppressed versus the tyrant. We might as well suggest that what is *really* needed is “action” now. I am sure once Novarra’s team hears they’re evil, oppressive and realize that developers think now is a time for action, that *that* will be the point when they rush out and make this change. Same for GWT.
I can’t disagree with much of the “manifesto” since it is just plain sense, but continue to regret the misguided, counter-productive tone.
Francois, there is nothing standard about transcoders. They just apply a set of, more or less smart, hacks we can call heuristics to give them more dignity than they deserver. The rules in the Manifesto come from experience and explain transcoder vendors which heuristics should be applied so that transocoders err on the side of preserving mobile content. Of course, you can call your site ‘m’ for mister, but this does not take anything away from the fact that 99.99% of the m.* sites today are mobile sites. Same logic applies to “application/xhtml+xml”. Today, it’s a great indicator that the sign is mobile and fully belongs in the manifesto.
Will W3C ever manage to create a set of rules as clear and actionable as those in the Manifesto in your opinion?
Luca
Pingback: MobHappy » Blog Archive » Reaction To The Latest Transcoding Mess
Just to point out that W3C is working on such guidelines in a dedicated Content Transformation Task Force (I happen to lead that task force), within the Mobile Web Best Practices Working Group. The guidelines document’s still a draft though:
http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/CT/editors-drafts/Guidelines/latest
The current approach taken by the guidelines is to recommend content tasting by the transcoding proxy:
– first, sending the original HTTP request to the server
– then, if the server replies with a “I don’t really know how to answer your request” response, sending an HTTP request with modified headers.
It’s along the lines of “No User-Agent spoofing unless it can’t be avoided”. See the following post for a bit more on that:
http://www.w3.org/blog/MWITeam/2008/03/25/content_transformation_guidelines
“Cache-Control: no-transform” is part of these guidelines, so as the use of the “link” element, and the need to append a “Via” header as stated in the HTTP RFC. I would be more reserved regarding the use of some of the other rules listed here as “MUST”: while it certainly is useful to take into account that the URI starts with “m.”, I could own a “m.daoust.org” egocentric web site where “m” is the standard French abbreviation for “Mister”. Nothing really mobile. Mere exception to the rule? Maybe. Mobile-optimized pages may be under the 30Kb limit, but are all pages under the 30Kb limit mobile-optimized? The “application/xhtml+xml” content-type may mostly be used in the Mobile World today because of its lack of support in Internet Explorer, but is that enough to say that resources served with that content-type must not be transcoded?
Jason, I think Sean is Sean Owen from Google.
Ed,
Transcioders do serve a useful purpose. There are several excelent transcoders that anyone can use to mobilize any site that needs it. I particularly like Mowser.com, google.com/gwt/n and skweezer.net.
The biggest problem I have with the Openwave transcoder as implemented by Sprint is that it’s use is mandatory, even it doesn’t work on particular site there is no way to get to the original un-transcoded version the the site. In my experience, Openwave fails to work properly, or at all, on a signifigant number of sites including most third party ringtone and wallpaer download vendors.
In the end I as a user suffer because many mobile sites and services thaI use no longer work on my Sprint phone.
The operators obviously bought and installed the transcoders because they serve a useful purpose – allowing users to access sites that wouldn’t otherwise be accessible. I sometimes wonder if the real objection behind these complaints is that content providers who have created custom mobile sites are angry that they are losing their captive audience.
Ricky, did you notice that m.* is the FIRST bullet?
Luca
By the way, Sean, what do you say to the fact that Novarra has joined W3C only to be able to claim that they support standards when they sell proxies configured to disrupt the mobile web?
Luca
Sean, as someone who has worked with mobile and understands developers, I cannot believe that you are still not seeing the issues that these aggressive reformatters cause to the whole ecosystem.
Why don’t you wake up and go tell Google that they should be joining the fight for a neutral mobile web, rather than bitching that the manifesto (which simply collects what developers have been saying all the way) is misguided?
Luca
Sean, if this is Sean Patterson from Novarra, I think your obvious bias in this matter makes your opinion unreliable. Not only that but I think you should have declared your vested interest on this subject in your comment.
Sean,
I think except changing the User-Agent header, the Google Web Transcoder is, in my opinion, being a good transcoder. The spearhead here points directly to carriers and their misguided installation of transcoders, not third party utilities like the GTW or Mowser. As Dennis said, most operator-installed transcoders are forced into the customer, while the GWT stays a third party transcoder that mobile site owners can opt-out easily. Don’t get all defensive here.
It’s not ideal to depend on a fixed list of host name, however this is but one mechanism with with we are asking the transcoder vendors to implement to avoid transcoding sites that are supposed to be mobile-oriented, and to avoid modifying the incoming request like the User-Agent header. While the GWT do modify the User-Agent header, you send the original User-Agent in a different header. This is not ideal because it involves changes to the site’s code to obtain the original User-Agent, but for me it’s a compromise I’m willing to make.
Sean ,
The problem with changing the user agent is that these transcoders are installed by the operator and by default ALL requests pass through them. In the case of Sprint there is no way for the user to opt out.
When a request hits a server that does it’s own content adaptation the server has no way of knowing that the request is coming from a mobile phone and serves content designed for desktop browsers. This is bad enough in the case of an informational web page but much worse if the target is a mobile content download like a ringtone, theme or game. The mobile web service can’t deliver the correct content because it can’t identify the phone.
As for the transcoder respecting a header like
cache-control: no-transform, that’s good too, transcoders should do that. But it doesn’t fix the problem of the 10’s of thousands of existing sites and services that are broken today because of these new transcoders. This problem didn’t exist before Novarrra, Openwave and Bytemobile created it, it’s not reasonable to expect all those web sites to sudenly have to add a new header to solve a problem that they didn’t create. These vendors introduced sumething that’s broken, it sould be their responsibility to fix it.Great post, Dennis, though Luca should also add m.* to the last bullet, as that’s quickly becoming the popular alternative.
These guidelines are fundamentally misguided. You want the transcoder to tell the server it is a mobile phone? It’s not. Why change the UA string? No, what you want is a way to say “please don’t transcode me” and have the transcoder redirect the browser directly to the target.
This recommendation is in the list, along with many other poor ones. How is it a good idea to depend on host name, and an incomplete hard-coded list at that?
For a simpler and more reasonable treatment, see:
http://dev.mobi/files/ContentTransformation_consultative.html
or maybe:
http://www.w3.org/TR/ct-landscape/