Verizon Using Novarra’s Transcoder

Verizon LogoThe largest US mobile carrier, Verizon Wireless has started funneling traffic between Verizon feature phones and the  web through a transcoder from Novarra.  Verizon calls the service “Optimized View” and has added promotional information and a FAQ to their customer website  There is also a page on the carrier’s developer site which has links to a opt-out form and to a  PDF document detailing the rules that the transcoder uses to determine which sites to transcode.

Given the disastrous effects that VodaFone’s UK roll-out of Novarra had on mobile sites and services, I’m rather apprehensive about this.  In the PDF, Verizon/Novarra say that they won’t change the User Agent header  or transcode sites that have submitted an opt-out request or have a URL corresponding to one of these patterns:

*.mobi,  m.*, mobile.*, avantango.*, wap.*, iphone.*, <domain>/m/*, <domain>/mobile/*, pda.*, wireless.*, wml.*, xhtml.*, <domain>/m/, <domain>/gmm/, <domain>/portable

For sites that have not opted out or do not use one of the mobile URL patterns, the transcoder will change the User Agent and a number of other  headers to ones mimicking a desktop browser. However the site will not be transcoded if it meets any of the following conditions:

  • It uses one of the following mobile specific DTDs:
    • <!DOCTYPE html PUBLIC “-//OMA//DTD XHTML Mobile 1.2//EN” “http://www.openmobilealliance.org/tech/DTD/xhtml-mobile12.dtd”>
    • <!DOCTYPE html PUBLIC “-//WAPFORUM//DTD XHTML Mobile 1.1//EN”
      “http://www.wapforum.org/DTD/xhtml-mobile11.dtd”>
    • <!DOCTYPE html PUBLIC “-//WAPFORUM//DTD XHTML Mobile 1.0//EN”
      “http://www.wapforum.org/DTD/xhtml-mobile10.dtd”>
    • <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML Basic 1.1//EN” “http://www.w3.org/TR/xhtml-basic/xhtml-basic11.dtd”>
    • <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML Basic 1.0//EN” “http://www.w3.org/TR/xhtml-basic/xhtml-basic10.dtd”>
  • It uses one the following MIME types:
    • Application/vnd.wap.xhtml+xml
    • Text/vnd.wap.wml
  • It has a self referencing link rel tag:
    • <link rel=”alternate” media = “handheld” href=mymobilesite.com/>
  • It sends a “Cache-Control: no-transform” header.

I don’t have a Verizon account but I did do some limited testing in a Verizon store.  I used a Motorola V950 which runs the Openwave 6 browser. I mostly tested my own sites.  The transcoder seems to follow the rules in the PDF.  None of my sites that use a mobile URL pattern, Doctype or no-transform were transcoded.  A couple of test pages I created as copies of mobile pages but with none of the Novarra trigger conditions where transcoded even though they were under 10KB.

I didn’t really test the quality of the transcoded pages but noticed that a two line navigation bar is added to the top and bottom of the page and pagination is used.  I thought the transcoded version of howardforums.com was reasonably usable, but my test sites, which were under 10KB, were needlessly split into two pages.

According to Verizon, the transcoder does offer users the ability to view the original version of any transcoded page by clicking a “Turn off Optimization” link in the bottom navigation footer. As a user, I consider this a very desirable feature which all transcoders should implement.

Secure HTTPS sites are transcoded, except for “banking sites”.  I wonder how Novarra identifies banks?  Users are warned that their security may be compromised when visiting a non-banking secure site through the transcoder.  I didn’t try any secure sites so I don’t know what this warning looks like.

Transcoders are the bane of mobile web developers. This one will probably not be as disruptive has the Vodafone UK one was because Novarra has learned some lessons from that debacle and is doing a better job of detecting mobile sites and also because the development community is more aware of the transcoding problem and how to work around it.  There should be no impact on m. and .mobi sites and others with a recognized mobile URL pattern. Sites that use a mobile doctype and don’t do browser detection will also be unaffected.  However, sites on .com domains that rely on the User Agent or other headers to optimize for specific handsets need to either get on the white-list or start checking the “X-Device” headers where the transcoder is putting the original values of the headers it’s modified as follows:

Original Header Renamed Header
User-Agent X-Device-User-Agent
Accept X-Device-Accept
Accept-Charset X-Device-Accept-Charset
Accept-Encoding X-Device-Accept-Encoding
Accept-Language X-Device-Accept-Language

The Verizon transcoder can be identified by the Via: header which it sends, “1.1 Novarra (Vision/7.3)
X-Mobile-Gateway: Novarra-Vision/7.3 (VZW; Server-Only)”

The Vodafone transcoder broke many content (ring tones, game, application) download sites.  I don’t see that happening with this one because Verizon already blocks almost all off-portal downloads.

The problem with transcoders changing headers is that it forces millions of mobile sites around the world to change their code in order to keep the same functionality. It’s an annoyance for capable mobile web developers who somehow find out that another  transcoder has been deployed somewhere in the world. The real problem is with sites that aren’t actively maintained or whose developers don’t actively follow the doings of the W3C or monitor the mobile development mailing lists, forums and blogs. Which, I suspect, is the majority of sites worldwide.

Verizon and Novarra claim that their transcoder partially follows the recommendations of the W3C Content Transformation Guidelines. However those guidelines are still a work in progress and the issue of whether transcoders should alter the User-Agent except in a very limited number of cases such as when a site returns no content when presented with a mobile browser User-Agent is still under discussion. It would be far better for the health of the mobile web if transcoders did not alter the User-Agent and other headers as recommended by the Rules for Responsible Reformatting: A Developer Manifesto a document that evolved from discussions on the wmlprograming Yahoo Group and which has been signed and adopted by several transcoder vendors, but not by Novarra.

Related Posts:
Vodafone’s Heavy-Handed Transcoder
How Web to Mobile Transcoding Should Work
Opting Out of Transcoding
OpenWeb and InfoGin Adopt the Developer Manifesto!

94 thoughts on “Verizon Using Novarra’s Transcoder

  1. @Tom

    > when a user has specifically asked for a transcoded experience,
    > CTG permits transcoders to change the UA. Luca, I think I’m right
    > in saying you object to this on the basis that the definition of
    > “opting in” is subject to interpretation?

    yes and no. It is true that I object that “opting in” gives too much wiggle room to abusers, but I also think that, as a general rule, users should get what the content owner has intended them to get.
    Now, I don’t think it would make much sense for a spec to try and regulate what users should do (or the same logic would lead to requesting that users should not avoid pop-up windows with ads, download MP3 with peer2peer, use a foreign proxy because on-line betting is illegal in their countries and endless more examples, also including chinese censorship). Having said this, I don’t think that a proxy spec should explicitly allow those cases just because users will do it anyway. There is a lot of difference between tollerating a phenomenon and creating rules that assume that the phenomenon is an accepted practice (which is sort of similar to why I don’t think CTG should mention HTTPS rewriting at all).
    From this viewpoint, I think the differences between CTG and Manifesto are much deeper than you keep claiming.

    @legalalien

    > A majority of sites, in my experience, make this trade-off,
    > and leave important functionality off limits to the mobile users.
    > Luca and others say that it’s the content owners’ prerogative.
    > But we’ve heard this argument many times before, for example,
    > when web sites refused to work with Netscape or Firefox
    > browser used by many Linux distributions.
    > It wasn’t right then, and it isn’t right now

    legalaline, I think you still need to explain why this is not right. So far, you have failed to convince me. With respect to your example, people and companies stopped creating MSIE-only sites because it did not make business sense to them, not because some company created a product to reformat MSIE sites and ISP paid $millions to deploy those proxies. Of course, companies are creating richer and richer apps these days to support users with increasingly powerful mobile devices, which is sort of similar to the “escape from MSIE-only” phenomenon you are referring to in some way.

    > It’s easy to recognize that your site is being accessed
    > by Novarra, so why not just account for it and serve a
    > version of your web site that works well with it?

    There are many subtle points to address here.
    First, novarra has arrogated the right to change an established standard from one day to the next. If developers rushed to recognize Novarra’s extra headers today, who would prevent company Padovva from performing even more deadly tricks, and refer back to Novarra as a moral justification for their abuse?
    Second point, why is Novarra stripping off the UA-Prof ‘x-wap-profile’ header?
    I agree that most mobile companies will look at the UA string to recognize mobile devices, but still UAProf is used by others (and don’t forget that UAProf is a widely adopted OMA standard). This tells me a lot about Novarra’s real goal to be fooling as many existing sites as possible, while using the ‘x-device-user-agent’ as an excuse that developers can still have a workaround if they really want. But this is an obvious attack on the rights of content owners. Sticking to discussions like the ones which characterized CTG means losing site of what the real issue is, i.e. whether content owners can retain the rights to their content or not.

    Anyway, thank you for bringing up all your points. I still think I am right, but at least I found the discussion much more honest and open than what one might ever hope for in W3C.

    Luca

  2. Terren, the manifesto has no provision for using modified UA string, even with user consent. In fact, it states in big, bold letters that modification is not allowed under any circumstances. At least CTG mentions “user preference” once, I think, in their guidelines.

    I think you’re missing my point: a transcoded site may *look* bad, but it’s more or less complete, while mobile sites look pretty, but usually contain only a subset of the original site. A majority of sites, in my experience, make this trade-off, and leave important functionality off limits to the mobile users. Luca and others say that it’s the content owners’ prerogative. But we’ve heard this argument many times before, for example, when web sites refused to work with Netscape or Firefox browser used by many Linux distributions. It wasn’t right then, and it isn’t right now.

    Now, what prevents you from treating the transocders, including the one from Novarra, as just instances of a browser? In order to deliver good mobile experience on a variety of phones you must already tailor your mobile web site to the requesting handset – otherwise, your users end up with the lowest common denominator. When new phones come to the market, I assume you update your mobile site to account for their capabilities and don’t expect Access, Opera, Jataayu or Teleca to compensate you for the extra work. It’s easy to recognize that your site is being accessed by Novarra, so why not just account for it and serve a version of your web site that works well with it?

  3. Terren: when a user has specifically asked for a transcoded experience, CTG permits transcoders to change the UA. Luca, I think I’m right in saying you object to this on the basis that the definition of “opting in” is subject to interpretation?

    Personally, I think this is very reasonable: if I, as a user, ask for a transcoded desktop site I should be able to get one – if this requires changing a UA then so be it, it’s my decision. CTG goes a little bit further, in requiring that even when this happens, if a mobile version is available I should be given the option of viewing it.

    CTG and the Manifesto agree in both spirit and in detail that the automatic opt-in to transcoding which Vodafone/Novarra visited upon us all is NOT acceptable.

  4. legalalien: I don’t think anyone here is talking about the case where the user opts in to a transcoded experience. If the user decides they want a transcoded experience, they know best. We’re talking about when the operator decides for the user that transcoding the full site is preferable to a tailored mobile experience, if one has been crafted by the content provider.

    I can tell you that with our product, the mobilized site is quite usable, but the transcoded version looks terrible… that’s the whole point here! We know which one looks better, and Novarra doesn’t. If Novarra makes a mistake and serves the crappy looking one, it doesn’t affect Novarra, it affects our customers, and we’re the ones that take the support calls and/or lose the business.

    It’s an externality for the transcoders/operators, meaning that the transcoders do not have to bear the burden of the costs they incur. Externalities are usually dealt with by forcing companies to bear that burden (e.g., fining industrial polluters), usually through government regulation. In this case we can potentially head off this abusive behavior with unambiguously worded guidelines, but the CTG does not take a strong enough stand to protect the content providers. That is why the Manifesto is more in line with my company’s and my customers’ needs.

    Transcoders have their place, but when content providers have spent resources to create a usable mobile experience, that needs to be protected, or everyone loses (except the transcoders of course).

  5. @legalalien

    > I’m not sure why it is necessary to repeat this simple
    > fact: I am not affiliated with Novarra

    Look. I am not going to discuss this endlessly. You started it by stating that you were not affiliated with Novarra with great emphasis, while signing yourself with an alias. It turns up you were working for Novarra in the past. These are facts. Who reads us will draw their own conclusion on whether you were totally honest or only “technically honest”.

    > For the last 2.5 years I’ve observed your crusade purely
    > as a user and an independent observer.

    The crusade started for me in September 2007. That’s one year and two months.

    > In that respect, I am certainly far less affiliated
    > with the industry than people who mobilize web sites
    > or compile databases of device capabilities for a living.

    Trying to undermine my credibility is consistent with what now appears to be part of a certain Novarra company culture. Be careful though, Novarra’s CEO already tried to go down the same path with poor results:

    http://www.mobilemarketingmagazine.co.uk/2008/05/talking-transco.html

    > Now, let’s agree that there is really no problem
    > with the sites accessed via a mobile URL – Novarra
    > transcoder does not alter UA string when requesting
    > those sites.

    the problem is that non-mobile specific URL can still be used for a totally legitimate mobile site. Our point is that Novarra should *never* spoof the UA.

    > Having a proper DTD or a MIME type will ensure no content transformation.

    The DTD and MIME type may be function of the User-Agent string. So if you change the UA string, you will effectively prevent sites from providing mobile content when this has been created.

    > Now, imagine if you came to an open supermarket,
    > but the owner would only let you shop in a small,
    > separate section of it with a limited selection of
    > products…only because you drive, say, a Fiat Punto.

    Now,as I explained before, I do not agree with this approach because I still think that the content owner is king over how they want to present their content to users.
    Having said this, your analogy would make some sense if transcoders were totally opt-in. What Novarra does is different though. They place the transcoder right in the middle of each and every HTTP request, so regular users won’t even get a chance to look at the specific shop the owner has set up for them. Of course, Novarra makes sure users have some obscure settings somewhere which gives people a theoretical possibility to opt-out, but 99% of the users won’t have a clue.
    This explains why OperaMini, which also transcodes in a way, did not cause the fury that Novarra caused with its very arrogant approach. When you position a transcoder like this, the only thing you can do is to err on the side of non-transcoding whenever in doubt that content may already be suitable for mobile, which is what the Manifesto achieves.

    > When I type http://www.somesite.com, I want to access
    > a *publicly available* http://www.somesite.com,
    > open to anyone with IE on Windows. There’s no
    > ‘exploiting content without consent”.

    Unless the transcoder is stripping out the advertisement which constitutes the business model of the website. That’s when site owners get, mildly put, irritated.

    > I never said that Novarra is the best answer for my needs.
    > In fact, transcoded sites do look terrible more often than not.

    This is consistent with my experience too.

    > This is a key difference: I don’t access the site with
    > my phone; I ask the transcoder to access the site and
    > send the content to my phone.

    Many mobile sites these days present a link that offers access to the full web site. This is OK, because it has been put there by content owners. Trying to do this behind their backs is wrong, wrong, wrong.

    > If the manifesto would take into account the needs of
    > users (including ex-Novarra web client developers and PMs),
    > I’d support it. Instead, it only seeks to protect revenue
    > streams of people who restrict access to information.

    you seem to forget two key facts here:

    – those who “restrict information” are actually the owners of that information, so they have every right to restrict it if they wish.
    – those who try to protect their “revenue stream” are more often than not tiny companies whose revenues amount to the one of the 7-eleven shop around the corner. Compare this to the revenue of the operators who install Novarra (billions of dollars) and you’ll also see why it is so incredibly unfair that those who already eat most of the cake go out of their way to also collect the crumbles.

    Of course, transcoding the way Novarra does it is harmful for the ecosystem, and will damage the very ecosystem which operators have every interest to keep healthy.

    Luca

  6. I’m not sure why it is necessary to repeat this simple fact: I am not affiliated with Novarra. I worked on their mobile *client*, which is a different product from the transcoder, and I left before they signed VF and before this whole thing with UA blew up. For the last 2.5 years I’ve observed your crusade purely as a user and an independent observer. Novarra does not pay me, and their future has no effect on my financial well-being. That’s my definition of ‘not affiliated”. In that respect, I am certainly far less affiliated with the industry than people who mobilize web sites or compile databases of device capabilities for a living. If you wish to continue patting yourself on the back for uncovering my secret past, be my guest, but I will talk about transcoding instead.

    Now, let’s agree that there is really no problem with the sites accessed via a mobile URL – Novarra transcoder does not alter UA string when requesting those sites. Having a proper DTD or a MIME type will ensure no content transformation.

    So, the UA string becomes important when the user types http://www.somesite.com, and the server uses UA string to serve a mobile version of content, presumably tailored to the capabilities of the phone’s browser. Now, your comparison to breaking into a supermarket is completely off-base here. When I type http://www.somesite.com, I want to access a *publicly available* http://www.somesite.com, open to anyone with IE on Windows. There’s no ‘exploiting content without consent”. I want to access the same information available to others with a different browser. Instead, I am served a ‘mobile’ version of the site. Now, imagine if you came to an open supermarket, but the owner would only let you shop in a small, separate section of it with a limited selection of products…only because you drive, say, a Fiat Punto. Because, you know, you can’t possibly afford the same products that people driving Fords and Volvos can, and the supermarket owner knows better what Fiat drivers need.

    I never said that Novarra is the best answer for my needs. In fact, transcoded sites do look terrible more often than not. But, I get the information I need because Novarra server acts on my behalf to get the full site. This is a key difference: I don’t access the site with my phone; I ask the transcoder to access the site and send the content to my phone.

    The ONLY reason I access the web on a small screen is to get to the information I need when I need it (as opposed to ‘when I get home’). And if the information is not accessible because I can only get a version of the site “optimized” for mobile phones, it hurts adoption of the mobile Internet even more. If the manifesto would take into account the needs of users (including ex-Novarra web client developers and PMs), I’d support it. Instead, it only seeks to protect revenue streams of people who restrict access to information.

  7. @Tom

    Tom, if we were in a world of gentlemen, CTG would be more than enough. In fact, there would be no need of CTG, since Novarra wouldn’t be spoofing in the first place.

    Since it’s a different world, we need to take a bit more “holistic” approach than assuming that everything can be framed and ruled by a standard document.
    The Manifesto takes a clear stance. You either support it or you don’t. CTG leaves to much wiggle room.

    Now, do you want to speak with one voice? please go back to CTG and demand that UA is not spoofed in any case on behalf of developers.

    > My point is that the hundred signatures on the document
    > don’t translate into a mandate to speak for the
    > thousands who didn’t sign it.

    bring on the thousands developers who did not sign the Manifesto because they would like to sign something else (and no, ex-Novarra Product Managers don’t count)

    Thank you

    Luca

  8. @legalalien

    > I left Novarra 2.5 years ago,

    well, you could have said it to start with. Having been a product manager at Novarra is certainly different than being a totally independent observer, don’t you think.

    Anyway, I was not trying to avoid your questions. I just assumed that they were clear after repeating the same concepts one thousand times. Anyway, if you insist, I will address your points again.

    > I worked with Sean Patterson and I don’t want to see his
    > (or anyone else’s, for that matter) name next to profanities.

    I met Sean once (very briefly) and I agree that he looked like a totally nice guy.
    Problem is he chose the wrong company or, at least, he did not do enough to convince them that they were adopting a very wrong strategy. Anyway, I never called him anything. In fact, I never called anyone personally anything in this whole matter. Only organizations and groups. Even when I was referring to those W3C freakers, I wasn’t thinking of Sean particularly. Novarra may be evil, but at least they have been coherent all the way.
    The adjective was meant for those who had preached how anyone, even those with no access to HTTP request/response headers (!), should be able to create BP-compliant content, yet they were available to make believe all of that never existed one second later, just because Vodafone told them so. More than that, they adopted Novarra’s recommendations verbatim into CTG. Very spineless, as Alex rightly commented.

    > perhaps this crowd could figure out how to respect
    > the users desire to get to the information they
    > need using the device of their choice

    Now, here you are assuming that Novarra is best answer to users’ needs. I disagree strongly, but if, as an accademic exercize, we assumed that you were right, Novarra would still be in the wrong. The fact is that the content on the Internet does not come for free. Each and every HTML page out there is the result of humans and companies investing time and resources to create their content. It’s their money that goes wasted if the content is not popular enough. For this reason, it’s not fair of another company to exploit that content without consent, not even in the name of the user. Of course, it’s in the interest of the content owner to make sure that mobile users also get serviced, but this is up to the content owner to decide. You wouldn’t break into a closed supermarket by night to get the milk, just because you’re going to leave 5 bucks by the cash register, would you?

    Anyway, as I said, this is an extreme case, because when Novarra spoofs the UA, websites are unable to recognize mobile devices and offer a customised experience (and no, placing the UA in a different HTTP header is not good enough). If you spoof (and do it in different ways on a operator by operator way), you effectively break the mobile web as a platform. This will make it harder (and less attractive) for developers to build for mobile, hence less mobile-optimised content for users, for operators and for everyone else (except Novarra, of course, which would obviously love to see as little mobile-optimsed content as possible, so they can transcoder everyone’s website and their dog’s website too).

    > Transcoding by default, with no choice to bypass the transcoder,
    > may be evil, but it has nothing to do with Novarra, Openwave, InfoGin,
    > Opera Mini or any other similar service.

    It would be easier for you to argue this if it wasn’t that InfoGin and Openwave have acknowledged that spoofing the UA is a killer for mobile sites and agreed to sign the Manifesto. Yet, also Openwave and Infogin have VCs and investors behind them…

    > carriers and content providers use technology as they see fit

    theoretically correct. Practice is different though. Novarra hires the best salesmen around the globe. Salesmen will by-pass the people with a clue and talk to the clueless (but powerful) execs, convincing them that a reformatted web is what they (and the users) want. Blogs like this will help set the record straight.

    And now , some points in your previous message:

    > Now, can somebody explain […] what’s so sacrosanct about UA?

    Yes, absolutely. There is no reason whatsaever to spoof the UA, unless you want to trick a web application which might have been tuned to recognise mobile devices. So, if you spoof the UA, you are implicitly saying “I don’t care if you want the end user to get a mobile site even if you have one. The network operator is smarter than you and we will extort your web content, reformat it and give it to the user”. How can you be surprised that everyone goes up in arms in this situation?

    > If you want to preserve ad revenue for your mobile site, add Cache-Control
    > or instruct transocders to pass your web site through by any of the
    > available means, and stop whining.

    without User-Agent you won’t even know you need to send Cache-Control, not to mention the fact that cache-control will break WML. We are not whining. We are demanding respect. (also, if you don’t want to be called funny names, I would be careful about the tone here).

    > Finally, a disclaimer: I am not affiliated with Novarra
    > or any other transcoder developer, any carrier,
    > any browser developer, or W3C.

    yes. We figured

    Luca

  9. Chris, David, Alex, Terren, Tommy: you all go to great lengths to emphasise the damage transcoders are doing to the industry. I agree with you: supporting the CTG is not, contrary to what you might read, the same as supporting the transcoding industry. My company builds mobile web services; my customers suffered when Vodafone launched the Novarra proxy too.

    In particular, Terren: the CTG doc specifically curbs the instance you mention, “when an operator transcodes by default, without the user’s awareness.” On the SHOULD vs MUST, please note that a section on compliance is being discussed which would require a deployment to specifically justify why it hasn’t met a SHOULD – precisely to tighten up and hold deployments more accountable.

    Tommy: you ask if “CTG are pro forwarding of the UserAgent or not”. The CTG specifically prohibits alteration of any HTTP headers, except in limited circumstances (such as the user asking for it) – and even then requires vendors to inform the user and let them access a version that *isn’t* transcoded.

    I suggest you read the document at http://www.w3.org/TR/ct-guidelines/ – it’s a work in progress, and being a standards doc quite a dry read compared to the Manifesto, but it covers an awful lot of ground.

    It’s worth spending some time to read it before criticising it, particularly given some of the hysteria this debate has caused. In particular it places a number of significant responsibilities on transcoder deployments, even when they do transcode, which you may not be aware of.

    Finally, with my “invisible” comment: I apologise if this denigrates the support of those who have signed the Manifesto. My point is that the hundred signatures on the document don’t translate into a mandate to speak for the thousands who didn’t sign it.

  10. @Luca: I left Novarra 2.5 years ago, so no, I am *not* affiliated. As in, I care little whether Novarra swims or sinks, though I worked with Sean Patterson and I don’t want to see his (or anyone else’s, for that matter) name next to profanities.

    I do know a bit about how transocders, devices databases, mobile web sites, etc. work, in part due to my past employment, and in part because I’ve been using mobile Internet since the very early days of WAP. Now, we all see that you can use Google. Can you also address the questions I raised?

    All I care about is getting access to the information I need. And I can tell you, that I’m sick and tired of trying to look something up (say my itinerary or box score of a baseball game) on my high-end feature phone, only to get a “mobile” version that has been “optimized” for a RAZR and does not show anything beyond the very basic information. Transcoders, whether connected to Yahoo or Google search, VZW, Sprint or VF portal, or through a Java app, fetch the information I need on my behalf. That’s why I reply to articles like this one, which seem to consider only one side of the story…though certainly the loudest one.

    Instead of concentrating on why replacing UA string is a capital offense, perhaps this crowd could figure out how to respect the users desire to get to the information they need using the device of their choice. Not only to what *you* think they need on a mobile phone, not only to what *you* think their mobile phone can handle, but to any piece of information that is otherwise available for public consumption.

    Transcoding by default, with no choice to bypass the transcoder, may be evil, but it has nothing to do with Novarra, Openwave, InfoGin, Opera Mini or any other similar service. These companies provide technology; carriers and content providers use technology as they see fit. Keep that in mind, too.

    Cheers,

    legalalien

  11. Hi Dennis – thank you for this artice, I’ve just read it and I think that assides from being an all round great article, I think it will be really useful for people who weren’t around when Novarra messed stuff up last time, as they can see a perfect summary of the points surrounding the whole issue about transcoding.

    It has also served as a great sounding board for the big names in mobile to show / remind us of their stances on this issue. Furthermore, for me personally it has brought to my attention this new CTG thing which I shall investigate asap. I agree with your statement that “It would be far better for the health of the mobile web if transcoders did not alter the User-Agent “ so I would quite like to know if CTG are pro forwarding of the UserAgent or not – since unfortunately the intent of the CGT expressed in this discussion has been overshadowed by the more serious debate as to exactly who Luca Passani at some point referred to as an asshole and why and when.

    Although for that matter, I for one am slightly disappointed that Luca did not use this particular oportunity to explicictly call Novarra, and indeed anyone who tampers with the UA, a bunch of total assholes*.

    *NB: Although I am more than happy for Luca to represent me and any of my fellow nameless mobile web developers, in attacking those who attack our sites, I don’t quite have the guts to do this myself. So where I use the word ‘assholes’ I am not saying that Novarra are actual Assholes, (i.e. as in the kind of people who just want to shit all over everything), I’m just using it as a short summary for ‘the kind of people who make it so that we don’t know what phone our customers are using so we can’t deliver them the correct content for their device, which in turn makes our normally happy customers call us up and call us assholes’!

    Anyway – great article Dennis – look forward to reading more !

    With kind regards, Tom

  12. @legalalien

    >Finally, a disclaimer: I am not affiliated with Novarra or
    > any other transcoder developer, any carrier, any browser
    > developer, or W3C.

    maybe you aren’t, but if you signed with your real name we could check whether it’s true or not.

    Luca

  13. I’m a mobile developer for a startup company. I signed Luca’s Manifesto because it is clearly in the best interests of my company, and of our customers, to force transcoders to respect tailor-made mobile web content.

    Transcoders spoofing user agents and forcing my customers to get the web-version of our app would result in a significant loss of usability for our users and thus a loss of money for us. It goes beyond screen size and layout – mobile apps are used in a different business context, a fact which is lost on pro-transcoder folks.

    Furthermore, transcoders have and will insert ads, which is a clear case of IP theft, when the user is not actively choosing to transcode. Transcoders also think it fine to break end to end security on https transactions. The CTG document does not prohibit any of these practices, in the all-too-common scenario when an operator transcodes by default, without the user’s awareness.

    The CTG’s ambiguity regarding altering the user agent (SHOULD NOT instead of MUST NOT) means that the document does not represent my interests, nor my customers’ interests, because it allows transcoders to exploit loopholes and claim adherence to the document, which is precisely what is happening with Verizon/Novarra. Luca called it.

    For those of us with a real stake in the outcome here, it is appalling that the CTG continues to take such waffling positions on what are so clearly indefensible practices.

  14. Can there be a discussion about transcoders that does not disintegrate into a bunch of name calling rants? My hopes are slim at this point.

    Can we at least stop pretending that we’re in some noble fight against brutal oppressors? Everyone involved in these discussions makes a living off mobile internet access. Verzion, Yahoo, VF, T-Mobile US have customers to milk, while Novarra has VCs to pay off. The people who signed The Manifesto (not the one that begins with a spectre story) likely make money creating mobile versions of web sites, selling ringtones and other content for mobile consumption, or supporting the first two categories in some fashion. This is a *business* for everyone involved, so stop playing innocent victims, please.

    Now, can somebody explain (without profanities, please, I have kids running around my laptop) what’s so sacrosanct about UA?

    Yes, I understand that a proxy shouldn’t modify content, but transocders are not simple http proxies, they’re agents fetching content on behalf of users.

    Yes, I realize that if UA is replaced, it’s impossible to determine which phone the request came from, and therefore impossible to serve a version of content customized for that phone. For some content, e.g., ringtones or themes, knowing the exact phone model is very important, but when I look at a web site, I don’t always want to access some subset of the desktop site based on what somebody thinks I need on the go or, better yet, on incomplete or incorrect information about capabilities of my browser that some device database contains. However visually appealing that subset it, or however open or complete that device database pretends to be.

    If you want to preserve ad revenue for your mobile site, add Cache-Control or instruct transocders to pass your web site through by any of the available means, and stop whining. Your mobile site should be doing one of those things anyway!

    Finally, a disclaimer: I am not affiliated with Novarra or any other transcoder developer, any carrier, any browser developer, or W3C.

  15. @Sean

    Re: “That said I think that iPhone + Android are the very ‘beginning of the end’ of the era of specialized content for mobile. This issue will slowly fade anyway”

    Isn’t iPhone and Android, proving that people prefer their mobile experiences designed for small screens and incorporating device and carrier network capabilities?

    I never approached this from the perspective of developing specialized content (other do so, but it;s not what we do.)

    I’ve always believed the best approach to developing for mobile was to deliver content through a web browser and enhancing it for mobile context. (For example, Android automatically links addresses within text to Google Maps, with your current location identified via GPS.)

    As long as there are abusive implementations that change what those of us who do not control the pipe or the browser choose to deliver the issue will not fade away.

    Cheers,
    David Harper
    Founder & CEO, Winksite
    One World. No Borders. ®

  16. > but also an invisible army of developers behind you.

    Tom, unless you’re trying to say something very subtle that I’m not getting, you know that’s simply not true, is it? I mean, that statement is the exact opposite of the truth. There is in fact a highly visible army of developers right behind Luca and everything he’s doing wrt fighting transcoding abuse (note: not fighting transcoding, fighting the *abuse*).

    Anyone could go right now and look at the Manifesto document signatories, or the archives of Mobile Monday London messages, or the archives of wmlprogramming, or the comments of various blog posts on this subject since Novarra started the abuse, and see that there are in fact numerous people right behind Luca, backing him up all the way. And these are people like yourself with weight, experience, and voice in the industry, often running successful companies, some of significant size.

    As regards “infighting” and “abusive attitudes”, I see only a strongly worded defence of principles – principles that Novarra (and others) are abusing. Abuse that other weak minded individuals are lying down and accepting under the guise of “not wanting to get involved”, “too immature”, “load of fuss about nothing” etc.

    Au contraire. This is not a storm in a teacup. People can skirt around the issue and try to convince themselves this all doesn’t matter, but if we don’t put up a fight now, and at least try and curb this abuse, then in 5, 10, 15 years, we’ll be trying to extracate what the mobile web should be from the mess that it has become. The scenario we face now should not be surprising. Abuse happens in other industries, in other scenarios. Heck, sometimes years later films get made about it, the abuse was that significant. The point is, lets not kid ourselves. This is significant. The internet’s going mobile. The web is at the forefront of that push. And here we have an attempt at an early hostile takeover. In days gone by, in other scenarios, a physical response would have been appropriate against these sort of abusers. Luckily we don’t live in that world any more, but a strong response is still necessary. We need to all stop being spineless here, and stand up for what’s right.

  17. @Tom.

    Re:”invisible army of developers”

    …adaption (i.e. bastardization) of content/sites that declare themselves mobile and who’s terms of use disallow adaption of content.

    …adding header/footers that wrap carefully crafted (down to the pixel) and branded mobile experiences.

    …adding ads to the copyrighted content of others.

    …interception of secure (SSL) transactions (potentially)

    …breaking of application, ringtone, wallpaper, game delivery, and the resulting revenue loss/customer support issues

    ALL of are great concern…

    …AND unfortunately ALL the above is happening across various carrier implementations of transcoding technology.

    Mobile web-based developers and businesses SHOULD be FREE to go about their business without having to jump through hoops, pump their fists, spend countless hours addressing blog posts, or otherwise fight tooth and nail to obtain respect and net neutrality.

    This interception by transcoders would NEVER wash if it was happening to sites accessed on the broadband web across ISPs.

    As for the “invisible army of developers” part…

    Winksite signed the Manifesto because we saw our properties being affected by the abusive practice of some. Mind you Winksite in turn represents 40K publishers who trust their content and audience members (multiples higher) to our platform. Other signatories represent similar/greater numbers.

    We’re the ones at the top of the pyramid who are left to sort it all out. It kinda looks like this: http://www.elatable.com/blog/2006/02/17/creators-synthesizers-and-consumers/

    Invisible not quite.

    Cheers,
    David Harper
    Founder & CEO, Winksite
    One World. No Borders. ®

  18. @Tom

    > There’s not only an invisible conspiracy against you,
    > but also an invisible army of developers behind you.

    We at StarTech.ro are definitely visible. And we support Luca’s initiative 100%. We’re not that noisy because there’s no transcoder around yet. And we definitely don’t need one.

    Alex

  19. > There’s not only an invisible conspiracy against you,
    > but also an invisible army of developers behind you.

    The same arrogant argument is leveled at any group set up to protect the rights of the powerless and/or disenfranchised.

    I and my no doubt invisible customers, for example, are grateful for someone who is speaking up: things have already improved somewhat thanks to his efforts. Otherwise I have no doubt at all that the transcoding machine would have steamrollered all objections.

    Luca speaks for more than himself, and in this situation is the only person who I’d trust is actually acting without self-interest. That makes him a whole lot more credible in my book, corporation-directed profanity notwithstanding.

    Chris

  20. Pingback: » Verizon Installs Novarra Transcoder - Mobilytics Has You Covered!

  21. @Tom

    > I’m not the only person who finds your aggressive
    > and abusive approach embarrassing.

    I am not perfect. I know.

    > Your request for Jo Rabin to remove points from the CTG
    > belies your lack of understanding: unlike the Manifesto,
    > no one person is responsible for the content of the
    > document. It’s debated and voted on in public.

    Actually, I was asking .mobi to remove support for UA spoofing, not simply Jo Rabin (who is the .mobi representative in CTG).

    wrt The Manifesto, it was created through a much more democratic process than CTG, since CTG, as I have explained, is managed by a cupola.

    Luca

  22. On my CTG participation: I’ve been calling for a more adult and polite approach to the problem since before you wrote the Manifesto. You can see this from blog posts I wrote, which you commented on, around the time. There’s nothing new about my support for this way of doing things, and I’m not the only person who finds your aggressive and abusive approach embarrassing.

    To summarise your position: your abuse is OK because you didn’t start it, and you don’t like the people who you disagree with. Very mature. There’s not only an invisible conspiracy against you, but also an invisible army of developers behind you.

    Your request for Jo Rabin to remove points from the CTG belies your lack of understanding: unlike the Manifesto, no one person is responsible for the content of the document. It’s debated and voted on in public.

    I’ll leave this now…

  23. @James

    >Irresponsible transcoding is a nightmare. I know first hand
    > how difficult it is to make progress whilst the gods
    > flick switches.

    why don’t you ask Jo Rabin to require that clause 1,2 and 3 of point 4.1.5 are removed from CTG on behalf of dotMobi?

    http://www.w3.org/TR/ct-guidelines/

    this change would effectively make it impossible to spoof the UA string or other headers.

    Luca

  24. @Tom

    > Your abuse wasn’t a one-off incident Luca.
    > The W3C is a “whore”, its participants “assholes”
    > and “fuckers”, transcoder companies are “arrogant bastards”
    > my remarks were “bullshit”, the CTG document “stupid spec
    > smells fucking novarra shit all over the place”

    OK, so, this is your strategy to get out of the corner you have placed yourself into. OK, then. I did use abusive language to react to the abusive *behavior* of Novarra, Vodafone and w3C, *but* that was on the WMLProgramming mailing list, a mailing list where developers calls a spade a spade. Sometimes you just need to
    speak from the heart about how you feel about certain things, and I am still convinced that in this case it is fully justified. Did you ever wonder how we were able to create the Manifesto in one 30th of the time CTG needed to get where they are now? it’s simple. On WMLProgramming people discuss about what they want to achieve and simply get there in the shortest possible time. fixing the wording happens with public scrutiny and a couple of iterations.
    I certainly did not call W3C “fuckers” the first time I was involved with them. The reaction to w3C is the result of many months (in my case, years) of frustration. A frustation which trying to work with W3C brings to anyone who wants to get anything practical done. As it is today, W3C is a waste of time in the best case, and, in the worst case, a tool in the hands of corporations small and big. People go around in circles for months getting nowhere practical. In the case of CTG, add the refusal to listen to developer voice and you have it: strong feelings brought about by the arrogance with which CTG partecipants are dealing with the whole thing. Always taking each and every decision in CTG so that it can be misused by transcoder vendors.

    > The Manifesto has just under 100 signatures. There are way
    > more than 100 mobile developers out there Luca, and you
    > don’t represent all of them.

    You know better than me that most people in most mobile companies (not to mention operators) typically do not expose themselves personally in situations like this: nothing to gain and everything to lose. The fact that 100 signed is in itself a *huge* success, since they represent the tip of the iceberg. Off-line, I got tons of support from people (even from operators) who did not want to expose themselves personally, but recognised the value of showing the world what transcoders really was doing and curbing their destructive strategy with the manifesto.
    Of course, further evidence is provided by the fact that there is no trace of pro-transcoder movement out there. There is no single company or traditional mobile blogger who has defended transcoders or attacked the Manifesto. So, I may not represent all of them developers, but I definitely represent what most of them think of transcoders.

    > Now you complain when developers are represented.

    this has to be explained, because it’s a textbook example of corporate manouvering. Up to about one and a half months ago, there was no trace of a mobile developer thinking that CTG was a good idea. For this reason the CTG people were real careful to avoid that any developer representative would be invited to CTG. That’s when you started expressing your viewpoints that CTG was not so different from the Manifesto after all, and CTG still could help. Automagically, you were invited into the CTG WG by Dan Applequist (Vodafone), Jo Rabin (dotMobi) and Dom (W3C). In short, a textbook example of how you set up an appearence of democratic process, when it’s really a cupola calling all the shots. Now, how can you blame me if some not so kind terms come to my mind to characterize these people and these organizations?

    Luca

  25. Interviewer: “So, why do you think the mobile web hasn’t taken off yet?”

    Insider: “Well, we’re all too busy beating each other up. We don’t have any time left to actually *build* anything.”

    :-)

    Irresponsible transcoding is a nightmare. I know first hand how difficult it is to make progress whilst the gods flick switches.

    But there are lots of ways around this.

    I think we should bury our syntactic differences and speak with a thousand voices – through the merit of the services we can produce regardless.

  26. Your abuse wasn’t a one-off incident Luca. The W3C is a “whore”, its participants “assholes” and “fuckers”, transcoder companies are “arrogant bastards” my remarks were “bullshit”, the CTG document “stupid spec smells fucking novarra shit all over the place”. Don’t try and paint this as out-of-context, a simple search of the WMLProgramming archives for profanity delivers reams of this stuff from you.

    You used to complain that W3C groups were only composed of people who paid to be there. Now you complain when developers are represented. If I didn’t know better I’d suspect that you just don’t like the W3C for some reason, and are looking for a justification you can find to support this view.

    The Manifesto has just under 100 signatures. There are way more than 100 mobile developers out there Luca, and you don’t represent all of them.

    There’s no need for it to be an either-or choice of course, given that the content of the Manifesto and CTG doc is so close.

  27. @Tom

    > I can only speculate that this is more likely a result of
    > polite conversation with them than calling them “fuckers”,”assholes”

    This is a blow under the belt, Tom. It is correct that, in the past, when I saw how W3C had absorbed Novarra’s requirements verbatim into CTG, I called them fuckers, but this happened in a different context, and bringing it up here only shows that you must be running short of arguments.
    The reality is that you have supported CTG in spite of the fact that I had clearly warned about the risk that Novarra could exploit it to justify abusive transcoding. This has happened and is bringing damage to developers.

    > I’ll say it again: Verizon are in the wrong here.
    > We all agree on that. It’s a shame that we can’t
    > unite more effectively in situations like this.

    You are late. Everyone has already united around the Manifesto in April. Developers and transcoder vendors alike. And also lots of operators which adopted the Manifesto for their transcoding requirements. There was no need for W3C to come up with a similar document which offered lower protection for mobile-optimised content. As I have explained endless time, this is not about standards (nothing about transcoders is about standards. it’s all just a big hack). This is about how much wiggle room you want transcoder vendors to have. With the Manifesto, they have little. With CTG, they have a lot more and transcoder vendors abuse it (as you have seen with your own eyes in this occasion). This was all to be expected and I had warned you and everyone that this was bound to happen.
    Our only hope, at this point, is that CTG does not become a W3C recommendation.

    Luca

  28. @Sean-> – That said I think that iPhone + Android are the very ‘beginning of the end’ of the era of specialized content for mobile.

    It may be in the future. But today is not reality!

    Regards.

  29. This has to stop already!

    We have tied ourselves to an industry where we have to develop software to account for the fact that the carriers can change the game with a single flick of a switch.

    We have seen this with both our mobile analytics product, and our SMS tools. It’s like hitting a moving target.

    We had to specifically build Mobilytics to grab the original user agent and IP address from backup headers, knowing that they were going to pull this some day. It’s not so bad except when the carrier (SPRINT) doesn’t even give you that info!

    Then they make claims of white listing, but our analytics clearly show that it is not the case.

    Very frustrating.

    Greg Harris

    http://www.mobilemarketing.net
    http://www.mobilytics.net

  30. Luca, I’m not sure how you can tell who authored which document, your forensic abilities must exceed mine.

    I understand that Novarra are pursuing the matter with Verizon as we speak. I can only speculate that this is more likely a result of polite conversation with them than calling them “fuckers”, “assholes”, or labelling anyone who talks to them as “collaborationists”.

    I’ll say it again: Verizon are in the wrong here. We all agree on that. It’s a shame that we can’t unite more effectively in situations like this.

  31. Pingback: Verizon transcodes mobile websites « Bango Customer Service Blog

  32. @Tom

    > Jo Rabin has raised an issue around this on the public BPWG
    > mailing list. I’m not sure what the procedure is
    > for going through formal channels to get to Verizon,
    > but I’m guessing someone there does.

    Tom, let’s not be ridicolous, please. The Verizon developer docs for transcoding have obviously been authored by Novarra’s Sean Patterson (the same guy who sits in the CTG working group), so, if there was a trace of will by Novarra to fix the problem, the problem would be already fixed (in fact, it would not be there in the first place).
    W3C have appointed the fox to guard the hen, so please give us a break and save us the theatrical version.

    Once Novarra will come to terms with the revolutionary concept that they cannot mess with other companies’ content, all of this problems will be a memory of the past.

    Luca

  33. Luca – if you aren’t prepared to read what people are writing, please don’t bother commenting on it.

    Sean doesn’t say that things are more or less OK. Sean and I are not disagreeing over what CTG says. He doesn’t even *mention* the CT document in his comment here.

    I’m asking you to point out areas of ambiguity in the document, not repeat your demand for UA spoofing to be outlawed in all situations.

    Can you see areas of *ambiguity* in the CT doc? Report them, and let’s sort them out to remove loopholes.

  34. Ray – Jo Rabin has raised an issue around this on the public BPWG mailing list. I’m not sure what the procedure is for going through formal channels to get to Verizon, but I’m guessing someone there does.

  35. @Tom

    > Sean and I aren’t disagreeing over what the CTG says.

    Maybe not, but you obviously disagree about what it means in practice.
    Again, your reaction was that Verizon is in breach of CTG.
    Sean thinks everything is more or less OK.

    > If there’s ambiguity in the CTG doc, please do point it out
    > on the public W3C mailing lists so it can be addressed.

    Are you kidding me? there has been one ton feedback and comments
    to CTG since August arguing that UA should not be spoofed.
    Under the influence of Google and Novarra, W3C has managed to ignore
    that feedback by stating that UA spoofing was not allowed,
    while allowing exceptions to spoof under certain conditions.
    This is what Novarra and Google desperately needed to happily keep spoofing, as this latest development in the transcoding saga has clearly shown.
    (As an aside, I am curious to know whether Novarra will also transcoded pages already transcoded by Google, or if they needed to turn to extra hacks to overcome the problem).

    Now, compare this to the Manifesto, whose first rule spells out loud and clear:

    “No USER AGENT Spoofing: under no circumstances should
    the original User-Agent string be removed, modified
    or moved to a different header.”

    http://wurfl.sourceforge.net/manifesto/

    If Verizon had demanded that the Manifesto rules are adopted,
    all of the pain we are seeing here would be avoided.
    Having seen Novarra in action for a long time now, I agree with Ray.
    Someone in Verizon has no clue of the existence of an ecosystem,
    and they ended up believing Novarra’s bullish marketing,
    which will repeat over and over again how much better a transcoded
    web experience is when compared to a mobile experience (!).

    To those Verizon employees I recommend the following reading:

    Novarra’s marketing in clear view (CEO Interview in which she claims that tricking the CNN site to return web content to mobile phones is perfectly acceptable) followed by (see comments) what the mobile ecosystem thinks of it:

    http://www.mobilemarketingmagazine.co.uk/2008/05/talking-transco.html

    A case study of Novarra’s transcoding (NYT’s transcoded web vs the
    mobile-optimised view, which Novarra totally disrupts because of UA
    spoofing):

    http://wurfl.sourceforge.net/vodafonerant/vodawsj/nigel.html

    Luca Passani

  36. In my opinion CTG must include:

    1. Proxies are not allowed to alter the content of an website or headers if this is mobile optimized.
    2. This MUST work in case of mobile redirect (the user enters the website from phone and it is redirected to another domain – mobile.domain.com)

    I’m sure that almost everybody can agree that these are good things for mobile users, for developers and for publishers. W3C is a sponsored organization. While sometimes is easy to agree about things/ideas, it will always be difficult to agree about money. This is another reason why guidelines are not an easy task.

  37. Has anybody contributing to this thread made contact yet with Verizon (preferably though its W3C links) or in any other way?

    Bango has started sending out emergency notifications to site owners to request they adapt their sites to avoid transcoding (unless they want it). interestingly, Bango is affected potentially since for Verizon users, credit card is a common payment method, so we may need to be worried about SSL interception of the secure dialogue between users.

    The concern would be that somebody unscrupulous at Novarra or Verizon “taps” the stream of credit card data and passes it to a bad person. Those familiar with Operation Ore will be aware that some people affected by credit card fraud may have ended up dead as a result of similar credit card data leakage. Lets hope thats not possible and that all sites collecting sensitive data using SSL are not intercepted.

    This is quite serious stuff.

    Knowing that there are powerful factions in all big pre-existing internet companies trying to hope mobile is “just like a handheld PC” one could believe there is a conspiracy to slow the use of great services that rely on device knowledge or capability. However, I think that incompetence or ignorance is much more likely as we found out in previous transcoder deployments.

  38. Sean has his view, which does not (in the case of modifying the UA) overlap with the contents of the CTG.

    I have my views; in the case of the UA string they overlap with the CTG.

    Sean and I aren’t disagreeing over what the CTG says. If there’s ambiguity in the CTG doc, please do point it out on the public W3C mailing lists so it can be addressed.

    Members of the CTG aren’t expected to hold identical views on these issues (I’ve certainly never been told how I should think) – in fact if there was no disagreement or debate, I doubt there’d be much need for a working group.

  39. @Tom

    > I can’t see why you think Sean and I disagree
    > over the contents of the CTG, and Sean doesn’t say
    > that everything is fine

    well, you clearly say that Verizon is in the wrong, while Sean says that there is a lot good and preserving the UA string is the *wrong* thing to do (!). The most criticism of Verizon that Sean expressed is that if someone’s site got broken, they should go ahead and complain (thank you, Sean. How magnanimous of you!).
    To me, considering that you both are (or have been) part of the group that brought CTG, this is disagreement.

    Cheers

    Luca

  40. A couple of remarks:

    a) I have found it difficult to identify precisely who is responsible for egregious content adaptations: transcoder vendors who do not include the necessary or recommended features in their products, or operators who do not configure their transcoders properly. Especially since the same operator may deploy different transcoders in different countries.

    b) For “So, where is this transcoder disobeying standards (honest question)?”, I suggest looking at Vodafone Spain and Vodafone Portugal.

    c) It is an excellent thing if Verizon users can easily switch off transcoding and access unaltered content directly.

    d) “I think that iPhone + Android are the very ‘beginning of the end’ of the era of specialized content for mobile.” There are paramount usability issues that make mobile-optimized content unavoidable, and which explain why there is an increasing number of iphone-optimized sites. Besides, mobile phones exhibit well-established features such as click-to-call, click-to-sms, click-to-videophone, pictograms, or rapidly emerging ones (such as location and two-2 codes) that are purely and simply unavailable in standard desktop markup.

    e) “This issue will slowly fade anyway” brings no solace to content providers whose business model has had and still has ample time to be damaged till then (especially content download sites).

  41. @Sean> That said I think that iPhone + Android are the very ‘beginning of the end’ of the era of specialized content for mobile. This issue will slowly fade anyway.

    I agree with you and I disagree with you (or at least, one of your implications).

    Transcoding’s days are numbered simply because no self-respecting site owner or brand won’t want to NOT have the last say in how their content is presented. Increasing numbers of sites will opt out and user behaviour will begin to swing towards the better ‘hand-crafted’ sites. And of course the browsers are getting better.

    But I don’t think this means that sites made for mobile *humans* will go away – in fact quite the opposite. There will be a rapid increase in the number of sites which cater for mobile users. The rise and rise of web+location, for example (as personified by Google Gears) is driving a whole new wave of made-for-mobile sites – which might not necessarily make a bunch of sense for a desktop user.

  42. Luca, I can’t see why you think Sean and I disagree over the contents of the CTG, and Sean doesn’t say that everything is fine.

    Verizon aren’t taking advantage of any loophole here: they’re completely misinterpreting a document they shouldn’t be referring to at all.

  43. @Tom and @Sean

    Both of you have been part of the CTG working group, yet you obviosuly have a different (and incompatible) reading of what CTG means. Tom thinks that Verizon and Novarra are in breach of CTG, while Sean states that everything is fine.
    How is this possible? To be honest, to me this is hardly surprising: this is just another example of W3C “standards” for which, after months (or even years) of discussion, W3C partecipants simply agree on “wordings” in which everyone will read what they want to read. In this particular case, as I have explained many times in the past, CTG seems to say that UA string shouldn’t be spoofed, but it really allows for a giant loopholes through which transcoder vendors can run a train. As they did in the Verizon case.

    @Tom

    So, are you happy now? as I had explained to you multiple times (and you refused to believe me), CTG is an excuse for Novarra to do exactly what they have done in the past: spoof the UA string. Sure, you can argue that Verizon is formally in breach of CTG, but what you get now is Verizon spoofing the UA string in W3C’s name (…and no, your comment to this blog post is not a powerful enough counter-move). You have been used, Tom. They wanted you to join CTG so that they could argue that developers were also involved. Good job.

    @Sean

    Yeah, I keep saying the same things over and over again, don’t I? too bad that Novarra has been equally resolute in trying to screw everyone else in the ecosystem, rather than playing by the rules that other transcoder vendors have decided to respect when they signed the Manifesto.
    Cache-control is not enough by itself. If a transcoder spoofs the UA, applications can no longer recognize mobile devices, which means that they won’t even know they need to send a Cache-Control in the first place (!). So, your argument is bogus.
    If you compare the two documents (Manifesto and CTG), you will see that the difference is “Do not spoof the UA string”. Only apparently does CTG demand the same. CTG allows UA spoofing under certain conditions which is what will be exploited by Google and Novarra to abuse and still claim complaince. I have been saying this before Verizon launched transcoding, and Verizon provided a textbook example of this, for those who still did not believed.
    Novarra may not have been dictating the document, but it has been driving the CTG exactly in the direction it wanted. Just look up the contributions by Novarra’s Sean Petterson in the W3C CTG archives and you’ll see what I mean. You did the same on behalf of Google, of course.
    Finally, neither Android nor iPhone will mean the end of content optimized for mobile, in my opinion. Quite the opposite, in fact. Interest in mobile web and mobile-optimised sites has never been so high as with the introduction of the iPhone, for which mobile- (or even iPhone-) optimized sites are created every day.

    Luca Passani

  44. Oh boy, I can only bear posting one more time on this. As usual my counterpoints are:

    – Some of the behavior here is quite good, like, respecting Cache-Control: no-transform. Good mobile sites ought to be doing this already, and if they do, there is no problem.

    – The same tired rantings from Luca are as tired as ever. It might be fun to think of this as a good-vs-evil struggle, but it’s far more mundane. For the W3C’s part, of which I was a long-time participant from Google, I can only say this picture of corporate control and unethical behavior is as silly as you would imagine, which one might realize if one were participating too. Novarra was nowhere close to dictating this document.

    – That said, Luca’s writings on this topic are 80% informed and valuable, and not surprisingly overlap 80% with the W3C’s recommendation. So, to pretend that he’s proposing a totally different, ethical, right approach is disingenuous. Fortunately, this is easy to verify by comparing these two public documents.

    – I again trot out the pretty good argument that *preserving* the User-Agent is the *wrong* thing to do. A transcoder should not masquerade as a phone!

    – That said I sympathize with anyone whose site just got broken by the change. To the extent that a site wasn’t following standards before, OK, unfortunate, but the site should adjust. To the extent that the transcoder *still* breaks behavior after relevant standards are upheld, *then* it’s time to complain! So, where is this transcoder disobeying standards (honest question)?

    – That said I think that iPhone + Android are the very ‘beginning of the end’ of the era of specialized content for mobile. This issue will slowly fade anyway

  45. Oh dear, here we go again.

    Well, Verizon are clearly in the wrong here (as authors of the docs for developers).

    As you point out, they are referencing a document (the CT guidelines) which says “do not reference this document” – because the guidelines are not finished yet. Not a good start.

    In addition, Verizon are recommending using URL schemes to identify mobile content – which is pretty well the opposite of the CTG doc. Their routine changing of the User-Agent also puts them in contravention of the guidelines.

    This is a textbook case of developers, the Manifesto and the CT doc all being in agreement that what Verizon are doing is wrong. Why not concentrate on doing something positive about it instead of perpetuating this unnecessary and unproductive in-fighting?

  46. I have to agree with Chris and Luca. Here we simply have a situation where a nascent mobile web ecosystem was being successfully nurtured by a worldwide community of developers from the bedroom to the experienced corporates, and then along come a handful of corrupt unethical bullies (Novarra being the worst offender) who stomp all over it and mislead those in positions of power who should know better (the network operators).

    To add insult to injury we have an apparently spineless (and I reach that conclusion with regret) industry body (W3C) who despite claims to the contrary has shown that they are simply going to allow corporate interests to prevail over those of the developer and user, and damn anyone who doesn’t like it. It is doubly unfortunate that the W3C is not properly independent but consists of those who should be policed, and is thus effectively a self appointed cartel that dictates to developers and users. “But they paid for the infrastructure!” I hear you cry. True, but sometimes technologies cross the boundary of industry playthings and into wider usefulness by society, and at that point responsibilities change.

    The mobile web, arguably the future of the internet, and a large part of mobile usage, has currently been sold out. Lets keep fighting to get it back.

  47. This has always seemed to me to be “gaming of the system” by corporate interests such as Novarra attempting to buy a figleaf of respectability: they’ve essentially bought a “licence to create derivative works” from an organisation that isn’t (or shouldn’t be in the position to issue one). And thus does the W3C claim de-facto jurisdiction over the entire mobile web.

    The fact that it’s Verizon doesn’t surprise me, though: they’re a great example of the “we own you” philosophy inherent to some Telco business models.

  48. (Note: I am Luca Passani, the editor of the Manifesto for Responsible Reformatting mentioned in the article)

    I am disgusted by Verizon’s choice to deploy a transcoder that spoofs the User-Agent string.
    The only reason to change the User-Agent string is to extort web-only content from publishers who might have a carefully designed mobile experience ready for users. How can this be acceptable?

    Verizon and Novarra have chosen to disrespect the neutrality of the web and to appropriate web content they have no rights to for their own illegitimate interests. To me, this is an abuse of gigantic proportions. I firmly believe (and so do thousands of mobile developers around the planet) that nobody should have the right to interfere with the content and the applications that companies have built with great use of time and resources.
    I can’t believe that Verizon can be so shortsighted not to see that by allowing this, they are undermining the whole mobile ecosystem and, ultimately, their own interest.

    What to say about Novarra? they are very unethical. The whole world has told them to refrain from spoofing the User-Agent, but they arrogantly try to redefine the rules of the mobile web for their own petty interests, by bamboozling ignorant carrier execs behind the scenes. Shame on them.

    What to say about W3C? to me, they have demonstrated that their “standards” are at the mercy of whoever pays the ticket to sit at the W3C table, no matter what everyone else (one army of mobile developers) is asking them not to do (do not allow User-Agent spoofing under any circumstances). How sad. TBL, if you are reading this, I urge you to use your veto power in W3C and stop this abuse perpetrated at the expense of mobile and web developers.

    One final note: in their frenzy to peddle their irresponsible technology, Novarra and Verizon did not even do their homeworks of waiting for the W3C guidelines to become a W3C recommendation. Right now, the W3C CT Guidelines are just a working draft, and by W3C definition:

    “Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.”

    Given the lack of ethics in W3C, I am sure that Novarra will succeed also in the final step, but right now this is just an example of the incredible arrogance with which the company has managed the whole thing.

    Luca Passani

  49. It’ll all end and all transcoders will vanish! :) Mark my words. All we need is some time,complaining users and more new phones.

Comments are closed.