Last week Opera released a minor upgrade to the Opera Mini browser. I like to follow the latest releases so I downloaded it to all my phones from mini.opera.com. I used each phone’s built-in browser so Opera could detect the phone model and send the optimal version of Opera Mini. All went well until I tried to upgrade my Samsung A920. I got a message “Your phone model could not be detected” This seemed strange as Opera had recognized the A920 in the past.
I though maybe something was broken on Opera’s side. So I went to the page Opera provides for users to report undetected phones, mini.opera.com/detect, and was surprised that the phone’s User Agent was “Mozilla/5.0 (compatible; OpenWeb 5.6.1.3-03) Opera 8.54“. That’s wrong, the A920 UA should be “Samsung-SPHA920 AU-MIC-A920/2.0 MMP/2.0 Profile/MIDP-2.0 Configuration/CLDC-1.1“.
It seems my carrier, the US CDMA/EVDO provider Sprint PCS, is unilaterally running all web traffic though Openwave’s OpenWeb reformattting transcoder which is replacing the phone’s User Agent header with it’s own.
The User Agent is the only way that mobile sites can identify the phone model and optimize their markup for different screen sizes, Javascript support, etc. If a proxy or transcoder changes the User Agent sites can’t deliver an optimal web experience. It’s even worse for providers of content like ringtones, games, wallpapers. Without the User Agent there is no way for the provider to know which content is compatible with the phone. The big multi-national carrier Vodafone has gotten a bad name with mobile developers for implementing similar transcoders from Novarra, Openwave and Bytemobile.
It actually looks like with this implementation for Sprint, Openwave is trying to sort of do the right thing and send the original User Agent if it detects mobile content. The detection seems to work with many sites that send only mobile content from a given URL. Opera delivers PC or mobile content from mini.opera.com depending on whether it detects a full-web browser or a mobile one. Opera’s multi-serving of content seems to confuse Openwave. And it’s not just Opera, users at the independent site SprintUsers.com, are reporting problems with several free ringtone sites including Rumkin.com and FunForMobile.com
It’s not easy to programmatically determine if a site is designed for mobile devices. You can’t rely on the Mime type or Doctype, there are lots of tag soup mobile sites served as text/html. Bad practice perhaps, but those sites work. And really, I don’t understand why OpenWave changes the User Agent in the first place. There’s nothing about the process of transcoding that requires changing the User Agent to function. The mobile web works best when sites can identify mobile handsets by User Agent. If the site sends mobile content no transcoding is needed. I’ve heard the excuse that transcoders need to send a desktop browsers User Agent because there are a handful of badly written full web sites that return an error or empty page when confronted with a mobile user agent. I’d be OK with a transcoder retrying with a desktop User Agent in those cases only, but not in the case of sites like mini.opera.com or Rumkin which return a usable page to ALL browsers and mobile content to mobile browsers. I’m forced to suspect evil intent, nothing else makes sense. It’s as if Openwave is changing the User Agent to fool multi-serving sites into delivering non-mobile content so that their transcoder can have more work to do and they can claim to have “fixed” a higher percentage of sites.
Don’t get me wrong, I’m not opposed to transcoding, I use and recomend Mowser, Skweezer and Google GWT for viewing full web sites on the limited built in browsers of feature phones like the A920. But I opt to use them and at the bottom of every transcoded page in all three of those transcoders there’s a link to view the original untranscoded page. Sprint has imposed OpenWeb on their users without any warning and with no way to opt out.
Even though I don’t like this transcoder being forced down my throat, it wouldn’t be so bad it if it improved the mobile web experience. I decided to spend the day using the Obigo Browser and OpenWeb transcoder instead of my normal Opera Mini.
First I tried Bloglines Mobile. I loaded my subscriptions list and clicked on the the first of my feeds, GigOm which had 4 unread items. After a long delay, OpenWeb reported that Bloglines had refused the connection. I retried and the GigOm feed loaded but there were no longer any unread items. Bloglines marks items read when you load them so it seems unlikely that Bloglines actually refused the connection. I think that Bloglines got the request, returned the page with my unread items and then an error occurred within OpenWeb causing it to loose my page. The error message blaming Bloglines seems bogus. If the server had actually refused the connection the Bloglines script would never have seen the request and my items would still be unread. I tried three more feeds and they loaded without error so this is a transient issue. Later, I got the refused connection error again, this time on the SMS Text News feed and lost several more unread items. Going for broke I next I tried loading all the unread items (about 30) in all my feeds as a single page, something I often do with Opera Mini. This actually worked, OpenWeb split the big page up into 9 little ones with no errors this time. I was mildly impressed although it didn’t make up for the earlier problems.
Next, I tried the mobile web version of gMail. On the login page OpenWeb showed a big warning that the connection was not secure! Very strange, Google uses https for gMail and other mobile browsers including WebKit and Opera do show the page as secure. I decided to press on, and entered my credentials. Other than the initial message that the page was unencrypted, gMail worked OK. Switching to Yahoo Mobile Mail, it worked most of the time although I got a couple more “Connection Refused…” messages. A lot of other sites including Engadget and MSNBC worked without any problems.
However, trying to follow a link to a page on TechCrunch’s full site failed completely. OpenWeb gave a strange error to the effect that the page I requested “exists but could not be loaded” and that the issue is being “investigated” . I also got “Page can not be displayed” trying to load the full web version of HowardForums. Overall I’d say I had trouble with about 10% of the pages I tried to visit with OpenWeb, a considerably higher error rate than with Opera Mini, Mowser, Skweezer or Google GWT. I’ll admit that I rarely use the Obigo browser for anything other than downloading content, mostly applications, so it’s possible some of the errors may been with with the browser rather than the transcoder.
Other impressions of OpenWeb were that it works like a typical transcoder, reformating pages into a single column. Cookies seem to be supported and web forms using both get and post worked (earlier versions of OpenWeb reportedly turned Posts to Gets which seems to be fixed now). OpenWeb doesn’t fold content to hide navigation links like Opera mini or Google GWT. Pages are split at around 30 KB which is probably about the maximum Obigo can handle so that’s good. OpenWeb seems to retain quite a bit of CSS formating including borders, background colors and floats on images. Browsing through OpenWeb seemed slow, there were often delays of 30 seconds or so before a page started loading, definitely not the normally snappy response I expect on Sprint’s 3G EVDO network. I also don’t like that OpenWeb reduces all images to a maximum width of around 120 px, considerably smaller than the A920’s full screen width of 176 px.
While I imagine that Sprint and OpenWave executives think they are improving the mobile web experience with this transcoder, I find that it worsens my experience. I’m put off by it’s slowness, loss of the User Agent on even a few sites and the high level of errors. But the worst part is that it’s being imposed on me without asking and with no way to turn it off or even visit an untranscoded page when an error occurs. Sprint is bleeding customers and it’s stock price is in the toilet, OpenWeb in it’s current format is likely to drive even more customers away. Sprint needs to make OpenWeb optional and OpenWave needs to add a “View in HTML” link to each transcoded page like Google does. If Sprint really wants to improve the browsing experience for users they should partner with Opera to bundle Opera Mini with all their new phones and offer it has a free download from their portal homepage!
I think it’s clear that Sprint doesn’t care at all about user experience. This was not even a consideration one way or the other regarding this transcoder roll-out. What is clear is that they don’t want to download all the content being accessed by all of their “unlimited data” plan customers. They promised “unlimited data” and now are breaking that promise because they must have found it too expensive to deliver.
Shame on you, Sprint. Shame.
The fact is Sprint haven’t done anything to improve the browsing experience for users , but i think sprint will improving it in the future.
Pingback: London Calling » Carnival of the Mobilists 115 is here
i dont want to sound like an idiot but how do i get on airgames and chat rooms thru my pc. my phone carrier is sprint mobile thank you
Pingback: User First Web » Verizon’s “Open Networks” Not Very Open, Sprint Breaks the Web
In Spain If that were working with Nokia Ad Service or Bango Vodafone will include a whitelist.
Thanks everyone for your comments.
E. casais, using a .mobi or m.*.com domain doesn’t seem help with OpenWeb on Sprint, at least I still get the wrong User Agent with thoese domains.
There seems to be some sort of Whitelist in place though as Yahoo and Google are getting the real user agent (the phone is detected correctly downloading Yahoo Go or Google Maps and gMail.
I’m going to do some more testing to try to isolate what effect different domains and MIME types have.
How is one supposed to identify a Vodafone user and apply the “no-transform” header? I’ve had to do this for Sprint users about a year and half ago when they started to compress all content from their gateway to the phone. Which Vodafone headers are we to look for?
For a few months now, Vodafone Portugal has been deploying the OpenWeb transcoding gateway which is now also in use at Sprint. Here is a summary of our experiences with it.
The presence of transcoding is revealed through a number of symptoms. The visible ones are as follows:
1) Formatting is reduced to its simplest form: text is uniformly black, links are in either in black or blue, underlined, on a white background. Fonts with serif are transformed into sans-serif fonts.
2) Text fragments that are emphasized with appear in a huge font 2 or 3 times larger than the normal text font.
3) Tables are stripped away and their contents rendered as a series of lines, without any justification or alignment.
4) External style sheets are eliminated; style elements defined within markup are eliminated as well. Hence, the entire corporate look and feel disappears. We have not checked the effect of style definitions associated directly to individual markup elements.
5) Lists are stripped away and their contents rendered as a series of lines, without bullets or numbering.
6) elements are entirely eliminated; pictograms, and presumably all kinds of plug-ins, become available.
7) Normal application links embedded in the page stop working, resulting in error messages in Portuguese.
8) The page is laid out so that there is no margin at all on the left side, whereas a wide margin appears on the right side.
9) Spurious extraneous links appear with labels in Portuguese: Página 1 2 3
10) At the top of the page, before any content specific to the application, appears a long series of numbers starting with SEC-SGHX660/1.0 TSS/2.5
[HTTP_ACCEPT] => text/vnd.wap.wml,
application/vnd.wap.wmlc,
application/vnd.wap.wmlscriptc,
application/vnd.wap.xhtml+xml,
application/xhtml+xml,
application/vnd.wap.connectivity-wbxml,
application/java,
application/java-archive,
application/x-java-archive,
application/vnd.wap.cert-response,
application/vnd.wap.signed-certificate,
application/vnd.wap.hashed-certificate,
application/vnd.wap.wtls-ca-certificate,
application/vnd.oma.drm.message,
application/vnd.oma.dd+xml,
application/vnd.wap.sic,
application/vnd.wap.slc,
image/vnd.wap.wbmp,
image/gif,
image/jpg,
image/jpeg,
image/png,
image/bmp,
text/html,
text/x-vCalendar,
text/x-vCard,
text/css,
text/vnd.sun.j2me.app-descriptor,
multipart/*,
text/plain,
audio/mid,
audio/midi,
audio/x-mid,
audio/x-midi,
audio/sp-midi,
application/vnd.smaf,
text/x-iMelody,
text/x-imelody,
audio/imelody,
audio/amr,
video/3gpp,
audio/mp3,
audio/mpeg,
audio/mpeg3,
audio/aac
[HTTP_ACCEPT_CHARSET] => ISO-8859-1,
US-ASCII,
UTF-8;q=0.800,
ISO-10646-UCS-2;q=0.600,
ISO-8859-2;q=0.500,
windows-1250;q=0.500
Here they are when accessing a site via http://*.*.net
[HTTP_USER_AGENT] => Mozilla/5.0 (compatible; OpenWeb 5.5.4.1-01) Opera 8.54
[HTTP_ACCEPT] => text/html,
application/xhtml+xml;
profile=http://www.wapforum.org/xhtml,
application/vnd.wap.xhtml+xml,
text/vnd.wap.wml,
image/vnd.wap.wbmp,
text/plain,
text/css,
image/jpeg,
image/gif,
image/png,
audio/mpeg,
application/x-shockwave-flash,
application/x-javascript
[HTTP_ACCEPT_CHARSET] => iso-8859-1,
windows-1252;q=0.3,
utf-8;q=0.2,
*;q=0.1,
iso-8859-2
Notice the incompatibilities between the content types and the charsets introduced by the fake user agent. Notice too that the advertised content types make the end-device supposedly unsuitable for downloading Java applications and a large range of multimedia content.
And here are the HTTP header fields when accessing a site via http://mobile.*.*:
[HTTP_USER_AGENT] => SEC-SGHX660/1.0 TSS/2.5
[HTTP_ACCEPT_CHARSET] => iso-8859-1,
us-ascii,
utf-8;q=0.800,
iso-10646-ucs-2;q=0.600,
iso-8859-2;q=0.500,
windows-1250;q=0.500,
*;q=0.001
[HTTP_ACCEPT] => text/vnd.wap.wml,
application/vnd.wap.wmlc,
application/vnd.wap.wmlscriptc,
application/vnd.wap.xhtml+xml,
application/xhtml+xml,
application/vnd.wap.connectivity-wbxml,
application/java,
application/java-archive,
application/x-java-archive,
application/vnd.wap.cert-response,
application/vnd.wap.signed-certificate,
application/vnd.wap.hashed-certificate,
application/vnd.wap.wtls-ca-certificate,
application/vnd.oma.drm.message,
application/vnd.oma.dd+xml,
application/vnd.wap.sic,
application/vnd.wap.slc,
image/vnd.wap.wbmp,
image/gif,
image/jpg,
image/jpeg,
image/png,
image/bmp,
text/html,
text/x-vcalendar,
text/x-vcard,
text/css,
text/vnd.sun.j2me.app-descriptor,
multipart/*,
text/plain,
audio/mid,
audio/midi,
audio/x-mid,
audio/x-midi,
audio/sp-midi,
application/vnd.smaf,
text/x-imelody,
text/x-imelody,
audio/imelody,
audio/amr,
video/3gpp,
audio/mp3,
audio/mpeg,
audio/mpeg3,
audio/aac,
text/vnd.wap.wmlscript,
*/*;q=0.001
Much better than the former case, but still inexact.
We also checked the effect of taking into account parameters suggested by Vodafone UK to prevent their gateway from transcoding mobile content, with the following results (all parameters were included at the same time):
e) Inclusion of the “no-transform” directive in the HTTP header field “cache-control”.
f) Structuring content properly with a DOCTYPE for XHTML mobile profile 1.0, and advertising it
with the MIME type application/vnd.wap.xhtml+xml and the correct charset.
These had no effect; every content gets transcoded.
Finally, we examined the situation when pages include a copyright notice as a textual comment in the markup,
as fields (supposedly usable by automatic content processors), and as part of the content visible to the end user. None of these had any effect, so the transcoder does not check for a copyright notice at all. We nevertheless suggest other developers to include these three elements, just to harden their legal standing in case they want to let their lawyers take over from the engineers (under the topic “prohibition of derivative works”).
We could not find a single site that was not mucked up in one way or another. It seems that the gateway transforms carefully crafted mobile sites to a format that corresponds more or less to i-mode HTML version 2.0, or to the kind of HTML level that was accepted by the KDDI gateway to feed HDML 3.x devices, many years ago. The only way for a site to get unharmed consists of pruning out everything that mobile devices are capable of today, i.e. tables, lists, style sheets, text attributes (emphasis, colours, etc), pictograms, plug-ins… and making sure that it is hosted under an URL that satisfies Vodafone’s idea of a mobile-enabled address. In short, a singularly egregious example of intrusion into third-party mobile service delivery by an operator. Importantly, this goes well beyond the issue of masking the user-agent from the application servers: there is a simple way to get it, but no matter what, the HTTP accept fields are no longer reliable, and the content is transcoded anyway!
And all this happens with the built-in phone browsers (no Opera mini).
Interestingly, it seems (from WWW-rumours) that the same transcoding gateway is deployed at Vodafone Spain, with some serious issues, but not as atrocious as the aforementioned ones.
Obviously, Sprint users are no experiencing several of the anomalies listed above (links not working, downloading of content no longer possible, unrecognized devices because of user-agent masking, reformatted pages and images), although it has been configured somewhat more intelligently than at Vodafone Portugal. Is there any difference in behaviour depending on the URL at Sprint? If so, some further improvements might be possible.
I hope this will help developers make some sense out of the possible mess they might have stumbled upon in Vodafone networks. If anybody has found a way — technical, legal or commercial — to eliminate transcoding via OpenWeb, please pass on the information.
Pingback: MobHappy » Blog Archive » Sprint Starts Hiding User Agents (and Breaking the Mobile Web), Too
I have the same experience – the browser doing hiding the user agent and sending its own string making our server not able to detect the mobile device, thereby compromising the user experience.
how do I get rid off this problem?
Pingback: Mike Rowehl: This is Mobility » Blog Archive » Sprint Hiding User Agent
Note that the W3C Mobile Web Best Practices Working Group is working on the development of guidelines for content transcoders, with the goal of avoiding this type of problems.
See http://www.w3.org/2005/MWI/BPWG/Group/TaskForces/CT/ for more details.
(a first version of these guidelines should hopefully be published in the upcoming few weeks)