jeff yen

Android in China: A rant, and how to get the Market working through the GFW (Great Firewall)

Okay, this first part is a rant; if you actually want to get to the tech part of this post, go to the part with all the bold text and arcane codes. If you'd rather read me rave about protectionist policies and information control here in China, then by all means proceed.

[...] dictatorships were absurdly sensitive to the public opinion they continually outraged; they always had to be in the right, to be morally impeccable [...]

- Stephen Maturin, The Surgeon's Mate by Patrick O'Brian

The above quotation is from a passage about Napoleonic France, but it is fairly applicable to the situation I've been dealing with ever since I arrived in China.

The government here has a juvenile obsession with monitoring and regulating the flow of information within, into, and out of the country, to the extent that China's version of the Internet is just an enormous walled garden. Try to eat of the forbidden fruit and you'll be stuck looking at server error messages, endless waiting in vain for pages to load, and (apparently) timed lockouts to access if you attempt to search for certain forbidden words. Try Googling "freegate" from a computer in China and you'll see what I mean; whenever I do it while my VPN or proxy isn't active, I get locked out of Google for a little while.

In fact, navigating the Internet proper is virtually impossible without a proxy or VPN service to break through the government-imposed censors.

With Facebook and Twitter (both blocked) integration spreading like a particularly virulent strain of herpes into every nook and cranny of the Internet, and more and more sites embedding videos from services like Youtube, Viddler, and Vimeo (all blocked), even unrelated pages will take forever to load, as the browser waits for all of those connections to time out before displaying anything.

I say the Chinese government's attitude to information management is juvenile, because it is really only those with under-developed or otherwise sadly misshapen self views that are so insistent on trying to control others' perceptions of them.

This weblog, in fact, was blocked by the Chinese government last year shortly after a critical post about Shanghai, and I had to do an end-run through my host to have the IP changed so I could once again access it from inside the country.

After some Chinese censor bot or bureaucrat (essentially the same thing) gets a whiff of this post, I have little doubt that I'll have to do the same thing again not long from now.

This all rather supports my impression that many aspects of the Chinese government, much like many of this country's nouveau riche, have never really grown up. They are used to always having their own way, having their every whim affirmed by the people around them, and so have developed a profoundly warped world view.

It has often been said that China has never had an expansionist colonial period; and traditionally speaking, that is true.

However, it's my view that this country's interests have never extended to the rest of the world. The only reason China has ever reached out across its borders is to pull something in; silver, back in the days of the tea trade; status (or 'face'), ever since they were drubbed in the Opium Wars; and more recently, the global economy.

They see little need -- aside from ancient feuds, minor spats with adjoining entities, and outdated territorialistic urges -- to venture out into the cold wastes of international waters, there to expend fuel, ammunition, and tax funds to take over territory which, to be honest, has very little value to them that their existing lands do not.

I believe their form of colonialism is simply to sit around, and suck as much of the rest of the world in, while giving as little as possible back.

It is this, among several other reasons, which convinced me not to purchase property here. I cannot in good conscience contribute to a system that I feel simultaneously opposes my presence on a moral and political basis, yet welcomes it on an economic one.

In the end, it may simply be a matter of cultural transparency. While there are innumerable facets of the U.S. government with which I am profoundly dissatisfied, they are at least logical and fairly predictable in their patterns of behavior. China's government, by comparison, sometimes acts like a drunken toddler, reeling from one whim to the other with little to no warning, expecting everyone else to follow along, and throwing tantrums when they balk.

So, down to how this affects me (besides having to use one of several available proxy/VPN services to browse Facebook, Twitter, Blogspot, Youtube, Blogger, Tumblr, many Google sites, and countless other websites, including those that depend on FBCDN for content delivery... even those that just have a snippet of embedded code from any of the above and several other ever-more-omnipresent social networks):

Android.

Life is hard being an Android user in China. I only bought my HTC Desire two days ago, and I'm already deeply familiar with the various issues affecting us.

The Chinese government has some kind of inferiority complex when it comes to Google; services go down, they come back up, then they go down again. The government here, unsatisfied both with Apple's hard bargaining and tightly controlled iTunes market, and Google's too-free-for-China Android Market and independent search policies, has decided to try its own route. They've created a new Android fork, called "oPhone" (original, right?), created a proprietary 3G standard, and are busily trying to corner the domestic market in protectionist ways that are a mind-boggling waste of time.

My HTC Desire came with a suite of apps preinstalled -- all Chinese -- and a link to something called "Go Market," which I can only assume is the local attempt at an equivalent to Android Market. It was at best a shoddy imitation, very much prone to the Chinese preference for overstated design and cramming as many features as possible into as small a space as possible.

I've seen how several Chinese online services operate, and to think how they might impact the mobile space just makes me shudder.

Aside from the fact that I would be locked into an incomprehensible market which had few apps, none of them of any use to me, many of the features the phone had by default simply didn't work.

So I wiped it, and reinstalled a popular custom mod (CyanogenMod). Voila; clean, functional bliss. I was even downloading apps from the Android Market, for about a day, until either the government caught on, or some kind of flag on my China Mobile account went up, and the downloads stopped.

Understandably, I am upset.

So here, for the edification of anyone in my situation, are the steps and alternative routes I've had to take to get the Market back up and running.

---------------------

Possibility 1: Get a VPN, and set it up on your Android phone.

This is likely the easiest and most effective way to get it done, aside from simply moving back to your home country.

Possible service providers are Freedur (Android setup info), Witopia (Android setup info), and 12vpn (Android setup info). I use Freedur, but fair warning, have not yet been able to get their VPN service working on my Android phone. The other two were mentioned in various Android support forums as being excellent and working fine with Android; I am currently in a customer support dialog with Freedur.

These three services are all available (i.e., not yet blocked) in China, and will all cost you from 3-5 USD per month. It is absolutely worthwhile to purchase a VPN service; besides being faster than free proxies, they are vastly more reliable and feature-rich. Most VPNs will also allow you to have simultaneous logins on a laptop/desktop and a mobile device, and most have clients for MacOS, Windows, and Linux in addition to compatibility (if not full-blown apps) for iOS and Android.

This will give you a secure connection through (usually) a UK or US server, meaning you have a little tunnel through which the real internet can leak through to your phone or computer.

 

Possibility 2: Use the Android Development Kit, various other hacks, and a VPN on your computer to download and install apps over USB to your phone.

This is really the only alternative I can think of if your phone doesn't support VPN, your VPN doesn't support your phone, VPN service isn't working for whatever reason on your phone, or you'd rather browse the Market in an emulator instead of on an actual device.

It is a hell of a hassle, and inconvenient, but it works, and is a decent stopgap until you get a working VPN or China and Google finally admit they love each other.

There are various helper apps that can help you out with this, but I'm going to outline the way to do this that depends upon the least amount of software.

NOTE: I do all this with the 1.5 version of the image, which IS NOT SUPPORTED by certain apps, such as Google Goggles. It would probably be safer to install the 1.6 version during Step 6, but I just found this out after writing up this whole piece of poop, so there you go.

I'm downloading 1.6 now, and will also be hosting it privately (a link is provided in Kumar Bibek's page, linked in the acknowledgments.

-----------------------------

1) First, you'll need to use a web proxy or a VPN to download the Android SDK (here), as anything related to Android development also seems to be blocked. If you have the option to, whitelist anything with "*.android.*" in a URL to go through your proxy. Any of the three VPNs mentioned above should work fine, along with software like "Freegate." As I assume this blog will get blocked by censors almost immediately after they find it, you'll be using one to see this anyway.

2) Once you download the Android SDK, extract the .zip somewhere convenient -- for the purposes of this tutorial I am assuming it will be on Windows, and to C:\android\ -- and run "SDK Manager.exe"

• This will not work.
• Don't worry.

3) Once the initial download fails, go into "Settings," and change the "HTTP Proxy Server" and "HTTP Proxy Port" to match your VPN settings.

• The Server should be "127.0.0.1" or "localhost," and the port varies depending on which VPN you're using (for Freedur, the default is 10174).
• Also make sure to look down in the "Misc" section on the same screen and tick the checkbox for "Force https://... sources to be fetched using http://..."

4) Go back to "Available Packages" and click "Refresh," which should now work just fine.

5) Choose "SDK Platform Android 1.5, API3, revision 4," along with the Usb Driver package, and hit install. With a typical Chinese network, running through a VPN, you're in for a bit of a wait, so go have a sandwich or a cup of coffee.

Once it's done, move on to the next step.

• If it fails repeatedly, change a VPN provider. Free proxy services like Freegate will generally not work properly for this, I believe because they periodically switch servers.

6) Download this file. I also have a local mirror on my personal server, e-mail me if the previous link goes down and I can send it to you. It should come down as "avd.rar" if you get it from the above link.

• Extract the RAR file (using 7-zip or something similar).
• Copy everything that gets pooped out into: C:\Users\[your user name]\.android\avd\
• This should be one folder called "AndEmu.avd" and one file called "AndEmu.ini"

7) Open the file "AndEmu.ini" in Notepad and edit it like so:

• On the second line, change the "path=" line to have the actual path to the AndEmu.avd folder, which should be: C:\Users\[your user name]\.android\avd\AndEmu.avd
• Save and close the file.

8) Go back to your SDK Manager, or run "C:\android\SDK Manager.exe" again.

• Click "Virtual Devices" -- there should be an entry named "AndEmu" in the list.
• If there is a green icon that looks like a torn piece of paper, click the AndEmu entry and then click "Repair." Click "OK" to any notices. The icon should turn into a green checkmark.
• Click "Start."
• This will flash open a few DOS windows, then finally you'll be presented with a window containing a graphic of a red Android phone (a G1, by the looks of it), and a keypad on the right.

9) This is actually a fully functioning instance of the Android 1.5 handset operating system. I assume you already generally know how to use it, but a couple tips:

• You can "flip open" the keyboard by using CTRL-F11.
• You can type using your normal keyboard, don't bother with clicking the keypad.

10) Set up the phone just as you normally would; enter your Google account, or create a new one, and sign in.

11) Go to the Market, find whatever app(s) you want, and install them. Remember to keep your VPN on during all this, as the reason we're going through this whole circus in the first place is because you need a VPN to get to the Market from China, and you can't use one on your phone for whatever reason.

• When you're done, go to C:\android\ and create a folder for your new apps, in this case we'll use C:\android\apks\

12) Once you have all the apps you want, open up a command prompt in Windows.

• Start, search for Command Prompt and run it (or use Windows+R and enter 'cmd' then hit enter).
• Go to C:\android\tools\ using the code:
  cd c:\android\tools
• Now you need to find out the name of your virtual phone. You do this by using the command:
  adb devices
• At this point there should just be one (if you have your actual phone plugged in there will be two, we'll get to that later), and I think the name will always start with the word "emulator." In my case the device's name is "emulator-5554."
• Now, since the emulator wraps up all those precious little program files you just installed into a single image file, you need to pull them out. You do this by using the code:
  adb -s [your device name] pull /data/app c:\android\tools
• It'll do some crunching, and when it's finished is when the real fun starts.

13) Open up your C:\android\apks\ folder and check out those names. You might be able to install them all in one go (I'm not sure but will include instructions for that in the next step), but since otherwise you'll be typing them one by one into the command prompt, feel free to rename them to something shorter.

• I'll give out some free advertising here and use com.google.zxing.client.android.apk, which is the application "Bar Code Scanner" by ZXing Team. I just renamed it to "qrScanner.apk".

14) Now it's time to plug in your real phone.

• First, make sure that "USB Debugging" is turned on.
• This setting will be under Home -> Menu -> Settings -> Applications -> Development.
• Then... plug it in. You should get a little notification on your phone that debugging is active.
• Check to make sure there's a new device by entering "adb devices" in the command prompt again. Note the name. In this case we'll just say it's GFWYOUFACE

15) In the Command Prompt (you still have one open at C:\android\tools\ right? If not, open a new one), enter the following (one for each of the apps you want to install):

• adb -s [your real phone's id] install [path and full name of the apk]
• So in this example, I'd be using:
  adb -s GFWYOFACE install c:\android\apks\qrScanner.apk
• It'll do some funky stuff, and should say "Success" then give you another prompt.

16) Check your phone; the app should be there.

I believe you can use this process to install any free/unpaid app from the Market. You can also use steps 14 and 15 to sideload apps that are not from the market, like... well, I'm not quite sure, but you can.

So, have fun.

------------

Now, there are several ways you can make this process go faster (.bat files and the like), and automate certain things for future use to make everything easier. I'm not going to deal with that, because I don't expect to be using this method very long, and to be honest I'm just too damn lazy to keep explaining this stuff.

That said, by no means did I come up with this on my own; this is simply a compilation of various sources of information around the web with the occasional injection of the word "poop," and a focus on the specific workarounds needed to get the Market working in China.

So, in case anyone actually reads this and finds it useful, I should give credit where credit is due. Many thanks to:

• HowToGeek's excellent write-up on how to install and run virtual Android systems on a PC.
• HowToGeek's excellent write-up on how to enable the Android Market in the emulator.
• A post by "Joe" in the comments section of the second write-up, explaining how to pull APK files out of the emulator.
• ExtremeTech's post on how to access the Android Market in a virtual Android machine, using a 1.6 build.
• Kumar Bibek, for doing all the hard work getting the Market onto not one, but two working system images, and being kind enough to share them with the whole world -- of course, we're only using the 1.5 build here (which seems to be enough for now).