Thursday, December 6, 2012

Securing Azure Virtual Machines or I got hacked and how you shouldn’t

I have been working with Azure for over a year now and this is the first time I have had a machine compromised.  I will say that this entire incident has been a good learning experience.  I think that anyone can learn from my mistakes here.

Thinking back this event is interesting from many angles.

  • a default machine provisioning behavior is opening a door that you (as the customer) needs to remember to close or not open in the first place.
  • the machine is really out there, in the wild, innocent, vulnerable.  Not protected in your datacenter behind all the other protections.
  • your IT Security folks might not yet be engaged in how to deal with deploying into public clouds.
  • you are doing test, exploration, or other work on systems that are not going to production.

 

Let’s go back in time to the week of November 26. 

My innocent Virtual Machine was compromised by an attack against the RDP protocol.  The well known RDP port 3389 was the vector.

The attacker succeeded in obtaining access, changing the local administrator password and pwning my box.  I can only image that it was then ‘sold’ on some botnet as later it ended up with a PayPal phishing site.

Then it began the process of being capacity to begin finding other vulnerable folks. This is where I noticed what was happening.

(BTW – the MSFT folks that handle this type of incident were extremely helpful in working out what happened, in what order, and how.)

 

First, lets look at the environment where I put my machine.

It was in Azure.  Placing any machine in Azure is equivalent to placing it in the corporate DMZ.  There is a load balancer between you and the outside world, and public endpoint definition(s) that define the public ingress port(s).

Beyond that, you can actually do bunches of different configurations of Virtual Networks, or Gateways.  You can link Services.  All kinds of stuff.  But remember, all IaaS machine to machine network traffic within a Service is wide open.  There are no private endpoints (as in a PaaS Service).

That said, any IT Pro would immediately say ‘duh’ or ‘you idiot’.  Yep.  Nothing I can comment there.

 

There are some default IaaS behaviors that are enabling this;

  • automatically opening an RDP endpoint for management
  • requiring complex passwords (false sense of security)
  • local administrator is not disabled
  • no real guidance around practices (not that anyone RTFM’s any longer anyway – they simply SearchTFM)

PaaS Roles don’t have these behaviors.  And the RDP connection is secured with a Certificate and a secondary user account.

Some of the Azure MVPs found my original MSDN forum thread and beat my blog post out.  My IT Pro background is leading me to some different ideas, and I felt I needed to tell a bigger story.

And, as I think about this; remember that you are administering these lockdowns at the same time that your machine could be getting probed, or brute force attacked.  So, choose a logical order, define a strong password at the point to machine creation (however you do it).  At least it gives you a fighting chance.

 

So, this is where i mention a bit of guidance.  Here are some options to pick and choose from.

Some initial mitigation suggestions to choose from:

  • Use the PaaS roles of Web or Worker as the front end machines whenever possible.  They are already hardened.
  • Rename the local administrator account.
  • Disable the local administrator account and create some uncommonly named user account for administrative access.
  • Choose strong plus complex passwords, or passphrases.  Not simply one or the other.  The OS can enforce complexity but not strength. 
    • A dictionary attack is likely to hit “P@ssw0rd” but it is unlikely to hit “Just a city boy, born and raised in South Detroit”.
    • My favorite example of password strength: http://xkcd.com/936/ 
  • Denying user access after X failed logon attempts (lock the account).  This is a Local security policy if not domain joined, or a Domain policy if joined.  Consider an automatic (timed) unlock as well, or you could have no recourse but to destroy your machine.
  • Do not allow the creation of the default RDP public endpoint.  This is only possible through the API / PowerShell.  Or delete the auto created endpoint after creating the machine in the Portal.
  • Only create the RDP endpoint when remote administration is necessary, and removing it after.  But remember that we are human, and unless you have some interface doing this for you, you will probably forget at some point.
  • Remove the RDP endpoint and use the Virtual Network Gateway feature of the Azure Virtual Network for secured remote administration without public endpoints.  This requires some ground based router, and the VPN is slow, but your ports are closed.
  • Remove RDP endpoint & use Azure Connect.  This is limited to IPv6 TCP traffic only, but that should cover anything required to manage the OS.
  • Avoid 3389 as the public port (I noticed my compromised machine specifically scanning for this port to spread itself) by using a port in the ephemeral range.
  • Use the Windows Advanced Firewall rules and define them appropriately.
  • Use Windows IP Security Policies and tightly define the sources from which RDP traffic can be accepted from.  This is highly effective, but a pain to set up.
  • Monitor the machine.  Azure provides metrics through the portal and API.  Discover a baseline.  Use an agent within the machine.  This only detects the compromise after it happens and is not preventative.
  • Take a snapshot of the clean state.  This is not a point and click thing in Azure today, but you can work this out using the Storage cmdlets through destroying your machine, making the diff disk, and reincarnating the machine.

 

Some of these practices are simply security through obscurity but, in this case, I think it’s OK.

  • Changing the name of the administrator account dramatically increases the search space for a username/password pair.
  • Changing the RDP endpoint is going to increase the search space for a TCP port/username/password trio.

I didn’t mention antivirus.  I won’t go into my theories there.  My personal opinion is that if all the tightening measures are done and you still get hacked, most likely your AV would have failed as well.

Always be ready for the possibility of rebuilding a machine.  Never assume you can simply move forward, or even install a backup.  The best you can do is revert to a known clean state.

 

Good luck!

8 comments:

BrianEh said...

Since writing this one behavior seems to have changed.
The first VM in a Service no longer appears to default to port 3389 for the RDP endpoint. One good step forward.

Anonymous said...

If there is too much to loose on the hacked machine there is a way to restore access. Fortunately the bot does not change the password. I found an account created by the name 2bb with administrator prvileges.

The information in your blog is mighty useful but will be more valuable if recovery solution is offered.

We can talk if you want to. Don't worry I'm not a software services salesman :)

BrianEh said...

IMHO, there are limited ways to recover your data. And the methods are little different than they would be for a VM in the enterprise.

Create a new VM and attach the disk. (BTW, you cannot just attach an OS disk as a Data disk, you must destroy the machine, unregister the disk from the Azure Library. Wait a bit. Register the disk as a data disk, and then you can move forward.

I had the best luck with downloading your virtual disk and recover. Again, no different than in the enterprise.

Sine all machines are in a DMZ, it would have to be a highly secured machine that I would even trust with data that I could not afford to lose. Or I did not have another copy of. Something that valuable, needs to be protected in a far better way than the front end machine of an Azure Service. It would at least be buried deeper within my application at another tier or another Service.

Anonymous said...

hanks for the post!

Now you should rename the user Administrator when creating a new VM, that is good to avoid hacking.

BrianEh said...

Yes, I have a script for that now. someplace...

krsna said...

does azure provide disabled open internet for VMs

BrianEh said...

"disabled open internet" - I don't fully understand what you mean.

You get VMs. The VMs can freely communicate out. However, you can only communicate with your VM over a publicly open port or you deploy a VPN using the Virtual Network feature (which is not free) and you reach your VMs through the tunnel.

One big point of my post is to not leave 3389 exposed. It is actively probed and targeted. If you need a secure way to get to he VMs, use something else, like a VPN or GoToMyPC / GoToAssist.

Anonymous said...

This must have been a terrible situation.
Things get better. When you create a new vm now the user Administrator is not there only the user you used when confiuring the vm.