Read on Twitter
Today is a good day for the security of AWS VPCs. A new API was released: route53:ListHostedZonesByVPC. This addresses a long-standing (six years!) deficiency in the Route 53 and VPC APIs.
You can now list all private hosted zones associated with a given VPC. I'll try explain.
You can now list all private hosted zones associated with a given VPC. I'll try explain.
Private hosted zones are awesome and have been available since 2014 (I think). They allow you to create DNS records that are resolvable only within a VPC and can even override regular DNS.
Think of them as a magical cloud alternative to /etc/hosts that apply to all your EC2s!
Think of them as a magical cloud alternative to /etc/hosts that apply to all your EC2s!
There's obviously a lot of sensible uses for them. And even some less sensible but still understandable uses for them. Maybe you want google dot com to resolve to your own 10.1.2.3 for.. good reasons?
But what if someone wanted to do it for evil? How would you know?
But what if someone wanted to do it for evil? How would you know?
Firstly, something to know about private hosted zones (PHZs): you can associate a PHZ in account A with a VPC in account B.
This requires an API call in account A first (granting authorisation to B) and an an API call in account B, completing the association. Tada, it's done.
This requires an API call in account A first (granting authorisation to B) and an an API call in account B, completing the association. Tada, it's done.
Of course these API calls are recorded in CloudTrail. The first one is in the trail for account A and the second in the trail for account B.
But what if account B belonged to your employer and account A belonged to a malicious employee from 5 years ago? How would you audit it?
But what if account B belonged to your employer and account A belonged to a malicious employee from 5 years ago? How would you audit it?
I'm not sure about you, but I haven't had a great experience finding an SIEM / log aggregator that will not timeout looking for a needle in a haystack that is 5 years old and 26TB of compressed logs ago. Not to mention, why would I go looking when everything is fine?
Well what if that employee was especially patient? What if they registered a private hosted zone for http://s3.amazonaws.com and kept the entry correct for 5 years, only to change it last week?
And they changed it to something reasonable, like an IP in the published range for S3 - but not a valid one? How would *you* debug the sudden failure of all API calls to S3 from a given VPC?
You'd check route tables, you'd check the status page, you'd file support tickets, etc.
You'd check route tables, you'd check the status page, you'd file support tickets, etc.
Maybe you'd even log onto an instance and try some API calls. Failed. You'd try `dig http://s3.amazonaws.com ` and get a response. You even think to validate the IP is in the correct range.
But it's timing out and AWS haven't acknowledged a service issue - friggin' status page!
But it's timing out and AWS haven't acknowledged a service issue - friggin' status page!
I don't know about you, but I wouldn't bother checking CloudTrail since day one for this huge account. I might check the last few weeks because a) it's quick and b) it's been working all this time! But that yields nothing.
Anyway, this has been a problem for a long time and few people have cared - or even known - about it. Maybe because it's an unlikely attack vector. @Ryan_Jarv knew about it - and published a blog on it.
https://blog.ryanjarv.sh/2019/05/24/backdooring-route53-with-cross-account-dns.html
https://blog.ryanjarv.sh/2019/05/24/backdooring-route53-with-cross-account-dns.html
But even he and @0xdabbad00 - experts in the field of AWS security - initially misunderstood the finer details and didn't realise the issue at first. So if they didn't immediately click, what chance do the rest of us have? https://twitter.com/Ryan_Jarv/status/1226236752618295296
tl;dr Thank you AWS for releasing this new API. Now it's time for me to go invoke it for every VPC in every account in my AWS org.
Which makes me think, it would be nice if Security Hub did that for me and highlighted zone associations with accounts outside my org.
Which makes me think, it would be nice if Security Hub did that for me and highlighted zone associations with accounts outside my org.
Addendum: twitter only posted half my thread at first because it thought "google dot com" (with a real dot) was a malicious link. I'm surprised that posting a thread isn't an atomic transaction. Lame.
