You can't make an omelette without breaking eggs - and this also applies to the construction of our services. But what happens if we are not thorough enough when it comes to decommissioning? We regularly create JFrog instances dynamically for the development of our service and also dismantle them after testing. The process is automated using Terraform, Crossplane and it's been deployed on Kubernetes. However, one small but crucial point has been overlooked: the deletion of all associated DNS entries.
At first glance, this seems harmless. After all, the instance does no longer exist and it's just a development environment. But the reality is different! We were recently contacted by a security researcher, a so-called bug bounty hunter. He drew our attention to a serious security vulnerability caused by the non-deletion of DNS entries.
These orphaned DNS records, also known as "dangling DNS records", are still valid since they were not removed after decommissioning the service. In a public cloud, an IP address is being released immediately and assigned to next customer. The bug bounty hunter got our previously used IP address and a matching DNS record. Fortunately, this situation was not abused and we got informed about the vulnerability.
What are the dangers of "dangling DNS records"?
- Loss of the subdomain: This can lead to negative press, damage to image and loss of trust.
- Cookie harvesting: Attackers could steal cookies that are stored on the subdomain.
- Phishing campaigns: Attackers could use the subdomain for phishing, which would be difficult for users to recognise as it's being issued by a trusted company.
- Theft of certificates: Let's Encrypt certificates are valid for 90 days. An attacker could steal these certificates and use them for phishing.
- Malware distribution: Attackers could host and distribute malware on the subdomain.
Tools
There are numerous tools developed by security researchers to detect "dangling DNS records". These tools are freely accessible and can be used by anyone. Here are a few examples:
- Eipfish: A tool that searches for dangling DNS using AWS Lambda Services. Eipfish on GitHub
- Paydirt: A tool that searches for such services on AWS and DigitalOcean. Paydirt on GitHub
- Sublist3r: Uses various search engines and passive DNS services. Sublist3r on GitHub
kmsec.uk provides an illustrative example in their blog post Passive Takeover, which shows how easy it is to take over a subdomain on DigitalOcean.
How can dangling DNS records be avoided?
Here you are some advises:
- Automation: Consider DNS records as part of the documentation process and ensure that they are deleted.
- Use passive DNS services: Regularly check the databases of passive DNS services and ensure that there are no dangling DNS records. There are services available such as SecurityTrails:Data Security, Threat Hunting, and Attack Surface Management Solutions for Security Teams or VirusTotal.
- Regularly check DNS records: There's a neat online service available, which is free to query a single domain: https://hardenize.com.
- Monitoring & Alerting: We haven't yet found a useful tool to monitor dangling DNS records.
How do the well-known cloud providers manage this?
The well-known cloud providers are aware of this problem:
- Azure: provides a PowerShell tool to find "dangling DNS records" to prevent subdomain takeovers
- AWS: provides a brief documentation about dangling delegation records in Route 53
- Google Cloud: There's no official documentation, but a Github project providing some help: https://github.com/domain-protect/domain-protect-gcp
Why did pay the bug bounty hunter?
As already mentioned in the introduction, where there's wood, there are chips. We are grateful that he shared this vulnerability and brought it to our attention. Despite of the high level of automation, the complexity is increasing and there are always things that can be overlooked. It is important for us to take something positive from this experience and deal with it properly.
We deleted all our dangling DNS records right after the tip-off and closed the security gap. Everyone at 4data is aware that security is paramount. And we paid a reward for this notice, as it helped us to improve and make our services safer.
Why did we write this blog post?
By admitting this and not trying to cover it up, we gain the trust of our customers. And while sharing this experience, we can help others to avoid doing the same mistake.
Conclusion: Never stop learning.