Why don't more organizations use inside-to-inside NAT or similar solutions to allow NAT hairpins?

Question

Inside-to-inside NAT aka NAT loopback solves hairpin NAT issues when accessing a web server on the external interface of an ASA or similar device from computers on the internal interface. This prevents DNS admins from having to maintain a duplicate internal DNS zone that has the corresponding RFC1918 addresses for their servers that are NATted to public addresses. I'm not a network engineer, so I might be missing something, but this seems like a no-brainer to configure and implement. Asymmetric routing can be an issue but is easily mitigated.

In my experience, network admins/engineers prefer that systems folks just run split-dns rather than configuring their firewalls to properly handle NAT hairpins. Why is this?

Possibly, but when using DNS on AD Domain Controllers (a common scenario) views don't exist. — MDMarra, May 17 '13 at 12:30
As a security measure, I don't want to publish my AD DNS zones to the Internet, so I already need split zones. — Zoredache, May 17 '13 at 15:33
Why would you need to publish your AD zone to the Internet? If your AD is ad.example.com or similar (like it should be!), then this problem will exist for all public example.com DNS entries and nothing internal is published externally. Of course, if you've named your AD the same as your public presence you must use split-DNS, but that's not a best practice AD design. — MDMarra, May 17 '13 at 15:51
I'd add that DNS views are only easier to troubleshoot if clients never move between them. Things can go oddly wrong when clients take a cached internal result outside. — LapTop006, May 18 '13 at 11:19
Clients shouldn't cache DNS answers, that's what DNS servers are for. I know Windows is very fond of silent (and deadly) cacheing, but that's one of the many reasons why I think it's unfit for production use. — MadHatter, May 19 '13 at 16:39
Totally unrelated to this question, but you should know that it's trivial to disable Windows client-side DNS caching — MDMarra, May 19 '13 at 16:41

the-wabbit · Answer 1 · 2013-05-25T12:11:56.160

12

Obviously, there cannot be the definite answer for this, but I could think of a number of reasons:

Elegance: NAT is not very elegant to begin with, but a necessity with IPv4's restricted address space. If I can avoid NAT, I will. With IPv6 the problem is going away at any rate.
Complexity: especially in a case where you do not have just one single "core" router creating all of your security boundaries, filter configuraions can become quite tricky. Either that or you would have to spread and maintain the NAT rules across nearly all of your router devices.
Reference: wherever firewall admins are different folks than the rest of the server admin team, they might be difficult to reach. In order to ensure that changes are not delayed by the firewall admin's backlog (or vacations), the option to keep the responsibility in the same team is chosen
Portability and common practice: Using DNS views is "just the thing everybody knows is done" to solve this problem. Not every boundary router would support loopback NAT in a straightforward way, less people are likely to know how to set it up correctly in your specific environment. When troubleshooting network issues, the crew would need to be aware of the hairpin NAT configuration and all of its implications - even when they are chasing an apparently unrelated problem

edited May 25 '13 at 12:11

answered May 21 '13 at 12:12

the-wabbit

41,007

1
In this situation NAT is being used anyway. This isn't adding NAT where there wasn't NAT before 2. In my example, they are all handled by the same device or cluster of devices. 4. As stated in my comment above, a very common scenario is using AD Domain Controllers for DNS internally. Windows DNS does not support views. Also, I've found that most somewhat modern router/firewall firmwares support NAT hairpinning in one way or another.

MDMarra

May 21 '13 at 12:59

1

@MDMarra some remarks: 1. - "there would be one NAT weirdness less to care about" is what I basically meant, 2. - your question does not explicitly say so, but with just a single router it obviously will be easier to handle, 4. with internal DNS in place which is mandatory for all clients, you do not need views - you just would create an internal zone and get the same effect as you've mentioned in your question, the reasoning above still applying. – the-wabbit May 21 '13 at 13:25

1

What NAT weirdness, though? What specific behavior would this introduce that causes problems? I worked at a university that had NAT hairpinning configured on a Netscreen cluster for 100+ external hosts and we never had a problem. – MDMarra May 21 '13 at 14:57

Also note that using hairpin NAT in this scenario is actually making it look more like the IPv6 solution, because under IPv6 the system will have one globally-reachable address; hairpin NAT simulates that behaviour. – Paul Gear May 21 '13 at 22:22

@MDMarra the most outstanding "NAT weirdness" for the "hairpin NAT" case would be the non-optimal routing path, meaning the rewritten packets would have to travel most of the path back they came in through. Seeing this in a packet dump when troubleshooting connectivity is confusing at best. I believe others have already mentioned the asymmetric routing issue in their answers, which is related. There is no doubt you can make it work just fine. But it is not worth the effort in a majority of cases IMO. – the-wabbit May 25 '13 at 12:09

Parts 2 & 4 of this are the core of it, i think. – Paul Gear May 29 '13 at 21:10

Dennis Kaarsemaker · Accepted Answer · 2013-05-19T16:33:43.093

There are a few reasons why I wouldn't do it:

Why put extra strain on the DMZ routers and firewall if you don't need to? Most our internal services are not in the DMZ but the general corporate area, with proxying services in the DMZ for occasional remote access. Doing inside-to-inside nat adds more load to the DMZ, which in our case would be significant.
If you don't do DNAT + SNAT, you will get asymettric routing, which is notoriously tricky to get right. So you SNAT and lose source IP infomation. However, it is bloody useful to link source IPs to people for troubleshooting purposes. Or occasionally nerfshooting purposes in cases of stupidity. "Hey this IP is doing something wonky on unauthenticated service X" "Oh, let's look in the imap server logs who it is".

Just a note, if your firewall is doing L3 routing and you have your routes for your external and internal clients take a hop at the firewall, you shouldn't have to worry about asymmetric routing, right? — MDMarra, May 21 '13 at 15:01

Paul Gear · Answer 3 · 2013-05-27T21:05:42.927

7

Disclaimer - this is a flamebait answer.

A common reason i think solutions like this are avoided is an irrational fear/hatred of NAT on the part of network engineers. If you want to see some examples of discussion on this, check out these:

http://packetpushers.net/show-86-connect-to-the-ipv6-internet-for-free-using-tunnelbroker-net/
http://networkingnerd.net/2012/04/02/ipv6-nat-and-the-sme-a-response/ (Tom's blog seems to keep attracting rather fervent NAT/IPv6 discussion)
http://libertysys.com.au/blog/nat-is-evil-but-not-bad (my blog)

From what i can tell, a lot of this fear stems from Cisco's poor implementations of NAT (so in that sense it may not be irrational), but to my mind the "classic" network engineer is so well-schooled in the "NAT is bad" worldview, that he or she can't see obvious examples like this where it makes perfect sense and actually simplifies the solution.

There you go - downvote to your heart's content! :-)

edited May 27 '13 at 21:05

answered May 21 '13 at 22:20

Paul Gear

4,507

1

I don't know if I'd consider it an irrational fear. Experience has taught me that NAT can break or do strange things with lots of stuff. – May 21 '13 at 22:40
1

@Kce But if you're already using NAT on your external interface, what difference does configuring NAT loopback make? – MDMarra May 21 '13 at 22:48
@MDMarra - None really. I'm was making the rather pedantic point that sometimes the network services team's fear of NAT is not irrational. – May 21 '13 at 22:52
I don't fear NAT, I hate it. – Michael Hampton May 25 '13 at 15:15
1

Edited post to include hatred. :-) – Paul Gear May 27 '13 at 21:06

hookenz · Answer 4 · 2013-05-22T22:58:38.100

My guess is:

Split DNS is more easily understood.
NAT Hairpin uses up resources on the router while split-dns doesn't.
Routers may have bandwidth limitations that you don't get through a split-dns setup.
Split-dns will always be better performing as you avoid a routing/NAT steps.

On the plus side for hairpin NAT,

Once it's setup it just works.
No issues with DNS caches for laptops moved between work network and public internet.
DNS for a server is only managed in one place.

For a small network with low traffic requirements to an internal server I'd go with hairpin NAT. For a larger network with many connections to the server and where bandwidth and latency are important, go with split-dns.

score 2 · Answer 5 · edited Apr 13 '17 at 12:14

2

From my perspective, this changed a bit between the Cisco Pix to ASA transition. Lost the alias command. And in general, accessing the outside address from the inside interface on a Cisco firewall requires some sort of trickery. See: How do I reach my internal server on the external IP?

We don't always need to maintain a duplicate internal DNS zone, though. The Cisco ASA can redirect DNS queries for external addresses to internal addresses if configured on the NAT statement. But I prefer to keep an internal zone for the public DNS zone in order to have that granularity and to be able to manage this in one place rather than step to the firewall.

Typically, there are only a few servers that may require this within an environment (mail, web, a few public services), so it hasn't been a tremendous problem.

edited Apr 13 '17 at 12:14

Community

1

answered May 17 '13 at 12:23

ewwhite

198,150

Is it trickery if its supported and documented by Cisco? Sure it's a little more work, but does that make it tricky or hacky? – MDMarra May 17 '13 at 12:27
Trickery, in that people don't know/expect/think about it if they're not already aware. It's a common question: How do I reach an external IP from the inside of my Cisco firewall. – ewwhite May 17 '13 at 12:30
Typically, there are only a few servers that may require this within an environment maybe in some places, but I worked at a university with 100+ servers/devices in a DMZ and I also worked at a testing/certification provider with 40+ servers spread across three DMZs. For smaller companies you might only have one or two servers exposed to the outside, but this isn't necessarily the case for everyone. – MDMarra May 17 '13 at 13:00
If the servers are in a DMZ, this is less of an issue, right? – ewwhite May 17 '13 at 13:12
In this instance "DMZ" means connected to the external inferface of said firewall with default deny rules between the zones in place. – MDMarra May 17 '13 at 13:16

score 2 · Answer 6 · answered May 23 '13 at 00:02

I can think of a few reasons:

Many organizations have split DNS already as a result of not wanting to publish all of their internal DNS information to the world, so it isn't much additional effort to handle the few servers that are also exposed via a public IP.
As an organization and its network grows larger, they often separate the systems servicing internal folks from servers servicing externals, so routing to the external ones for internal usage may be a much longer network path.
The fewer modifications that intermediary devices make to packets, the better it is when it comes to latency, response time, device load, and troubleshooting.
(very minor, admittedly) There are protocols that NAT will still break if the NATing device doesn't go beyond the headers of the packet and modify data in it with the new IP address(es). Even if it is just a case of institutional memory, it is still a valid reason for people to avoid it, especially if they are dealing with a protocol that they aren't 100% certain about.

score 0 · Answer 7 · answered May 25 '13 at 02:26

0

If I were going to use NAT loopback I would be a bit worried about how the NAT device will handle spoofed source addresses. If it doesn't check which interface the packet came in, then I could spoof internal addresses from the WAN and send packets to the server with internal addresses. I couldn't get replies, but I might be able to compromise the server using an internal address.

I would setup NAT loopback, plug into the DMZ switch and send packets in with spoofed internal source addresses. Check the server log to see if they were received. Then I would go to the coffee shop and see if my ISP is blocking spoofed addresses. If I found my NAT router wasn't checking source interface, I probably wouldn't use NAT loopback even if my ISP is checking.

answered May 25 '13 at 02:26

Kent England

1

Sorry, I may be misunderstanding what you're saying, but RFC1918 addresses aren't routed over the internet, so I don't see what trying to spoof them across the WAN is going to do. They won't even make the first hop. – MDMarra May 25 '13 at 02:30
The destination address is the public address of the NAT on the port that is mapped to the server. The source address is one of the private IP addresses on the inside of the NAT. Source addresses aren't used in routing and aren't checked by every public router. The only difference from a legit internal query is that the packet is coming in from the WAN. – Kent England May 25 '13 at 02:55

Why don't more organizations use inside-to-inside NAT or similar solutions to allow NAT hairpins?

7 Answers7

Linked