How can I connect to a MongoDB Replica Set behind a proxy?

Question

I have a MongoDB Replica Set on the cloud service. For security reason, the replica set is available to the cloud's internal network.

I followed that cloud service's guide and setup a proxy to each member of the replica set, on the proxy-server:

0.0.0.0:27017 -> member1-private-ip:27107
0.0.0.0:27018 -> member2-private-ip:27107
0.0.0.0:27019 -> member3-private-ip:27017
...

I'm able to connect to every member of the replica set from a public network in standalone mode:

mongoUri = new MongoClientURI("mongodb://usr:pwd@proxy-server-public-ip:27017/db")  ;
client = MongoClient(mongoUri);

But when I try connect it in replica set mode:

mongoUri = new MongoClientURI("mongodb://usr:pwd@proxy-server-public-ip:27017,proxy-server-public-ip:27018,proxy-server-public-ip:27019/db?replicaSet=replcaSetName");
client = MongoClient(mongoUri);

I would fail with connection error, since the replicat set tell the driver to use each member's internal addresses (inaccessible from public network).

p.s: I could connect to the replica set in replica set mode on the proxy server.

How can I connect to my replica set behind a proxy server?

---

Update: I use public address of the proxy server while connecting.

Answer 1

I have recently had a similar situation. Consider some DNS cheating, where the client uses "real" DNS names, and the mongodb replica set members use the same names but override them in /etc/hosts to point to themselves.

If you have 3 members, then have three DNS names that are what your client will use to route to the proxy, e.g.:

member1.mynetwork.mydomain -> (proxy address)
member2.mynetwork.mydomain -> (proxy address)
member3.mynetwork.mydomain -> (proxy address)

Then, on your mongodb replica set members, create an /etc/hosts entry on each box that matches, but points to their own host IP, e.g.:

/etc/hosts:

10.1.1.11 member1.mynetwork.mydomain
10.1.1.22 member2.mynetwork.mydomain
10.1.1.33 member3.mynetwork.mydomain

Build the replica set config with each member's "host" field as member1.mynetwork.mydomain:27017 and so on.

Configure the client to connect to something like:

[member1.mynetwork.mydomain:27017,member2.mynetwork.mydomain:27017,member2.mynetwork.mydomain:27017]

The replica set will respond to the driver with a cluster definition based on its own replica set member list, which will be the same names:

hosts=[member1.mynetwork.mydomain:27017,member2.mynetwork.mydomain:27017, member3.mynetwork.mydomain:27017]

And you should be in business.

If your proxy situation can't host multiple DNS names, you can change the ports in all configs (including local bind ports on the mongodb members themselves) to the 27017/27018/27019 scheme.

Some, including myself, will consider the local /etc/hosts overrides to be icky and a hack, depending on your server / VM / container management situation.

But if you're in a bind, I think it's a more elegant hack than rewriting the mongodb responses.

Answer 2

I've asked this question in May, but now I'm here to answer my own question.

As discussed in the comments, MongoDB server will return a list of its members, with their configured addresses. Since a mananged MongoDB replica set is configured with private addresses, MongoDB server will supply member's private addresses.

To resolve this issue, we need a dedicated proxy for Mongo client connections. The proxy should intercept MongoDB server's response to isMaster command and override the private address to the server's public address or your proxy server's address. After client received these intercepted address, they would able to connect these address in replicaSet mode.

Here is some code in Node.js:

clientConn.on("data", dataHandler(proxyConn, function(data) {
  const msg = new WireMessage(data);
  if (msg.isCommand("isMaster")) {
    remoteClient.recordForInterception(msg);
  }
  mongoConn.write(data);
}));

mongoConn.on("data", dataHandler(proxyConn, function(data) {
  const msg = new WireMessage(data, {skipBody: true});
  var changed = false;
  if (remoteClient.shouldInterceptReply(msg)) {
    // To Intercept message, we need a full parse
    msg.parseBody();
    changed = remoteClient.interceptReply(clientConn, msg);
  }
  if (changed) {
    // Serialize intercepted data
    data = msg.serialize();
  }
  clientConn.write(data);
}));

interceptReply = function (conn, replyMessage) {
  if ('isMaster'.toLowerCase() !== replyMessage.toLowerCase()) {
    return false;
  }
  var doc;
  if (replyMessage.body.metadata) {
    doc = replyMessage.body.metadata;
  } else if (replyMessage.body.documents instanceof Array && 
    replyMessage.body.documents.length > 0) {
    doc = replyMessage.body.documents[0];
  } else {
    this.logger.warn("No document to handle: %s.", replyMessage.toString());
    return false;
  }    
  return interceptHosts(doc, conn,  hostMappings);
};

interceptHosts = function (doc, conn, mappings) {
  if (doc.hosts) {
    var hosts = doc.hosts;
    for (var i = 0; i < hosts.length; ++i) {
      var host = hosts[i];
      hosts[i] = getReverseAddress(host, conn, mappings);
    }
  }
  if (doc.primary) {
    doc.primary = getReverseAddress(doc.primary, conn, mappings);
  }
  if (doc.me) {
    doc.me = getReverseAddress(doc.me, conn, mappings);
  }
  return doc;
};

function getReverseAddress(endpoint, conn, reverseAddressMapping) {
  var hostMap = reverseAddressMapping[endpoint];
  if (hostMap && hostMap.host === "0.0.0.0") {
    // If we are listening on ANY address, use 
    //   the effective address the client connect us.
    return conn.localAddress + ":" + hostMap.port;
  } else if (hostMap) {
    return hostMap.host + ":" + hostMap.port;
  } else {
      return endpoint;
  }
}

Changing the client library to map private addresses to public address should be a alternative solution. But it might be a hard work to support all languages and push your customized library to your partner's development machine.