I had a backup mail server eat a bunch of messages and I want to just configure a simple server to send 450 for all messages.
However, I'm not sure this is a good idea because if the sending servers on retry do not attempt to send to the higher priority server, the messages can never be delivered.
I failed to find anything in RFCs on this, but do servers retry on a higher priority server or forever attempt on the same server that responded with 450?
When SMTP delivery fails (without a receiving permanent 5xy negative completion SMTP response) and there's no alternate/backup MX defined, IIRC the RFC's leave it up to the sender implementation if delivery is attempted again and how frequently and for how long subsequent delivery attempts will be made. The standards also leave it up to the sender implementation if and how soon temporary delivery delay notifications get returned to the original sender and/or how quickly a permanent delivery failure error notification is returned.
When more than one MX record is defined for the recipient domain then for every delivery attempt a standards compliant sender should attempt delivery to all MX records, one after the another and honouring their priority, until either one MX accepts the message ( the sender receives a 2xx positive completion SMTP response), or one MX rejects the message (with a permanent 5xy negative completion SMTP response) or until all have been contacted.
When none of (primary and or backup) MX records have permanently accepted or rejected the mail message for your domain, then I would expect that the same, sender implementation specific, failure path is followed as for the case when no alternate MX records exist. I other words: "the sender may queue the message for a later delivery attempt or give up".
The whole concept of a multiple MX records and a backup mail server under your own control is the fact that you control it and you don't have to rely on the queuing and retry policy of the original sender, or the absence thereof.
When you don't want that control; when you don't want your back-up MX to queue your mail but want to leave queuing and delivery retrying up to the sender, simply don't configure a backup MX record at all. That should have the same effect (if not a better effect) than setting up a backup MX pointing to a server that only generates errors and which in reality won't accept and queue your email.
Is that when your primary server goes offline and/or becomes unavailable due to an outage or planned maintenance, the RFC's and a correct implementation of the SMTP protocol by the sender should ensure that e-mail messages addressed to your domain get sent to your backup MX, where the mail will be accepted
and queued for as long as it takes for the (planned) outage to end.
Your backup mailserver (and not the sender) will control if/when mail delivery delay/failure notifications will be sent to the original sender or not.
Once the outage has ended and your primary mail server and mailboxes are online again, you can flush the queue on the backup MX and (almost) immediately receive all queued mail. With the ETRN SMTP command for example.
As suggested in the comment by jcaron:
Is there any reason to have a second MX sending 450s? Having nothing at all (i.e. no server answering on that port, or no second MX listed at all) should have the same result and would be less work... But be aware that whatever the solution, senders do not keep mail forever. While many will retry for days, some will have a much shorter timeout (and some will blacklist your address at the first error!)
The right solution to your XY problem is not to have a backup MX. There is never any need to have more than one MX. Every legitimate sender retries delivery for well over a day, often over a week. In the event that your server has downtime and cannot accept the mail immediately, leaving it as the responsibility of the sending party's mail system, rather than some sketchy third party queuing service, is the obvious right thing to do. Not only does it avoid your mail getting "eaten"; it also avoids trusting a third party with authority to intercept your mail, and it ensures the sender will be notified (by their own mail system) in the event that an extremely long outage renders their mail undeliverable.
As for your actual question, how retry works for SMTP is under-specified, and I'm not sure there is any guarantee that a conforming sender would loop back to trying your primary after finding it unreachable, rather than continuing to try to deliver to the fake secondary MX reporting temporary failures. So even without the above reasoning why you shouldn't have a secondary MX, I think it's a bad idea to try this particular hack.
In case it's relevant, my experience with this is 22+ years of continuously running a personal/small-site mail system, and never having used more than one MX.
