Looking at the numerous meanings and usages of the preposition "zu" we can only give an incomplete answer here to cover some special cases where we have a loose rule indeed.
Whenever we use "zu" we also indicate some spacial correlation, a direction, or other defined relations.
Der Kellner beugte sich zu dem Gast herab.
Alle Besucher sind zu dem Fest gekommen.
When we consider this then a meaning of a sentence may change:
Er hat mir ein Muster gebracht. - He gave me a sample.
Er hat ein Muster zu mir gebracht. - He brought the sample to me.
Er gehört zu mir. He became a part of me.
Er gehört mir. "He is mine."
Now let's look at both of your examples:
Er hat etwas zu mir gesagt und nicht zu jemand anderem.
Er hat [zu] mir etwas gesagt, was andere nicht hören sollten.
Both have nearly the same meaning, but in the first case it is important that the speech was directed to you. Hence you can not leave out the "zu". This is different in the second case. The direction is not important, you leave out the "zu". When you inserting it there it is not wrong but it became a rather superfluous filler.
Sie hat das Geld zu mir gebracht und nicht zu Paul.
Sie hat mir das Geld gebracht und nicht Paul.
Again, direction is emphasized in the first sentence whereas it is the person whom she gave the money to who plays the important role in the second sentence. This would in theory not only work with "bringen" but also with "geben" but here it is simply not used this way. Therefore it sounds awkward and wrong to a native's ear.
It may be confusing, but in another context "geben" indeed is used with "zu":
Meine Urgroßeltern mussten ihren Sohn zu Pflegeeltern geben.
In this case it is not a direction but a change in relation being responsible for using "zu" (like in "Er gehört zu mir." above).