EventHandler can be used to catch various mouse events (mouse up, mouse down, mouse clicked, mouse dragged). Use MousePosition to add some intelligence.
Example:
DynamicModule[{col1 = Green, col2 = Blue}, Graphics[
{
EventHandler[
Dynamic[{col1, Disk[]},
ImageSize ->
Tiny], {"MouseClicked" :> (col1 =
col1 /. {Red -> Green, Green -> Red})}],
EventHandler[
Dynamic[{col2, Disk[{1, 1}]},
ImageSize ->
Tiny], {"MouseClicked" :> (col2 =
col2 /. {Blue -> Yellow, Yellow -> Blue})}]
}
]
]

The circles can be clicked independently. An action is defined for each object separately.
Amazingly, this even works for 3D Graphics:
DynamicModule[{col1 = Green, col2 = Blue},
Graphics3D[
{
EventHandler[
Dynamic[{col1, Sphere[]},
ImageSize ->
Tiny], {"MouseClicked" :> (col1 =
col1 /. {Red -> Green, Green -> Red})}],
EventHandler[
Dynamic[{col2, Sphere[{1, 1, 1}]},
ImageSize ->
Tiny], {"MouseClicked" :> (col2 =
col2 /. {Blue -> Yellow, Yellow -> Blue})}]
}
]
]

Here's some code to test the scalability:
With[{n = 100},
DynamicModule[{col = RGBColor @@@ RandomReal[{0, 1}, {n, 3}]},
Graphics[
Table[
With[{j = i},
EventHandler[
{Dynamic[col[[j]]], Disk[RandomReal[{-10, 10}, {2}]]},
ImageSize -> Tiny,
{"MouseClicked" :> (col[[j]] =
RGBColor @@ RandomReal[{0, 1}, {3}])}
]
], {i, n}
]
]
]
]

This is reasonably responsive. However, try the version with n=1000:

On my PC the color changes only after about three seconds. So, if you need to interact with many objects this may not be the best solution.
EventHandlerandDynamiccolor on each primitive that you want to track. I don't think you can develop a robust solution (or maybe near impossible) by using just one outerEventHandlerand infer which primitive is being clicked. So, in this case, it's just the same as the example under "Scope" in the doc page you linked to. – rm -rf Jan 06 '14 at 19:19