Is there any way to improve performance of scene.ray_cast on scenes with thousands of collection instances?

Question

I'm using scene.ray_cast to check for objects between two points in a scene with thousands of collection instances. For some reason performance per ray fired seems to be tied to the number of objects visible. When I cull objects further than 200 meters from the camera and ray origin point the speed of raycast increases dramatically, however the process of setting the viewport visibility of these objects is very slow so it isn't a good solution. Any insight into why this happens is appreciated

Answer 1

Some suggestions.

Will speculate somewhat, sans any question code.

1. Use mathutils.kdtree.KDTree to find the objects within a distance. Example.

2. Use foreach_set to set the viewport visibility of the objects from a list of values. This avoids iterating over the objects, like a "batch set" from a list. Looping over large numbers of items is always a rate determining step in a python script. Using numpy in conjunction with index list from step 1, can very quickly set the property value for 1000s of objects.

Test script. Hides (hide viewport) all objects outside of 10 (dist) bu from (0, 0, 0) (co)

import bpy
from mathutils.kdtree import KDTree
import numpy as np
context = bpy.context
choose some collection, or scene objects etc.
coll = context.collection

obs = coll.all_objects
n = len(obs)
print(n)
build a kdtree
tree = KDTree(n)

#load in the locs
for i, ob in enumerate(obs):
    tree.insert(ob.matrix_world.translation, i)
tree.balance()
objects within dist bu of co
dist = 10
co = (0, 0, 0)
obs_r = [i for _, i, _ in tree.find_range(co, dist)]
print(obs_r)  # indices of obs, eg [2, 3, 5, 6, 22]
'''
test print to show all within dist of origin
for i in obs_r:
    print(obs[i].name, obs[i].location.length)
'''
hide = np.ones(n) # all true (hidden)
set objects within dist to not be hidden
hide[obs_r] = 0
#print(hide)
set the viewport visibility to show only for obs within dist
coll.all_objects.foreach_set(
        "hide_viewport",
        hide,
        )
call raycast....

Somewhat related https://blender.stackexchange.com/a/194328/15543