API: Is there any faster way to add/link material nodes?

Question

I'm looking at an importer addon that needs to create thousands of image texture nodes and links but with the nodes.new() and links.new() methods there are huge slowdowns; it's currently taking 20 seconds to create a couple thousand materials and close to 10 thousand nodes/links (it takes 5 minutes on the heaviest import which has over 5 thousand materials). I presume this is because of internal update functions and I'm wondering if there's a way to prevent them at least until the create/link operations are done.

All I could try up to this point was batching all the link operations together, unfortunately there was no change, unlike with scene linking.

Since the code and data involved are complex, here's a snippet that emulates the bottleneck but to a smaller degree (takes ~45 seconds to run here), I believe the importer itself is slower due to the added node tree complexity of the shaders, which are in node groups.

import time
import bpy
group = bpy.data.node_groups.new('dummy group', 'ShaderNodeTree')
for i in range(5):
    group.inputs.new('NodeSocketFloat', str(i))
start = time.time()
for mat in range(5000):
    mat = bpy.data.materials.new(name = str(mat))
    mat.use_nodes = True
    tree = mat.node_tree
    nodeGroup = tree.nodes.new('ShaderNodeGroup')
    nodeGroup.node_tree = group
    for i in range(5):
        nodeTex = tree.nodes.new('ShaderNodeTexImage')
        tree.links.new(nodeTex.outputs[0],nodeGroup.inputs[i])
print((time.time() - start))

Are you sure this is not an XY Problem (https://en.wikipedia.org/wiki/XY_problem)? Perhaps what you really need is an efficient way to use multiple images in a single shader - by combining them to a single texture atlas? For example, here's a shader that uses a GIF converted to a vertical spritesheet of 12 frames: https://i.imgur.com/t751aiX.png — Markus von Broady, Apr 29 '21 at 22:18
@MarkusvonBroady Well, while I agree there should be a better way to address the volume of data, it's beyond my scope; files already exist this way and I did not design the "pipeline" this is in. As for XY, I do not know. I've refactored the slowest parts of this addon to great speedups, except this one. In the large import, those 5 minutes setting up materials are more than parsing, instancing and linking the thousands of objects involved. I've timed bits of this code and while texture loading takes its fair time, 6/20s or 30/~300 on large, I'm p confident the bottlenecks are on the node tree — Aini, Apr 29 '21 at 23:10

batFINGER · Accepted Answer · 2021-04-30T15:25:39.057

Make 1 material and copy it.

In as much as I agree with the comment of @MarkusvonBroady in that investigating making a single material and passing an argument re which image to use, especially if texture image is the only difference per material. eg akin to Is possible play video in reverse mode into Texture Image node? with a static input like Object.pass_index.

Can get some improvement in script above by making the material once and copying in N - 1 times. (Note tested with 500, since life's too short for 5000 on my crap machine ATM)

import time
import bpy
group = bpy.data.node_groups.new('dummy group', 'ShaderNodeTree')
for i in range(5):
    group.inputs.new('NodeSocketFloat', str(i))
start = time.time()
mat = bpy.data.materials.new(name="0")
mat.use_nodes = True
tree = mat.node_tree
nodeGroup = tree.nodes.new('ShaderNodeGroup')
nodeGroup.node_tree = group
for i in range(5):
    nodeTex = tree.nodes.new('ShaderNodeTexImage')
    tree.links.new(nodeTex.outputs[0],nodeGroup.inputs[i])
def cp(i):
    m = mat.copy()
    m.name = f"{i}"
    return m
even quicker without renaming
mats = [mat.copy() for i in range(1, 500)]
or consider using generator.
mats = [cp(i) for i in range(1, 500)]

print((time.time() - start))

after running a few times this was 47x quicker than using question code. (with 500)

That works great! The snippet with 5000 took half a second here. A basic implementation on the addon is showing great promise. Thank you. — Aini, Apr 30 '21 at 12:36

API: Is there any faster way to add/link material nodes?

1 Answers1

even quicker without renaming

mats = [mat.copy() for i in range(1, 500)]

or consider using generator.