0

I have a blender python script that I used to produce a lot of different dataset for my machine learning project. The problem that I have is that I use a lot of nested loop to simulate a lot of different scene. This is my full script

import bpy
import os
import glob
from math import pi

scene = bpy.context.scene
directory = '/home/cgal/testimageSFS/Hasil_blender/'
list_ob = '/home/cgal/3d_scans/*.ply'
lampu = bpy.data.objects["RGB"]
lampu_utama = bpy.data.objects["Lamp"]

counter = 0

rgb = {'r':(1,0,0) ,'g': (0,1,0) ,'b' : (0,0,1)}

for img in glob.glob(list_ob):

    #create face object
    bpy.ops.object.select_all(action='DESELECT')
    bpy.ops.import_mesh.ply(filepath=img)
    obj = bpy.context.selected_objects[0]
    nama = str(counter)
    obj.name = nama

    #modify face
    face = bpy.data.objects[nama]
    face.scale =(0.00001,0.00001,0.00001)
    face.rotation_euler.x = pi/2
    face.location.z = 3.23

    #attach skin
    mat = bpy.data.materials.new(name = "Skin")
    mat.use_nodes = True #Make so it has a node tree

    #Add the vertex color node
    vc = mat.node_tree.nodes.new('ShaderNodeVertexColor')
    #Assign its layer
    vc.layer_name = "Col"
    #Get the shader
    bsdf = mat.node_tree.nodes["Principled BSDF"]
    #Link the vertex color to the shader
    mat.node_tree.links.new( vc.outputs[0], bsdf.inputs[0] )
    face.data.materials.append(mat)
    #take Pic
    for x in range(-10,11,2):
        for y in range(-7,-2,1):
            for z in range(-6,11,2):
                #change center_lamp position
                lampu_utama.location.x = x
                lampu_utama.location.y = y
                lampu_utama.location.z = z
                xs = str(x+10)
                xs = xs.zfill(2)
                zs = str(z+6)
                zs = zs.zfill(2)
                utama_loc =xs + str(y+7) + zs
                for pow in [1000,5000,10000]:
                    #change RGB power
                    lampu.data.energy = pow
                    power = str(pow)
                    power = power.zfill(5)
                    for i in range(-5,6,1):
                        #change RGB light position
                        lampu.location.x = i

                        #take normal picture
                        for ob in scene.objects:
                            if ob.type == 'CAMERA':
                                lampu.hide_render=True
                                bpy.context.scene.camera = ob
                                file = os.path.join(directory,nama+'_'\
                                +'_'+utama_loc+'_'+str(i+5)+'_'+power+'_a' )
                                bpy.context.scene.render.filepath = file
                                bpy.ops.render.render( write_still=True )

                        for n,color in rgb.items():
                            lampu.data.color = color
                            col=n

                            #record RGB
                            for ob in scene.objects:
                                if ob.type == 'CAMERA':
                                    lampu.hide_render = False
                                    bpy.context.scene.camera = ob
                                    file = os.path.join(directory,nama+'_'\
                                    +'_'+utama_loc+'_'+ str(i+5)+'_'+power+'_'+col )
                                    bpy.context.scene.render.filepath = file
                                    bpy.ops.render.render( write_still=True )

    #delete face object
    bpy.data.objects[nama].select_set(True)
    bpy.ops.object.delete()
    counter=counter+1
    break

The problem is I keep having my PC freeze at some point (not fixed sometimes in 6 min sometimes in 7 min) even when I run the script without GUI, this is the command that I used blender myscene.blend --background --python myscript.py. So is there any solution to fix this besides reducing the number of possible scene on my script? Thank you

My PC specs :
ubuntu 18.04
i7 3770
gtx 1080ti
16 GB of RAM

Full blend file

link to one of .ply I used : download here

Wanttobepro
  • 23
  • 6
  • How many ply files? Some parts could be put before the main loop. like deselecting all, getting the camera list. Also could create one material for all as it seems it does not depend on anything. You can also loop over the files in cli instead of doing it inside blender. – lemon Mar 12 '20 at 08:56
  • There is a 10 .ply files but the freeze already happened when I just used 1 ply files as you can see I put break at bottom. For material, I am not quite sure what you suggest since it will be different model later on when I erase the break and I don't know how to make a loop with CLI since all the object are blender, can you elaborate on that? – Wanttobepro Mar 12 '20 at 09:09
  • if crashes on the first file, the CLI "trick" is not the point. How many render results do you have before the crash? Also what is happening if you render manually without script? – lemon Mar 12 '20 at 09:13
  • Around 10.000 images with size of 256x256 although sometimes before reaching that already freeze. I don't understand what you mean by render manually, if I render it manually it works fine. I used to run this script without the x y z loop and it run just fine until the last .ply file but since I need more variation for my dataset I add those – Wanttobepro Mar 12 '20 at 09:19
  • 10000 means you don't go to the end, I presume. Trying to help, few suggestions: look at memory/cpu usage when running it. Reduce the loops (ranges, steps, etc. for test purpose) to see what is happening. Eventually try it without the very low scaling (just an intuition). – lemon Mar 12 '20 at 09:51
  • I see, I will try. For the cpu usage usually around 30-50% while for GPU I don't think it uses much like below 1000 while the capacity I have is 11000 if i remember correctly. Is there any way to make use of this spare GPU? Like using CUDA to help doing the process? – Wanttobepro Mar 12 '20 at 12:11
  • Could you upload the blend file you are using so that we can see all the settings? I could test it using some ply I've. Can use this site for uploading https://blend-exchange.giantcowfilms.com/ and paste the result link here. – lemon Mar 12 '20 at 12:14
  • I have edited my post with my blend. Thank you for helping me this far – Wanttobepro Mar 12 '20 at 12:22
  • See https://blender.stackexchange.com/questions/7358/python-performance-with-blender-operators Agree with lemon's suggestion to glob and loop in a batch script (shell script, bash python whatever) and process the imports one at a time. Also consider setting up markers for each camera and render an animation with a frame per camera. https://blender.stackexchange.com/questions/43764/bind-camera-to-marker-via-python – batFINGER Mar 12 '20 at 12:44
  • Hi, so the problem already occurred on my 1st .ply object so I don't think import is the problem for now and I only use 1 camera on this project and to be honest the reason I used for loop is because I don't know how to correctly select the camera on script. I do find a good point from the post you mention and will try to implement those low level API to make it more efficient – Wanttobepro Mar 12 '20 at 13:16
  • FYI, i'm testing it with a 134M bytes ply file. This is running (from inside blender) and have 400 results so far. The memory usage is stable up to 3GB and gpu/cpu also... in progress. But will take much more than 10 minutes on my computer. – lemon Mar 12 '20 at 13:24
  • What is your ply file size? – lemon Mar 12 '20 at 13:30
  • My .ply files is 4MB which is lower than the one you used. On my PC it is also stable the GPU usage is 600-700 MiB only while CPU usage is 37-40%. Which is why I dont understand why it crash at some point – Wanttobepro Mar 12 '20 at 13:51
  • Took your ply file... so far about 16k pictures rendered (final result should be around 64 000 images). But no issue running it directly from inside Blender. I think I'll stop that in few and test again from command line. But I do not have the same env (Windows, I7 proc and RTX gpu). – lemon Mar 12 '20 at 14:37
  • ok, I'm over 11k images running from command line. No issue so far. Probably there is something in either Blender linux version or in your environnement. Will comment again when finished... we'll see... – lemon Mar 12 '20 at 15:19
  • bad news: here the 65340 images were generated correctly. Bad news as no way to understand what is hapenning for you. So, maybe check all versions (Blender drivers) test again, or consider to report a bug for the linux version... + when solved consider rendering in jpg format for instance (lighter than png) – lemon Mar 12 '20 at 17:38
  • Okay really appreciate your help. I will try to use windows version and see if the bug persist and if still yes, I will try to run it on different PC – Wanttobepro Mar 13 '20 at 01:45

1 Answers1

1

Ok so apparently when I run it on same PC but with windows OS it can run without any crash so far. So I will submit this into bug for linux later on. Thank you for all the helpful comment especially lemon who go as far as trying my code in his/her PC

Wanttobepro
  • 23
  • 6
  • FYI that's probably the same case as https://blender.stackexchange.com/a/170197/19156 . Could you give it a try under Linux? – lemon Mar 19 '20 at 13:14
  • Actually I forgot to update but my windows also crash when I tried to run on 2nd time. So I tried using it on my friend's PC and it can run without problem. Maybe the problem is the PC I am using – Wanttobepro Mar 21 '20 at 05:46
  • Consider "placing lampu.hide_render=True" (or False) outside of the "for ob in scene.objects:" loops. – lemon Mar 21 '20 at 08:02
  • Yeah thanks again for the help, I will change it for future creation – Wanttobepro Mar 23 '20 at 03:34