How can I speed up import images into memory and make training more efficient

Asked Jun 20 '17 at 04:12

Active Jun 20 '17 at 04:12

Viewed 77 times

My memory is 32GB, windows 10, cpu 7700k, GTX1080

50k images[400MB] cost me 30 minutes. In a CAPTCHA example, a simple net and in tunning hyper-parameters step with small values of MaxTrainingRounds, just 15 minutes training time consuming.

If Mathematica Kernel crashes, then I should import images again, it's very often[long term] and annoying.

So, I can DumpSave, too many disk memory costs, acceptable.

Can I speed up the import this images for purpose of training network?

Any other methods to make the whole thing efficient? wdx, readlist as txt?

asked Jun 20 '17 at 04:12

HyperGroups

8,619
1
26
63

Did you try to store your images in one multipage file, like tif? I guess, reading of the one big file is faster for traditional hard drives. I've tried such tif's up to 2GB with 20k frames, the import time is ~ 1min. – Rom38 Jun 20 '17 at 04:21
You can refer my question How to use Mathematica to train a network Using out of core classification? And how do to generate a .h5 file – partida Jun 23 '17 at 05:10
@partida Hi, There is a bug in partida ... see https://mathematica.stackexchange.com/questions/148714/nettrain-always-crashes-in-out-of-core-training-with-netencoder-setting – HyperGroups Jun 23 '17 at 07:26
@Rom38 hi, that's fast, what's your images' dimension? 100100 or 300300? – HyperGroups Jun 25 '18 at 09:41
@HyperGroups, they are from 288288 to 768512. But of course, high-res is shorter.. – Rom38 Jun 25 '18 at 11:09

How can I speed up import images into memory and make training more efficient

0 Answers0