2

I have a nested dataset, which has been created to be the correct size from using Table, and I wish to replace parts of the dataset with the data. My dataset would look like (with placeholder values for the number of repeats etc, so that you can run the code yourself):

NumberOfRepeats = 3;
AllParameters = {58422.1`, 58427.6959583`, 58433.2919159`, 
   58438.8878726`, 58444.4838286`, 58450.0797838`, 58455.6757383`, 
   58461.2716919`, 58466.8676448`, 58472.4635969`, 58477.1595489`, 
   58480.3555014`, 58483.5514534`, 58486.7474049`, 58489.9433561`, 
   58493.1393068`, 58498.135255`, 58503.7312019`, 58509.3271479`, 
   58514.9230932`};
NumberOfExperiments = 5;
NumberOfWaveforms = 2;

NumberOfBlocks = 20;


(Create dataset for data)
Blocks = Table[<|"Block Number" -> i, 
    "Parameters" -> <|"Freq" -> AllParameters[[i]]|>, 
    "Waveforms" -> 
     Table[<|"Waveform Type" -> j, 
       "Repeats" -> 
        Table[<|"Repeat number" -> k, "Spectrum" -> {}, 
          "Fitting Parameters" -> {}|>, {k, NumberOfRepeats}], 
       "Fitting Parameter" -> {}, "Spectral Splitting" -> {}|>, {j, 
       NumberOfWaveforms}]|>, {i, NumberOfBlocks}];


DatasetRawDataAnomaly = 
 Table[<|"Experiment" -> i, "Data" -> Blocks|>, {i, 
    NumberOfExperiments}] // Dataset

What the code will generate prior to a replacement My issue is wanting to replace an element of that dataset. I have tried using ReplacePart Dataset after the replacement of spectrum in experiment 1, block number 1, wavefunction 1, repeat 1, but I need to make so many changes, that it is too slow, taking up to half an hour with the code I am running this in. An example of what I would do is:

DatasetRawDataAnomaly = ReplacePart[DatasetRawDataAnomaly,
   {ExperimentType, "Data", BlockType, "Waveforms", WaveformType, 
     "Repeats", RepeatCounter, "Spectrum"} -> IntensityType];

Where IntensityType is a list of lists of values of the form {{1,2},{3,4},{5,6}...}

I think running something like:

IntensityType = {{1, 2}, {2, 3}, {3, 4}};

DatasetRawDataAnomaly[1, "Data", 1 , "Waveforms", 1, "Repeats", 1 , 

{(<|#, "Spectrum" -> {IntensityType}|> &)}] (Ones for: Experiment number, 

Block 1, Waveform 1, Repeat 1);

would be quicker, but is there a way to change the whole dataset, rather than extracting the part and changing it?

Thank you for any help you can provide.

UPDATE: While it may not be a solution for everyone, the quickest way to do this is to avoid using the dataset, and convert to the dataset at the end. I.e. just use the association, where you can just select the element and equate it to a new value without having to use ReplacePart or anything. Then when you need to use the dataset, just convert the whole association into a dataset. This is much much quicker and still leaves one able to use the dataset functionality when needed.

Xyive
  • 115
  • 6
  • Hi and welcome to the Stack Exchange! Could you perhaps show two example datasets, one pre-replace and one post-replace? It would help me better visualise what you are aiming for. Even images of those datasets would help. Thank you! – Carl Lange Jan 14 '21 at 23:06
  • 1
    @CarlLange I have added some photos, which should help clarify (though the formatting looks different as not all the parts are full). I replace the entry in the dataset with lst in this example. Does this help clarify the issue? The issue is not how it can be done, as ReplacePart works. I am wondering if there is a quicker way to do it, as ReplacePart is incredibly slow. – Xyive Jan 15 '21 at 13:57
  • Maybe worth trying this idea? – Rolf Mertig Jan 18 '21 at 20:57
  • @RolfMertig I hadn't tried that, however, it doesn't seem to make any impact in the computation time. Implementing that code and then doing a quick check for timing, the two methods seem pretty similar:

    `AbsoluteTiming[

    For[i = 1, i <= 10, i++, DatasetRawDataAnomaly[[1, "Data", 1, "Parameters", "Freq"]] = i ]; ]

    AbsoluteTiming[ For[i = 1, i <= 10, i++, ReplacePart[DatasetRawDataAnomaly, {1, "Data", 1, "Parameters", "Freq"} -> i]; ]; ]` These give 3.50897s and 3.49784s, which is roughly similar.

     – Xyive Jan 19 '21 at 15:38
  • 1
    @Xyive Interesting. Please send a mail to support@wolfram.com and politely complain... – Rolf Mertig Jan 19 '21 at 15:42

0 Answers0