How to reshape an array faster?

Question

I have a large array (arr) of size 22 X 225 X 225 and I want it to be reshaped to a size of 22 X 50625. ArrayReshape can do it perfectly well. However, it is too slow.

Is there any way to make it faster?

There is hardly any faster way than ArrayReshape. For the matrix A = RandomReal[{-1, 1}, {22, 225, 225}];, my machine runs B = ArrayReshape[A, {22, 50625}]; // RepeatedTiming in 0.00072 seconds. Try to use packed arrays. — Henrik Schumacher, May 03 '18 at 15:37
@HenrikSchumacher I was thinking to flatten each of the 22 arrays of 225 X 225. — user36426, May 03 '18 at 15:41
That could be done with Flatten[A, {{1}, {2, 3}}] or Flatten /@ A, but is much slower, at least in the current version (11.3). — Henrik Schumacher, May 03 '18 at 15:43
Almost the same for me in 11.1. ArrayReshape: .0064, Join@@@A: .0064, Flatten/@A: .0048. Interestingly, ArrayReshape is the only one that does not unpack. — Theo Tiger, May 03 '18 at 16:08
I get on order 0.01 sec for both ArrayReshape and Flatten/@A (Flatten marginally faster) with v11.1.1/linux. Something change w/ 11.3? ( I doubt your hardware is 10x faster than mine.. ) — george2079, May 03 '18 at 16:11
.. (oops I was using AbsoluteTiming . RepeatedTiming does give a much faster time here ) — george2079, May 03 '18 at 16:23
@george2079 Might be so. In version 11.0.1, the example above needs 0.00279, hence four times longer. I also believe to recall that combinations of Flatten and Partition have been faster than ArrayReshape in earlier versions. — Henrik Schumacher, May 03 '18 at 16:25

Henrik Schumacher · Answer 1 · 2018-05-03T19:17:56.240

Indeed, somebody must have put some effort in optimizing ArrayFlatten. That's good to see!

Here a benchmark for different methods and array sizes:

Needs["GeneralUtilities`"];
funs = <|
   "ArrayReshape[#,{...}]&" -> (ArrayReshape[#, {Dimensions[#][[1]], 
        Times @@ Dimensions[#][[2 ;; 3]]}] &),
   "Flatten[#,{{1},{2,3}}]&" -> (Flatten[#, {{1}, {2, 3}}] &),
   "Map[Flatten,#]&" -> (Map[Flatten, #] &),
   "Compile[...][#]&" -> (Compile[{{a, _Real, 2}},
        Flatten[a],
        RuntimeAttributes -> {Listable},
        Parallelization -> True,
        RuntimeOptions -> "Speed"
        ][#] &),
   "Partition[Flatten[#],...]&" -> (Partition[Flatten[#], 
       Times @@ Dimensions[#][[2 ;; 3]]] &)
   |>;
nlist = 2^Range[2, 9];
benchmark = Benchmark[Values[funs], RandomReal[{-1, 1}, {#, #, #}] &, 2^Range[2, 9]];

g = ListLogLogPlot[
  AssociationThread[
   Map[Style[#, Bold, FontFamily -> "Courier"] &, Keys[funs]],
   benchmark
   ],
  Joined -> True,
  PlotMarkers -> Automatic,
  PlotLabel -> Row[{"Timings on Mathematica ", $Version}],
  ImageSize -> Large,
  AxesLabel -> {"n", "Time (s)"},
  Ticks -> {nlist, Automatic}
  ]

Both performed on Intel 4980 HQ @ 2,8 GHz, 16 GB 1600 MHz DDR3.

score 5 · Answer 2 · answered May 03 '18 at 16:25

5

This is a little too long for a comment. On my machine (MMA 11.2, Win7-64bit), there is a signifcant difference in performance between ArrayReshape and a direct Flatten application, with ArrayReshape being roughly 3x faster:

arr = RandomReal[1, {22, 225, 225}];

RepeatedTiming[ArrayReshape[arr, {22, 50625}];]    (* Out: {0.0065, Null} *)
RepeatedTiming[Flatten[arr, {{1}, {2, 3}}];]       (* Out: {0.019, Null}  *)

RepeatedTiming[Flatten /@ arr;]                    (* Out: {0.2, Null}    *)

Mapping Flatten is very slow in comparison.

The mapping method also unpacks the array:

Developer`PackedArrayQ@arr                                   (* Out: True  *)

Developer`PackedArrayQ@ArrayReshape[arr, {22, 50625}]        (* Out: True  *)

Developer`PackedArrayQ@Flatten[arr, {{1}, {2, 3}}]           (* Out: True  *)

Developer`PackedArrayQ@(Flatten /@ arr)                      (* Out: False *)

answered May 03 '18 at 16:25

MarcoB

67,153
18
91
189

Yepp, using Map should not be encouraged as much as several years before. When I have to map over packed arrays, I try to do so with a Listable CompiledFunction`. – Henrik Schumacher May 03 '18 at 16:28
@HenrikSchumacher oh no, that's not what i want to hear! a lot of my code is strongly based around mapping; i guess i'll be on the lookout for alternate methods in the future. – Ben Kalziqi May 03 '18 at 17:35
@BenKalziqi Of course, you could do much worse than Mapping, e.g. with For. ;) Of course, you have to balance effort for prototyping, debugging, and maintaining vs. pure runtime. And Map is usually a good trade-off. – Henrik Schumacher May 03 '18 at 17:46

How to reshape an array faster?

2 Answers2