8

I need to convert tons of date strings into absolute times. This generates an example data set of 10,000 strings:

dateStrings=DateString[#,{"Year","-","Month","-","Day"," ","Hour",":","Minute",":","Second"}]&/@
              Range[3576302100-10000,3576302100];

I think the simplest way is 

AbsoluteTime /@ dateStrings

and this gives exactly the result I want. But on lists of hundreds of thousands of dates, it is quite slow. On the example data set, it takes about 5 s. I can speed it up it a bit by telling AbsoluteTime what date format to expect:

AbsoluteTime[{#, {"Year", "-", "Month", "-", "Day", " ", 
                  "Hour", ":", "Minute", ":", "Second"}}
] & /@ dateStrings;

And this takes about 3.8 s.

How can the performance be improved further?

Sjoerd C. de Vries
  • 65,815
  • 14
  • 188
  • 323
ArgentoSapiens
  • 7,780
  • 1
  • 32
  • 49
  • Did you try to evaluate this function? Something like f[x_Integer] := Evaluate[DateString[ x, {"Year", "-", "Month", "-", "Day", " ", "Hour", ":", "Minute", ":", "Second"}]]; and f /@ Range[3576302100 - 10000, 3576302100]? For me this is about 4 times faster. But there is a hint that x can not be interpreted as a date or time input. Perhaps there is a way to define f in a way that this warning does not occur. – dotcs Apr 30 '13 at 20:48
  • @chromate, thanks, but you're performing the reverse of the desired conversion. The input data are strings. – ArgentoSapiens Apr 30 '13 at 20:55
  • Ahh sorry, my fault. Unfortunately the same approach in the other direction is not faster than your solution. Perhaps somebody else knows how to speed things up in this case. – dotcs Apr 30 '13 at 21:13

2 Answers2

11

You can get about 100 times faster by using Java, without any particular tuning, but you will have to provide the date format explicitly. Here is the solution based on Java reloader.

Implementation

  1. Load the Java reloader

  2. Compile the following class:

    JCompileLoad@
"
  import java.text.ParseException;
  import java.text.SimpleDateFormat;
  import java.util.*;

  public class JAbsoluteTimeFinder{  
    public static double[] absTime(String[] dates,String format) throws ParseException{
     Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone(\"GMT\"));
     double[] result = new double[dates.length];
     int ctr = 0;
     SimpleDateFormat formatter = new SimpleDateFormat(format);
     for(String d: dates){
         Date dt = formatter.parse(d);    
         calendar.setTime(dt);            
         result[ctr++]= (long) (calendar.getTimeInMillis()/1000);
     }   
     return result;      
  }
}";

  3. Define the top-level Mathematica function as 

    ClearAll[getJAbsoluteTime];
getJAbsoluteTime[dates : {__String}, format_String] :=
   JAbsoluteTimeFinder`absTime[dates, format] 
      + AbsoluteTime[{1970, 1, 1, 0, 0, 0}] + 3600*$TimeZone;

Benchmarks

AbsoluteTime/@dateStrings;//AbsoluteTiming

(* {5.554688,Null} *)

getJAbsoluteTime[dateStrings,"yyyy-MM-dd HH:mm:ss"];//AbsoluteTiming

(* {0.043945,Null}  *)

getJAbsoluteTime[dateStrings,"yyyy-MM-dd HH:mm:ss"]==AbsoluteTime/@dateStrings

(* True *)

It may well be that one can get this still faster by tuning the solution or using a different Java library.

Community
  • 1
Leonid Shifrin
  • 114,335
  • 15
  • 329
  • 420
6

AbsoluteTime is faster on DateLists as compared to arbitrary DateStrings. Since the date format is known in this case, converting the strings to DateLists first gives a speed improvement of a factor of 10:

AbsoluteTime /@ (ToExpression /@ (StringSplit[#, {" ", "-", ":"}] & /@ dateStrings));

Not as fast as the java programming variant, but good enough for government work ;-)

Leonid Shifrin
  • 114,335
  • 15
  • 329
  • 420
hippo3773
  • 1,246
  • 6
  • 10
  • 1
    both StringSplit and ToExpression will work for a list of strings without the Map which isn't only slightly shorter but also a little faster than your version: AbsoluteTime /@ ToExpression[StringSplit[dateStrings, {" ", "-", ":"}]] – Albert Retey May 05 '13 at 21:04