pyspark.pandas.
to_datetime
Convert argument to datetime.
or DataFrame/dict-like
If ‘raise’, then invalid parsing will raise an exception
If ‘coerce’, then invalid parsing will be set as NaT
If ‘ignore’, then invalid parsing will return the input
strftime to parse time, eg “%d/%m/%Y”, note that “%f” will parse all the way up to nanoseconds.
unit of the arg (D,s,ms,us,ns) denote the unit, which is an integer or float number. This will be based off the origin. Example, with unit=’ms’ and origin=’unix’ (the default), this would calculate the number of milliseconds to the unix epoch start.
If True and no format is given, attempt to infer the format of the datetime strings, and if it can be inferred, switch to a faster method of parsing them. In some cases this can increase the parsing speed by ~5-10x.
Define the reference date. The numeric values would be parsed as number of units (defined by unit) since this reference date.
If ‘unix’ (or POSIX) time; origin is set to 1970-01-01.
If ‘julian’, unit must be ‘D’, and origin is set to beginning of Julian Calendar. Julian day number 0 is assigned to the day starting at noon on January 1, 4713 BC.
If Timestamp convertible, origin is set to Timestamp identified by origin.
Return type depends on input:
list-like: DatetimeIndex
Series: Series of datetime64 dtype
scalar: Timestamp
In case when it is not possible to return designated types (e.g. when any element of input is before Timestamp.min or after Timestamp.max) return will have datetime.datetime type (or corresponding array/Series).
Examples
Assembling a datetime from multiple columns of a DataFrame. The keys can be common abbreviations like [‘year’, ‘month’, ‘day’, ‘minute’, ‘second’, ‘ms’, ‘us’, ‘ns’]) or plurals of the same
>>> df = ps.DataFrame({'year': [2015, 2016], ... 'month': [2, 3], ... 'day': [4, 5]}) >>> ps.to_datetime(df) 0 2015-02-04 1 2016-03-05 dtype: datetime64[ns]
If a date does not meet the timestamp limitations, passing errors=’ignore’ will return the original input instead of raising any exception.
Passing errors=’coerce’ will force an out-of-bounds date to NaT, in addition to forcing non-dates (or non-parseable dates) to NaT.
>>> ps.to_datetime('13000101', format='%Y%m%d', errors='ignore') datetime.datetime(1300, 1, 1, 0, 0) >>> ps.to_datetime('13000101', format='%Y%m%d', errors='coerce') NaT
Passing infer_datetime_format=True can often-times speedup a parsing if its not an ISO8601 format exactly, but in a regular format.
>>> s = ps.Series(['3/11/2000', '3/12/2000', '3/13/2000'] * 1000) >>> s.head() 0 3/11/2000 1 3/12/2000 2 3/13/2000 3 3/11/2000 4 3/12/2000 dtype: object
>>> import timeit >>> timeit.timeit( ... lambda: repr(ps.to_datetime(s, infer_datetime_format=True)), ... number = 1) 0.35832712500000063
>>> timeit.timeit( ... lambda: repr(ps.to_datetime(s, infer_datetime_format=False)), ... number = 1) 0.8895321660000004
Using a unix epoch time
>>> ps.to_datetime(1490195805, unit='s') Timestamp('2017-03-22 15:16:45') >>> ps.to_datetime(1490195805433502912, unit='ns') Timestamp('2017-03-22 15:16:45.433502912')
Using a non-unix epoch origin
>>> ps.to_datetime([1, 2, 3], unit='D', origin=pd.Timestamp('1960-01-01')) DatetimeIndex(['1960-01-02', '1960-01-03', '1960-01-04'], dtype='datetime64[ns]', freq=None)