Here is a sample result from a small project I did to partition ecg data. My approach was a “switching autoregressive HMM” (google this if you haven’t heard of it) where each datapoint is predicted from the previous datapoint using a Bayesian regression model. I created 81 hidden states: a junk state to capture data … Read more
You could just shift the to column and perform a direct subtraction of the datetimes. df[‘overlap’] = (df[‘to’].shift()-df[‘from’]) > timedelta(0) Applying this while grouping by id may look like df[‘overlap’] = (df.groupby(‘id’) .apply(lambda x: (x[‘to’].shift() – x[‘from’]) > timedelta(0)) .reset_index(level=0, drop=True)) Demo >>> df id from to 0 878 2006-01-01 2007-10-01 1 878 2007-10-02 2008-12-01 … Read more
Question: How to do that on a Spark Dataframe in an efficient way? Spark DataFrame is simply not a good choice for an operation like this one. In general SQL primitives won’t be expressive enough and PySpark DataFrame doesn’t provide low level access required to implement it. While re-sampling can be easily represented using epoch … Read more
seasonal_decompose() requires a freq that is either provided as part of the DateTimeIndex meta information, can be inferred by pandas.Index.inferred_freq or else by the user as an integer that gives the number of periods per cycle. e.g., 12 for monthly (from docstring for seasonal_mean): def seasonal_decompose(x, model=”additive”, filt=None, freq=None): “”” Parameters ———- x : array-like … Read more
Convert your x-axis data from text to datetime.datetime, use datetime.strptime: >>> from datetime import datetime >>> datetime.strptime(“2012-may-31 19:00”, “%Y-%b-%d %H:%M”) datetime.datetime(2012, 5, 31, 19, 0) This is an example of how to plot data once you have an array of datetimes: import matplotlib.pyplot as plt import datetime import numpy as np x = np.array([datetime.datetime(2013, 9, … Read more