1

I'm struggling to filter some data from an accelerometer. I need to do it in python (and not just on the device collecting the data) to fix the data I have, so that I can build some classification algorithm

I have the following data on ROLL from an accelerometer (1st graph). I tried the following:

  • transform the outliers using winsorize (2nd graph)
  • and then, filter the data using Butterworth filter (3rd graph)

original, transformed and filtered

Problem is that winsorization technique does not fit my problem here, it creates points at the top and the bottom for the extreme values, and it generates those peaks after filtering. And it does not take into account when those outliers are within the given limits as well.

would someone have a solution to correctly remove those outliers? Any python function? I'm happy to remove completely those outliers, rather than transform them. Below is the dream, expected output after filtering:

the ideal world

If I focus on 1 piece of outliers, we can see the following (my data distribution is a bit weird, I have a couple seconds every few seconds): data when zoomed in

jupyter notebook below

%pylab inline
warnings.filterwarnings("ignore",category=DeprecationWarning)
import pandas as pd, numpy as np        # Data manipulation 
import matplotlib.pyplot as plt
from scipy.stats.mstats import winsorize
import scipy.signal as signal

df=pd.DataFrame.from_csv("/tmp/file.csv")
df.reset_index(level=0, inplace=True)
df=df.iloc[2000:18000]

df_with_outliers_trimmed=df.copy()
df_with_outliers_trimmed["ROLL"]=winsorize(df_with_outliers_trimmed["ROLL"], limits=0.05)


df_filtered=df_with_outliers_trimmed.copy()
# design the Butterworth filter
N  = 1    # Filter order
Wn = 0.03 # Cutoff frequency
B, A = signal.butter(N, Wn, output='ba')
df_filtered["ROLL"] = signal.filtfilt(B,A, df_filtered["ROLL"])


x_axis='Date'
plt.rcParams["figure.figsize"] = (17,10)
fig1, (ax1,ax2,ax3) = plt.subplots(nrows=3, ncols=1)

xlim_start=min(df["Date"].iloc[0],df["Date"].iloc[0])
xlim_end=max(df["Date"].iloc[-1],df["Date"].iloc[-1])
# xlim_start=datetime.datetime(2018,7,3,12, 51)
# xlim_end=datetime.datetime(2018,7,3,12, 59)
xlim=[xlim_start,xlim_end]

ax1.plot(df[x_axis],df['ROLL'], 'r.')
ax1.set_ylabel("ROLL")
ax2.plot(df_with_outliers_trimmed[x_axis],df_with_outliers_trimmed['ROLL'], 'r.')
ax2.set_ylabel("ROLL with outliers trimmed")
ax3.plot(df_filtered[x_axis],df_filtered['ROLL'], 'r.')
ax3.set_ylabel("ROLL filtered")
ax1.set_xlim(xlim)
ax2.set_xlim(xlim)
ax3.set_xlim(xlim)

csv file: https://file.io/yqf3ei

Jeremie
  • 113
  • 1
  • 5
  • Could you comment on how you want the data to look like? I guess, you could delete all points that deviate more than a given threshold from the average of their surrounding....put this a pure blind guess... – Dschoni Aug 10 '18 at 12:56
  • good point, just added a picture of what I'm trying to get. yes I could indeed do that, I was wondering if there was some already existing filter that does that automatically and that I could apply on all my datasets with a rather good level of confidence. I'm probably gonna work on that if I cannot get any better solution. – Jeremie Aug 10 '18 at 14:12

1 Answers1

3

Have you tried just median filtering the data ? That's what I'd try as a first step. Winsorizing really doesn't work on a data set that has a changing mean.

It looks like python has scipy.signal.medilt to perform this.

Peter K.
  • 25,714
  • 9
  • 46
  • 91