0

There are instances when neural networks, after being trained continually on a subset of data, tend to drastically lose their performance. I've attached an image below depicting the same. I am looking for an algorithm to detect this sudden fall in performance. enter image description here I'm quite new to this area and therefore don't have any idea where to start. My initial guess would be to first smooth the plot, so as to reduce the noise.

Edit 1: I highly appreciate the good advice that I have received in the comments and answers. I have added some code that generates live time-series, that'll help better understand my problem. The code is designed to run in Jupyter Notebook cells.

import matplotlib.pyplot as plt
from IPython.display import clear_output
import time
import numpy as np
def test():
    accuracy = []
    for _ in range(1000000000):
        time.sleep(1)
        accuracy.append(np.random.randint(30))
        clear_output(wait=True)
        plt.figure(figsize=(20,5))
        plt.plot(accuracy)
        plt.show()
test()
desert_ranger
  • 125
  • 6
  • It's not clear to me where the "sudden change" that you're looking to identify appears in the plot. I'm not familiar with the details of your end application. – Jason R Jul 12 '22 at 17:36
  • @JasonR Thanks for your comment. I have modified the image. I'd like this question to be domain independent and just get a signal processing perspective: How does one identify a sudden drop in a rising plot? – desert_ranger Jul 12 '22 at 17:46
  • Would simple thresholding work here? Either absolute (drops below 20) or relative (drops to less than 10% of the average of the last 10 values). – Hilmar Jul 12 '22 at 18:14
  • @Hilmar Thank you! I think thresholding would probably work. Could you please elaborate? Is there a package or library that does this? – desert_ranger Jul 12 '22 at 18:38
  • @desert_ranger we have no idea what produced this graph, but the thing is, whatever you used to produce this graph has the numbers you need to average. Since you know what an average is, not quite sure what you'd need a library for – just basic math. – Marcus Müller Jul 12 '22 at 19:05
  • @MarcusMüller I need to do this process in real-time. I don't think it is trivial and it would be great if there was a python library somewhere that does it. – desert_ranger Jul 12 '22 at 19:17
  • 2
    ah, so it's python. Yes, calculating the average over ten numbers in python is trivial, pretty sure: it's quite literally sum(values[-10:]) / 10. That's it. That's the code. – Marcus Müller Jul 12 '22 at 19:30
  • Related – OverLordGoldDragon Apr 13 '23 at 16:32

2 Answers2

2

The change in any signal with respect to another variable is defined by it's derivative with respect to that variable. In order to compute the "derivative" when you have a discrete signal (like in your case), (by asuming that the samples are spaced by a unity) you should use the finite difference in the following way

 s_change = s - s_old
 s_old = s

"s" is the current sample, "s_old" is inizialized to zero the first time then and it is updated to contain the value of the previous sample.

if "s_change" is huge it means that the signal suddenly changed, if its positive it means it's a sudden increased, if it's negative it means that it's a sudden drop.The bigger it's value the bigger the sudden change.

You can average each N samples and also compute the s_change each N averaged samples if you wanna filter out some noise.

I hope this is what you were looking for.

Dan Boschen
  • 50,942
  • 2
  • 57
  • 135
Chemes Allioua
  • 36
  • 1
  • Ricky Lyons has a nice write-up here https://www.dsprelated.com/showarticle/814.php on different digital differentiator implementations that may be of interest – Dan Boschen Jul 15 '22 at 04:35
1

The concept of quickest detection might be of use to you. The general idea is to form a test statistic that accumulates over time, and to declare changes when the test statistic exceeds some threshold. This is a good in-depth analysis of the method.

zcsttn
  • 111
  • 2