3

I want to algorithmically process energy meter data. The energy meter measures a heat or power producer or a heat or power consumer (but not both, so the measured energy will always have a positive sign). No additional information is known about the energy system (like maximum load) neither about the type of energy meter - only data stored in a database can be accessed. Processing will be done by an algorithm looking at data for a given time interval (no live processing).

Usually, data are weakly monotonic of the form

2015-04-01 00:00 20.78 kWh

2015-04-01 00:05 30.80 kWh

2015-04-01 00:10 73.99 kWh

2015-04-01 00:20 82.30 kWh

2015-04-01 00:25 82.30 kWh

2015-04-01 00:30 83.44 kWh

...

The energy produced or consumed for a given period is simply the difference of the energy meter counts. So far, so good. However, the algorithm has to deal with the following three problems:

1. Outliers "above" have to be detected as invalid data.

2015-04-01 00:00 20.78 kWh

2015-04-01 00:05 30.80 kWh

2015-04-01 00:10 500 kWh

2015-04-01 00:20 82.30 kWh

2015-04-01 00:25 82.30 kWh

2015-04-01 00:30 83.44 kWh

....

2. Outliers "below" have to be detected as invalid data.

2015-04-01 00:00 20.78 kWh

2015-04-01 00:05 30.80 kWh

2015-04-01 00:10 20 kWh

2015-04-01 00:20 82.30 kWh

2015-04-01 00:25 82.30 kWh

2015-04-01 00:30 83.44 kWh

....

In unlikely cases, there might be several consecutive outliers above or below or a combination of both.

3. A reset of the energy meter has to be detected automatically.

2015-04-01 00:00 20.78 kWh

2015-04-01 00:05 30.80 kWh

2015-04-01 00:10 3.99 kWh

2015-04-01 00:20 12.30 kWh

2015-04-01 00:25 12.30 kWh

2015-04-01 00:30 13.44 kWh

...

After a reset, counting starts a again from another level (a reset is simply a level shift). The level the counting starts from after the reset is often zero, but can also be any other positive number. A reset can occur at an arbitrary point in time (usually not too often).

To my eyes, problems 1. - 3. seem ubiquitous in measurement engineering and must have been already addressed. Nevertheless, I couldn't find any literature on this topic. Does anybody know about existing solutions to this problem? All help will be highly appreciated.

410 gone
  • 3,751
  • 17
  • 34
Daniel
  • 33
  • 3
  • without performing a root cause analysis it figure out the root of the problem. This issue becomes cost/benefit. If the issue is frequent and causing sufficient headaches then there will be more pressure to address the problem. If the issue is one off or occasional then outliers will be discarded. Base on the data presented it looks like a defective meter. – Mahendra Gunawardena Aug 05 '15 at 11:05
  • An essential part of this question is how you intend to do this processing. Do you want to push the raw data into a computer which has a piece of software (either "home-made" or, for example, Excel) to perform the data analysis or do you want the energy meter itself to perform this analysis? – Wasabi Aug 05 '15 at 11:32
  • Along the same lines as Wasabi's comment; do you want to post-process the data or does the data processing need to be done live? Acausal processing (which can't be done live) will be much easier. – Chris Mueller Aug 05 '15 at 11:34
  • So your question is about processing the data and not about getting the data right? So is this a computer programming question? You already know the logic that you want to use, you now need to put it into some sort program. – hazzey Aug 05 '15 at 12:04
  • Generally, you shouldn't discard data that's outside your expected range without explanation.

    If you decide that you can go ahead, all you need is a simple program to detect and remove those data points.

     – DLS3141 Aug 05 '15 at 14:04
  • If this is a software design issue, problems 1 and 2 are somewhat trivial. #3 however is a tough nut to crack since the reset value isn't necessarily 0. This means that you can't simply discard a value if it is lower than the previous value, since perhaps a reset occurred. And even going "if value B is lower than A (the previous one) but is greater than C, then a reset occurred" is faulty, since I assume it's possible that there were two errors at B and C and there was no reset. – Wasabi Aug 05 '15 at 22:16
  • @ChrisMueller: The data will be post-processed by a MATLAB-algorithm, running once a day and looking at the data of the last 24h. Data will be in 5-minutes steps. Thus, causal processing is possible. To avoid that outliers at the border of a 24h-interval remain undetected, it's possibly best to do the processing with overlapping intervals. – Daniel Aug 06 '15 at 06:39
  • @Wasabi: You're right, part of the problem is that outliers (which is invalid data) and a reset (which is valid data) can break (weak) monotonicity in the same way and there can be in principle multiple consecutive outliers. However, a reset as in #3 seems to cause a 'structural break', which should be detectable by the fact that (weak) monotonicity is given before and after this 'structural break'. – Daniel Aug 06 '15 at 07:01
  • The main problem, though, is that #3 makes #2 impossible to diagnose. How do you know that the example given in #2 wasn't actually a reset? Imagine a day with no outliers and one reset. Only, right after the reset there is a huge increase in output. If A is before the reset, B is after the reset and C is after B, it is possible that C > A > B, which is exactly what you have in outlier #2, only it's supposed to be legal. The only way to differentiate is to define some boundary between #2 and #3 (maximum consumption between steps, for example, but I'm not sure if that is sufficient). – Wasabi Aug 06 '15 at 12:04
  • @Wasabi: That's a possible case indeed. I think the problem could be tackled by looking a more data after C (I will use time indexes from here on). If we have X_{t+2} > X_{t} > X_{t+1} and also X_{t+h} > X_{t+h-1} > ... > X_{t+3} > X_{t+2} for some some "big" number "h", I would consider X_{t+1} as valid, otherwise not. – Daniel Aug 06 '15 at 16:22
  • @MahendraGunawardena: Updated the case, thx! – Daniel Aug 06 '15 at 16:25
  • But @Daniel, there is no way to know if the case given in #2 is actually an outlier or if at 00:06 there was a reset (which explains why the value at 00:10 is lower than at 00:05) and then between 00:11 - 00:20 (using the given time-stamps, even though the steps should be of 5 minutes) there was a huge spike in energy consumption that took the value back up to 82.30, after which it is weakly monotonic. You can look at the entire range after 00:10 and still not know if it was an outlier or a reset. So a boundary condition (such as your "h") must be defined before anything can be done. – Wasabi Aug 06 '15 at 17:09
  • @Wasabi: I got you're point finally, thx. The same problem occurs if we have X_{t+3} > X_{t} > X_{t+2} > X_{t+1}, where we don't know if X_{t+2} and X_{t+1} are outliers or a meter reset occured between t and t+1. There is simply no way to distinguish these cases by (weak) monotonicity alone. We could simply discard X_{t+1} and X_{t+2} in the hope that we don't loose too much relevant data. – Daniel Aug 07 '15 at 06:56
  • If resets and spikes aren't strongly linked, @EnergyNumbers' answer can deal with such issues (with a margin of error, of course). So this can be done, but it requires a non-trivial statistical approach and far more information than weak monotonicity. – Wasabi Aug 07 '15 at 10:35

2 Answers2

2

There are two ways to do it.

The old way

The traditional way is to develop a set of somewhat arbitrary rules based on the errors you manually classify. You filter out non-monotonicity (easy), identify resets (easy), and try to spot other bad values (trickier). That gives you a set of values to mark as missing, and then you analyse the rest of the data. This method is not well grounded in theory, but you will have the (somewhat unsatisfactory) defence that: "it's how lots of other people do it".

Best practice

The best-practice way to do it is to write down the probability of everything relevant, and then apply Bayes' Theorem to work out what the most likely real time series was, given your recorded observations.

You start with a prior distribution for the rate of use of energy, based on preceding work.

And then create probability distributions for the ways that errors can happen: a meter reset, a dropped decimal point in recording; a dropped digit; a completely junk reading. Add in a distribution for the measurement error of the meter itself: they've usually got either a datasheet or an accredited standard which has an error range defined.

The statistics should account for things like a real usage spike and a reset coinciding. You might need to specify a joint distribution if they are linked: for example, a power cut could conceivably result in a meter reset and a power surge, as things like heaters, fridges and freezers would all come back on at full power when power is restored.

And then you calculate a posterior distribution for actual energy use, which is the thing you're interested in.

Pros and cons

The second method has the advantage of being rooted in rock-solid theory. It is, however, quite a lot of work to set up the distributions; and in pretty much every real-world case, there isn't an analytic solution, so you have to find for a numerical solution (e.g. using markov-chain monte-carlo). Software packages such as Andrew Gelman's STAN will do that part of the work for you.

Before you start, chart

Either way, start by charting your raw data. The eye will pick up informative patterns.

Community
  • 1
410 gone
  • 3,751
  • 17
  • 34
  • Could you elaborate on how to identify resets and distinguish them from "below" outliers followed by surges in consumption such that the second measurement after the reset is greater than the measurement before the reset? How can you know in the #2 case given in the OP that the outlier isn't actually a reset followed by a consuption spike between 00:11-00:20? Would this have to be done via statistics as well (assuming consumption spikes and resets are infrequent enough that both happening almost simultaneously is rare enough as to be disconsidered)? – Wasabi Aug 06 '15 at 19:35
  • 1
    @Wasabi edited to try to cover this – 410 gone Aug 06 '15 at 19:59
  • I think, I'll have to do it "the old way", as the "best practice approach" seems simply impossible to apply properly, and stick to filtering out non-monotonicity and identify resets using some (arbitrary) boundary conditions as @Wasabi suggested. Furthermore, I think that other bad values which do not violate non-monotonicity are not a too serious problem, as they don't change the total energy consumed "in the long run", but lead only to a wrong distribution of the energy consumed within a given interval (provided that the values at the start and end of the interval are correct) – Daniel Aug 10 '15 at 11:47
0

Discarding outliers is easy, you simply discard (or ignore) them. The hard part is deciding what is a outlier and what is valid data. This comes down to defining what is possible and likely.

One obvious check in your case is against the maximum power the system can consume or produce. If the delta between two readings exceeds that, then something is wrong somewhere. For example, if the system being measured can't consume more than 200 kW, then the meter can't increase by more than 17 kWh in a 5 minute period.

Note that the maximum power production can be different from the maximum power consumption. Maybe your system can consume up to 200 kW, but can't ever produce more than 50 kW. That means in 5 minutes the possible valid range is +16.7 kWh to -4.2 kWh.

You may know other things about your system, which you can use to detect invalid readings. For example, even though its power consumption is bounded by -50 to +200 kW, maybe it can't change faster than 10 kW per 5 minutes. Or, if the power is being produce by solar cells, then getting power production at night must be wrong. Surely there are things you know your system can't do.

Again, it all comes down to defining what valid is. For example, you say the third reading in case 2 is wrong? How do you know that? If it's possible for the system to generate 10.8 kWh during a 5 minute interval, then consume 62.3 kWh during the next 5 minutes, then case 2 may be all correct readings after all. You haven't said anything about your system to tell this is not the case.

Olin Lathrop
  • 11,401
  • 1
  • 22
  • 36
  • My post was misleading, I meant to say that the energy meter measures EITHER the energy produced or consumed (but not both). You're idea with using the maximum rate of change is great, but unfortunately the necessary additonal information is not (always) given - the algorithm should work for all kind of energy sources and sinks. The only criterion is (weak) monotonicity. I thought to use some kind of rank-order correlation to detect the 'structural break' of the meter reset and treat outliers separately. What separates valid and invalid data seems to be essentially the frequency of occurence. – Daniel Aug 06 '15 at 07:24