Dynamic exporters for smartpireadout

[TriCX]

The definition of a struct and returning a map of it goes inline with my idea.
Nevertheless, I would define it more generic.
Let's assume ade7878 is just one of a bunch of inputs,
or we have some readings with different measurement intervals.
For example accumulated watt hours do not need to be recorded every 5 secs.

My proposal is to define a totally generic "measurand".
First shot:
type Measurand struct {
   value float64
   unit string // or are there real units somewhere in go?
   timestamp Time
   name String // used to identify the value and meaning
   properties Type to be defined (maybe it is useful to provide information about time series or accumulated data, don't know)
   sensorID String or int // to identify the sensor (e.g. phase number, or temperature sensor living room, etc.)
}

ade7878.go might return a slice of measureands to handle a dynamic number.
readout.go could handle multiple inputs (such as ade7878) via this generic interface.
readout.go interates through the received slices and identifies the read values via their name and sensorID.

My Raspberry Pi does not only host the SmartPi unit. It is also connected to an Arduino.
The Arduino counts S0 pulses from a watt meter, see
https://github.com/Trickx/ArduinoPulseCounting
(In need to update the readme.)

I think my setup is typical for other customers.
Therefore, why not handle multiple sensors via this go project?
I'm not sure whether enerserve is open for additional sensor data,
but I will compare my energy consumption measured via pulse counting with data from ade7878.

[SuperQ]

The definition of a struct and returning a map of it goes inline with my idea.
Nevertheless, I would define it more generic.
Let's assume ade7878 is just one of a bunch of inputs,
or we have some readings with different measurement intervals.
For example accumulated watt hours do not need to be recorded every 5 secs.

My proposal is to define a totally generic "measurand".
First shot:
type Measurand struct {
   value float64
   unit string // or are there real units somewhere in go?
   timestamp Time
   name String // used to identify the value and meaning
   properties Type to be defined (maybe it is useful to provide information about time series or accumulated data, don't know)
   sensorID String or int // to identify the sensor (e.g. phase number, or temperature sensor living room, etc.)
}

ade7878.go might return a slice of measureands to handle a dynamic number.
readout.go could handle multiple inputs (such as ade7878) via this generic interface.
readout.go interates through the received slices and identifies the read values via their name and sensorID.

My Raspberry Pi does not only host the SmartPi unit. It is also connected to an Arduino.
The Arduino counts S0 pulses from a watt meter, see
https://github.com/Trickx/ArduinoPulseCounting
(In need to update the readme.)

I think my setup is typical for other customers.
Therefore, why not handle multiple sensors via this go project?
I'm not sure whether enerserve is open for additional sensor data,
but I will compare my energy consumption measured via pulse counting with data from ade7878.

This is somewhat similar to how the Prometheus library works.  We already have most of the fields for your struct as part of our library[0].  I've already started to decouple things from the ReadoutValues() function.  In theory we could add additional functionality without it.

The only big issue with the Prometheus library is that we don't offer Get() functions for other parts of the program to pull the status of the registered metrics.  If we had this, we could simply send all of the readings to that library.

There are some other pre-built metrics libraries out there that do pretty much what you want.  I'll talk to of my more experienced Go developer friends to see what would make this easier.

[0]: https://github.com/prometheus/client_golang

[SuperQ]

I almost forgot to comment about "accumulated watt hours do not need to be recorded every 5 secs."

Actually, I would like to see this updated more frequently.  In order to correctly calculate the average watts used, we need to take the first derivative of the watt-hours.  In Prometheus, we provide a couple functions to handle this well, but we need several samples in order to calculate this.  For example rate(watt_hours_total[15s]) would take 3 samples, which is the minimum necessary to handle a counter reset/rollover.

Using cumulative counters for calculating gauge values eliminates the whole sampling problem, lost samples reduce fidelity, but not accuracy.

[TriCX]

Hi,

accumulation needs to be done inside the ade7878.
The device is capable to do so, but calibration support is missing so far.
This is why I would be fine with an interval of a minute or so for accumulated data.

The issue I see with self-accumulation is that, accuracy is not reachable.
If we would increase measurement frequency from currently 0.2 Hz to 1Hz or even 10Hz,
accuracy would not increase much.
Transient effects when turning on/off concumers are much faster than this and we loose this information.
In other words averaging over 5secs results in similar poor accuracy than averaging over 1sec ... 0.1sec.

[SuperQ]

Hi,

accumulation needs to be done inside the ade7878.
The device is capable to do so, but calibration support is missing so far.
This is why I would be fine with an interval of a minute or so for accumulated data.

The issue I see with self-accumulation is that, accuracy is not reachable.
If we would increase measurement frequency from currently 0.2 Hz to 1Hz or even 10Hz,
accuracy would not increase much.
Transient effects when turning on/off concumers are much faster than this and we loose this information.
In other words averaging over 5secs results in similar poor accuracy than averaging over 1sec ... 0.1sec.

Yes, I know accumulation is done inside the ade7878.  You're missing the point of why I want faster polling of those accumulation values.

I want to use accumulation counters of watt-hours and amp-hours in order to use first derivative mathematics to produce more accurate average watt/current gauges.

You have to remember that I am using a much more high performance time-series database than sqlite.  With Prometheus 5 second samples for the smart pi are under 400kB/day.  2GB can store 10 years of 5 second samples with no down-sampling required.

So, really, I do need very high sampling rates of the ade7878's accumulation counters whenever the calibration issues are solved.