Dynamic exporters for smartpireadout

[TriCX]

The implementation of the software is progressing on github.
https://github.com/nDenerserve/SmartPi

I by myself don't use the provided webserver.
Instead, I'm usinf smartpi as metering device and export the data towards emoncms.
https://openenergymonitor.org/

I imagine the usage of the smartpi will be manifold.
Currently export towards

• MQTT
  
• CSV
  
• SQLite3
  
• EMONCMS
  
• Prometheus
  

are available or planned.
I expect more in the future.

Therefore I propose a generic interface to load and instanciate exporters dynamically.
The interface towards the hardware is implemented in ade7878.go.
It implements a function "ReadoutValues", which currently returns a list of 25 float values for further processing.
readout.go itself makes use of these values to export them to various receivers.

I propose to enhance the config file with a list of exporters and their configurations.
Depending on configured exporters, corresponding classes are instanciated and used via a common interface.
The interface should define the numbers of values, their meaning (e.g. name) and of course their values.
Each exporter could be implemented in a separated file, which keeps the code clean and more compact.

This way each user can easily decide which exporter to use, without wasting MIPS on embedded devices for unnecessary processing of the read values.

What do you think about?
   

Regards,
TriCX

[SuperQ]

Now that we've refactored the readout.go file a bit, the main loop is much cleaner.  I don't see a lot of utility to come up another interface, and then a whole bunch of different binaries to do their own interfaces.

For example, adding Prometheus support is not terribly complicated with the current single reader binary.

https://github.com/nDenerserve/SmartPi/pull/32

[TriCX]

Seperation into different binaries is not my intention.
The idea behind is to define a clear structure and keep readout.go to th erequired minimum.
With your pull request, you add multiple lines of code to readout.go.
This makes it hard to maintain common code if each export functionality is implemented there.
This is why I propose the definition of an interface and the implementation export functions in different files.
This way you can add an exporter with an import and the function call within readoud.gp => additional lines.
We could even separate an exporter into a dedicated repository.

In the next days I can re-write the code to give an example at least for MQTT and maybe some other axisting export functions.

Best
TriCX

[jens]

The idea is good. Just how does it look with the distinction momentary values, time series. CSV values are usually time series

[TriCX]

I don't think we need to distinct between different types at this interface.
If this needs to be handled this should be done inside the exporter or the external tools itself.
I don't know much about prometheus, but for EMONCMS or MQTT the type does not matter.
For MQTT a name (aka topic) is required to publish data.
The mapping between a name to a type should be done external.
Why a visualisation of total accumulated power consumption on a gauge should be false? We can't know!
I was more thinking about a time stamp for reference on the interface.
This makes more sense.

[SuperQ]

Because of the way different systems bootstrap themselves in different ways, there will probably still need to be some glue in the readout main.

But, we can cleanup the files a bit with something like this:

https://github.com/nDenerserve/SmartPi/c...t-refactor

I can do similar patches for MQTT and other interfaces.

[TriCX]

In my opinion readout.go should be kept clean and short.
Putting the implementation of data exporters into this file blows up and creates a mess.
It should be separated into different files.
Then we need some functions to call.
I belief we can manage to define an interface, which fits it all
- at least for periodic data publication.

The interface may introduce a slice to handle an undefined number of data elements.
Each data element itself (currently voltages, currents, and so on) could be represented by a struct.
This struct contains all relevant data, such as
- value
- unit
- tmie series or accumulated flag
- accuracy
- whatever

Best,
TriCX

[SuperQ]

In my opinion readout.go should be kept clean and short.
Putting the implementation of data exporters into this file blows up and creates a mess.
It should be separated into different files.
Then we need some functions to call.
I belief we can manage to define an interface, which fits it all
- at least for periodic data publication.

The interface may introduce a slice to handle an undefined number of data elements.
Each data element itself (currently voltages, currents, and so on) could be represented by a struct.
This struct contains all relevant data, such as
- value
- unit
- tmie series or accumulated flag
- accuracy
- whatever

Best,
TriCX

Sure, but did you look at the patch I linked that does move the Prometheus changes out to a separate file?

As far as defining a struct that represents all the data, I totally agree, I've been thinking about this design a bit, and have already refactored much of the ReadoutValues() function such that it will be much easier to change the return function, or simply drop the ReadoutValues() function entirely.

I've also been thinking about changing the way the main readout.go loop works.  Rather than sample the data every 5 seconds, we can reduce that to a 1 second loop, and each exporter can select how often it wants to get updates.

[jens]

My suggestion is that we run all exporter as stand-alone programs. The current values can be taken from the shared file. Whenever the file changes, the respective program can process the data. File monitoring might be https://github.com/fsnotify/fsnotify.
So we would follow the approach: small programs for small tasks.

[SuperQ]

I've posted a proposal branch to my fork that contains a rework of the return structure of ReadoutValues(). Please let me know what you think.

This is currently incomplete as I've only updated the function, and not the use in the readout server.

https://github.com/nDenerserve/SmartPi/c..._interface