Analytics

Analytics: What is it?

Analytics is trying to make sense by recognising patterns in vast quantities of information. There are many similarities with search engines. In search engines the data is in the form of words organized in to pages. Analytics is a natural progression of search technology into this space.

The underlying software here on the Raspberry Pi cluster actually began life on the UpBoard Intel quad node. The first incarnation was ElasticSearch OSS edition. When the Intel Quad died one day (with no further support by AAEON Technology Inc. an ASUS associated company <- dig) there was a rush to migrate to Raspberry Pi commputers. Fortunately cloud providers at the time were already deploying ARM architecture CPUs. which were also used in the Raspberry Pi.

As it turns out the demise of the Intel board created the start of an ecosystem which to this day maintains compatilibity with Amazon Web Services. This in itself is quite valuable as a training aid. Budding cloud engineers can gain a detailed understanding of how large scale cloud providers work. If you do build your own training cluster, however, you'll become a geek! You've been warned.

Some the the early notes follow:

Cluster migration off Intel to ARM64
Elasticsearch-OSS Version 7.9.x

The current round upgrades are mostly about moving off Intel nodes over to ARM64 nodes in the Portainerized Swarm. Some search back-ends have already moved to ARM using Elastic's official images. The main analytics cluster is still running on Intel however. Kibana and Logstash are now running on ARM via unofficial images (you're looking at the images right now). This limits the upgrades currently to the 7.9 versions but as Elastic starts providing official ARM64 images these will be retired.

The V2 Power Analyzer is also in trouble with the new Renault ZOE electric whereby the current measuring probe has maxed out. The new 30 amp current clamp is on my desk waiting to be fitted.

This page is a collection of images produced by the analytics cluster and other devices (in near realtime). The data is essentially energy production (solar) and consumption (EV and house) as well as $$ of course.

Import, Export, Import Replacement, Nett Grid, EV-Charger

Power Flow

Rated power flow $$

Import ($0.27), Export ($0.20), Grid ($sum), Import Replacement ($0.27[House] or $0.70[Car fuel])

The hardware:

The docker containers:

  • Elasticsearch-oss
  • Logstash-oss
  • Kibana-oss
  • Filebeat-oss
  • vsftp
  • chrome-headless
  • httpd
  • ElasticHQ

Waveform analysis peak-to-peak, rms, frequency
Real time waveform analysis

Generation with peak/min volts(rms)/frequency(Hz)
Power Generation

The following graphs are produced by the ev-charger power analyzer v2