OpenSearch VS ElasticSearch - Can you switch ?

Why are there two "open source" search engines which are essentially the same? Well, the answer seems to lie in the distribution of the proceeds of the intellectual property. Elasticsearch and Amazon Web Services had a very public falling out over the last couple of years. Essentially Elastic's principal Shay Bannan explained the situation, from Elastic's perspective, in a blog post titled:

Amazon: NOT OK - why we had to change Elastic licensing

Elastic's main objections are:

You may not use the licensed work to provide an "Elasticsearch/Kibana as a Service" offering.
You may not hack the software to enable our paid features without a subscription.
You may not remove, replace or hide the Elastic branding and trademarks from the product. (e.g. do not replace logos, etc).

AWS's response has essentially been:

Keeping Open Source Open – Open Distro for Elasticsearch

However, the initial open source attempt was thwarted somewhat by a trademark infringement lawsuit filed against Amazon in 2017. The ultimate response has been the "forking" of yet another project named OpenSearch.

There's something of a running commentary on Forbes here: https://www.forbes.com/sites/davidjeans/2021/03/01/elastic-war-on-amazon-web-services/

What are the main changes between the two open source distributions? Examining the GitHub repository shows a lot of the changes are in the form below:

essentially any mention of "elastic" is replaced with "opensearch". Obviously that overly trivialises the changes, but it does reflect a willingness to avoid any trademark disputes and goes a long way to explaining why OpenDistro went the extra step of moving to OpenSearch. On the other non-trivial hand OpenSearch has a lot of features which replace xpak paid license functions.

At this point the somewhat philosphical overview terminates and the practicalities of the situation we are presented with are evaluated. The only question considered from this point is: can you actually jump horses in mid race ?

Can you switch ?

Yes! However, the process is not for faint hearted at this time. Below is an exhaustive list of what it takes to get the ElasticBeats-OSS (Open Source Software edition) installed and running. A lot of compatibility type stuff just plain doesn't work out of the box, but it is still do-able and the instructions here show you exactly how.

Can you just plonk OpenSearch over the top of your existing indices? You'll have to read on to get the answer (spoiler alert - HELL YEAH!).

This overview ECS dashboard image is produced from data re-indexed from the running ElasticSearch-OSS 7.9.3 cluster into the OpenSearch 1.0.0 cluster. There are lots of migration paths available to users. This one was toe in the water verification that things can be made to work as claimed by the OpenSearch team. A lot of stuff is un-documented at this point and bits are still missing. However it's quite impressive as it stands right now and it can only get better.

And you get some extra cool stuff that would otherwise take a lot of time and money to reproduce. Things like Alerting are a big deal for operations people - devops and secops essentially see these functions as must have.

Those users with Transport Clients are out of luck. Transport is no longer supported at all.

There are stategies open which can allow transport level clients to interoperate, essentially by making OpenSearch look and sound like ElasticSearch, but the OS FAQ doesn't recommend this as a long term pathway.

Installation

Both systems are dockerized installs here. Elasticsearch-oss edition has been running for a couple of years now. It's installed on an AAEON UP Intel Z8350 quad core RPi form factor board. OpenSearch was installed using the official containers on a Ryzen 3700X with 64GB ram. Both versions also run on ARM64, and in fact Kibana was moved off the AAEON UP to an RPi4-8GB to free up memory (it actually seemed to run faster too).

Kubernetes and Swarm environments will have trouble locking memory as recommended by OpenSearch. While the locking stuff works on bare metal and Docker Compose, the cluster orchestrator environments haven't sorted that out yet, and there are long standing tickets open too. The approach here has been to disable swap completely on candidate search nodes.

Metricbeat template installation

Adding the metricbeat templates resulted in failure.

The first issue is that communications with OpenSearch's cluster is encrypted by default (which is philosphically a good thing). The certificate is a self-signed example however, so verification is turned off for now. If left on metricbeat declares the cluster is a fake.

metricbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['https://localhost:9200'] \
  -E output.elasticsearch.ssl.verification_mode=none \
  -E output.elasticsearch.username=opensearch_user \
  -E output.elasticsearch.password=opensearch_password \
  -E setup.kibana.host=localhost:5601

However, metricbeat failed to add the metricbeat template (i.e. the Elastic Common Schema or ECS template). Closer inspection reveals that the cluster had reported its version as 1.0.0 which causes metricbeat to have a heart attack. Furthermore when metricbeat enquired about the version number of Kibana/OpenSearch-dashboards it ran in to a similar issue.

2021-08-08T15:20:24.580+1000    INFO    kibana/client.go:119    Kibana url: http://localhost.localdomain:5601
2021-08-08T15:20:24.814+1000    INFO    kibana/client.go:119    Kibana url: http://localhost.localdomain:5601
2021-08-08T15:20:24.822+1000    ERROR   instance/beat.go:951    Exiting: Kibana API is not available in Kibana version 1.0.0
Exiting: Kibana API is not available in Kibana version 1.0.0

Generic compatibility issue worthy examination in detail

There are essentially two ways to attempt to fix this.

The easy way (which only partially works) is to attempt to use the opensearch.yml line :

compatibility.override_main_response_version: true

a temporary measure which makes OpenSearch spoof the version number. The goal is to make metricbeat (and hence any other such "legacy" clients) think the cluster is compatible and therefore it sends the metricbeat template over to the cluster. However such a setting doesn't appear to be available for OpenSearch-dashboards.

The intended purpose of version spoofing is to allow for rolling cluster upgrade from elasticsearch-oss nodes to opensearch nodes. Once completed it's recommended to remove the setting. Rolling node upgrades make use of the transport interface which is now pretty much reserved exclusively for intracluster communications - and as such no longer needs to be supported for external client applications.

before from the setting:

curl -k -u admin:admin https://localhost:9200/
{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "ic-tUQoGTnyXVU_D8FmG9Q",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.0.0",
    "build_type" : "tar",
    "build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date" : "2021-07-02T23:22:21.383695Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

to this where OpenSearch spoofs its version numbers to an ElasticSearch compatible 7.10.2

curl -k -u admin:admin https://localhost:9200/
{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "ic-tUQoGTnyXVU_D8FmG9Q",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "tar",
    "build_hash" : "34550c5b17124ddc59458ef774f6b43a086522e3",
    "build_date" : "2021-07-02T23:22:21.383695Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Unfortunately there is no corresponding setting for opensearch-dashboards. This issue has been raised (which I contributed to while writing this piece) here https://github.com/opensearch-project/OpenSearch-Dashboards/issues/656

Essentially (from examining the source code) Metricbeat and friends make this query to Kibana/Opensearch-Dashboards server:

http://localhost:5601/api/status

resulting in lots and lots of stuff (among which is "number":"1.0.0") :

{
  "name": "42f975ee9b4f",
  "uuid": "9bb7854c-ae61-4509-a4f6-e23f8be61e38",
  "version": {
    "number": "1.0.0",
    "build_hash": "b15ad258e6960003e609a70889cda5fb39f90947",
    "build_number": 36473,
    "build_snapshot": false
  },
  "status": {
    "overall": {
      "since": "2021-08-09T06:43:30.711Z",
      "state": "green",
      "title": "Green",
      "nickname": "Looking good",
      "icon": "success",
      "uiColor": "secondary"
    },
    "statuses": [
      {
        "id": "core:opensearch@1.0.0",
        "message": "OpenSearch is available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "core:savedObjects@1.0.0",
        "message": "SavedObjects service has completed migrations and is available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:advancedSettings@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:bfetch@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:charts@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:discover@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:indexPatternManagement@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:devTools@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:embeddable@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:management@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:expressions@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:navigation@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:legacyExport@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:inspector@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchDashboardsUtils@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchUiShared@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchDashboardsUsageCollection@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchDashboardsReact@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchDashboardsOverview@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:share@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:savedObjects@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:savedObjectsManagement@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:urlForwarding@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:uiActions@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visDefaultEditor@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visualizations@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visualize@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:ganttChartDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:queryWorkbenchDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:notebooksDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:traceAnalyticsDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:reportsDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:apmOss@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:console@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:dashboard@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:data@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:home@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:inputControlVis@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:mapsLegacy@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:opensearchDashboardsLegacy@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:regionMap@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:tileMap@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:timeline@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:usageCollection@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeMetric@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeMarkdown@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeVega@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeTimeseries@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeTagcloud@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeTable@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeTimeline@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:visTypeVislib@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:alertingDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:anomalyDetectionDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:indexManagementDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      },
      {
        "id": "plugin:securityDashboards@1.0.0",
        "message": "All dependencies are available",
        "since": "2021-08-09T06:43:30.711Z",
        "state": "green",
        "icon": "success",
        "uiColor": "secondary"
      }
    ]
  },
  "metrics": {
    "last_updated": "2021-08-09T06:43:29.426Z",
    "collection_interval_in_millis": 5000,
    "os": {
      "platform": "linux",
      "platformRelease": "linux-5.11.0-25-generic",
      "load": {
        "1m": 2.40283203125,
        "5m": 2.50439453125,
        "15m": 2.4560546875
      },
      "memory": {
        "total_in_bytes": 67456917504,
        "free_in_bytes": 43529277440,
        "used_in_bytes": 23927640064
      },
      "uptime_in_millis": 28344000,
      "distro": "Amazon Linux",
      "distroRelease": "Amazon Linux-2 (Karoo)",
      "cpuacct": {
        "control_group": "/",
        "usage_nanos": 53148102530
      },
      "cpu": {
        "control_group": "/",
        "cfs_period_micros": 100000,
        "cfs_quota_micros": 400000,
        "stat": {
          "number_of_elapsed_periods": 11903,
          "number_of_times_throttled": 0,
          "time_throttled_nanos": 0
        }
      }
    },
    "process": {
      "memory": {
        "heap": {
          "total_in_bytes": 118300672,
          "used_in_bytes": 103872216,
          "size_limit": 1526909922
        },
        "resident_set_size_in_bytes": 172150784
      },
      "pid": 1,
      "event_loop_delay": 0.26618899777531624,
      "uptime_in_millis": 12071045
    },
    "response_times": {
      "avg_in_millis": 0,
      "max_in_millis": 0
    },
    "concurrent_connections": 1,
    "requests": {
      "disconnects": 0,
      "total": 0,
      "statusCodes": {},
      "status_codes": {}
    }
  }
}

The practical upshot of the inability to spoof the Opensearch-Dashboards API version number means that beats cannot use this method to install its dashboards. From reading the beats source code it seems that some hard coded decisions were made to the effect that anything prior to version 6 does not even have the API or at least it was considered unusable.

Metricbeat Dashboard only installation

Running a similar command to the above to load the metricbeat dashboards results in a similar failure.

metricbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['https://localhost:9200'] \
  -E output.elasticsearch.ssl.verification_mode=none \
  -E output.elasticsearch.username=opensearch_user \
  -E output.elasticsearch.password=opensearch_password \
  -E setup.kibana.host=localhost:5601 \
  --dashboards

the above commands produces lots of output ending in:

2021-08-08T15:20:24.580+1000    INFO    kibana/client.go:119    Kibana url: http://localhost.localdomain:5601
2021-08-08T15:20:24.814+1000    INFO    kibana/client.go:119    Kibana url: http://localhost.localdomain:5601
2021-08-08T15:20:24.822+1000    ERROR   instance/beat.go:951    Exiting: Kibana API is not available in Kibana version 1.0.0
Exiting: Kibana API is not available in Kibana version 1.0.0

Alternative required for loading dependencies into OpenSearch

As it stands it's clearly not possible to directly load the Beats dependencies into OpenSearch. Scouring the web produces an array of posts where people propose to run up a temporary Elastic stack. The aim is to load and then save the necessary components. This was tried, with the aid of Portainer, however it produced mixed results.

The main issue with this approach stems from the exported objects containing Elastic random name generator references. While these are entirely consistent within the exported objects they seemed to contain references to the cluster (in this case elasticsearch-793-temp) here and there.

Some reverse-engineering needed

The main issue blocking ultimate success are the version checks carried out prior to any attempt to load stuff. Essentially, these checks have to be bypassed.

The final result developed here was arrived at by a mixture of examining the publically available source code for Beats and watching log files from the Elastic stack. Beats is written in Go. From Wikipedia (loved by academics world-wide) "Go is a statically typed, compiled programming language designed at Google by Robert Griesemer, Rob Pike, and Ken Thompson.". Fortunately Go is fairly 'C' like and this author is old enough to have produced a lot of code in Kernighan and Ritchie's language.

What became clear was precisely how Beats inserts its dependencies into Kibana and ElasticSearch. Armed with that knowledge it's fairly straight forward to emulate the steps directly using, the Swiss army knife like, curl. It then becomes feasible to load the depencies directly in to OpenSearch 1.0.0 albeit manually.

A fully manual load has to be carried out in a specific order with the component files being produced by export commands to beats. There is no doubt some enterprising soul shall script these steps to once again make everything seemless. Please ping me @electricbrain2 on twitter when that happens.

Manually load the elastic template in to OpenSearch

The first step is to produce the artefacts needed by the upload process.

metricbeat export template > metricbeat.template.json

then hacking out the incompatible "lifecycle" clause using vi:

"lifecycle": {
   "name": "metricbeat",
   "rollover_alias": "metricbeat-7.9.3"
},

Then load it into the OpenSearch cluster:

curl \
    -k \
    -u admin:admin \
    -XPUT \
    -H 'Content-Type: application/json' \
    https://localhost:9200/_template/metricbeat-7.9.3 \
    -d@metricbeat.template.json

Manually load an index-pattern in to Opensearch-Dashboards

Again the next step is to produce the artefacts.

metricbeat export index-pattern --es.version 7.9.3 > metricbeat.index-pattern.json

This time the OpenSearchDashboards (OSD)/Kibana (KBN) API is involked. Only the index-pattern is imported in this step - note how the dashboard import function is used to carry out this function.

curl \
  -XPOST \
  -u admin:admin \
  -H 'osd-xsrf: true' \
  -H 'Content-Type: application/json' \
  'http://localhost:5601/api/opensearch-dashboards/dashboards/import?force=true' \
  -d@metricbeat.index-pattern.json

Manually load a dashboard in to Opensearch-Dashboards

The final step is load each dashboard, one at a time, until all those relevant to your project are done. In this part of the manual load process only the dashboard is imported with the index-pattern specifically excluded.

curl \
  -XPOST \
  -u admin:admin \
  -H 'osd-xsrf: true' \
  -H 'Content-Type: application/json' \
  'http://localhost:5601/api/opensearch-dashboards/dashboards/import?exclude=index-pattern&force=true' \
  -d@/usr/share/metricbeat/kibana/7/dashboard/Metricbeat-host-overview.json

Can you just plonk OpenSearch over your existing indices!

Hell yeah! The image below is the result of directly loading OpenSearch over the top of ElectricBrain's solar analytics system's ElasticSearch indices.

There are a couple of extra steps to get the new security stuff going but when combined with the steps above on the ingestion side we arrive at a complete viable solution with all the extra features. And completely redistributable and SaaS-able too.

The first thing that pops up in the documentation is "You'll need to convert the settings in config.yml to OpenSearch". Of course there's very little in the way of documentation regarding such a process, which is sort of to be expected given the early release state I guess.

Then, once you've figured out those parameters and their names, then you're old Java Virtual Machine options don't work anymore. The demo set from official image were copied over the top of the original ElasticSearch configs here to get things going.

Opensearch then lit up! There is one final step required however. Since OpenSearch has security built in, and ElasticSearch-OSS doesn't (because it's an X-Pak paid option), none of the needed indices are present in your existing data.

[2021-08-12T02:34:21,855][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:21,857][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:24,354][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:24,355][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:24,356][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:24,356][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:26,854][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:26,855][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:26,856][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:26,856][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:27,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:29,355][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:29,356][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:29,357][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:29,357][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:31,858][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:31,859][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:31,859][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:31,860][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:34,360][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:34,360][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:34,361][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:34,362][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:35,259][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:36,863][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:36,863][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:36,864][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:36,864][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:39,363][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:39,364][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:39,364][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:39,365][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:41,866][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:41,866][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:41,867][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:41,867][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:43,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:44,368][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:44,369][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:44,369][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:44,370][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:46,867][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:46,868][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:46,869][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:46,869][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:49,368][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:49,369][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:49,369][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:49,370][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,260][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:51,870][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:51,871][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:51,872][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:51,872][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:54,371][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:54,371][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:54,372][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:54,372][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:56,873][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:56,874][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:56,874][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:56,875][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for internalusers while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for actiongroups while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for config while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for roles while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for rolesmapping while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,261][WARN ][o.o.s.c.ConfigurationLoaderSecurity7] [opensearch-node1] No data for tenants while retrieving configuration for [INTERNALUSERS,
 ACTIONGROUPS,
 CONFIG,
 ROLES,
 ROLESMAPPING,
 TENANTS,
 NODESDN,
 WHITELIST,
 AUDIT]  (index=.opendistro_security and type=null),
[2021-08-12T02:34:59,373][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:59,373][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:59,374][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:34:59,374][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:01,875][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:01,876][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:01,876][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:01,877][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:04,376][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:04,377][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:04,378][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin),
[2021-08-12T02:35:04,379][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-node1] Not yet initialized (you may need to run securityadmin)

Setting up OpenSearch security on your existing system

Setting up the new security indices is accomplished with the aid of the securityadmin.sh program. This is well documented on the OpenSearch site and even has a working example, which was simply copy/pasted into the container's console via Portainer. It needed a couple of extra steps to get the JAVA_HOME and friends environment variables setup, since that's not done in the official container. Here's what worked:

[opensearch@941f8cb2a77a ~]$ export JAVA_HOME=/usr/share/opensearch/jdk
[opensearch@941f8cb2a77a ~]$ export PATH=$JAVA_HOME/bin:$PATH
[opensearch@941f8cb2a77a ~]$ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh 
[opensearch@941f8cb2a77a ~]$ cd /usr/share/opensearch/plugins/opensearch-security/tools/
[opensearch@941f8cb2a77a tools]$ ./securityadmin.sh -cd ../securityconfig/ -icl -nhnv    -cacert ../../../config/root-ca.pem    -cert ../../../config/kirk.pem    -key ../../../config/kirk-key.pem
Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=kirk,OU=client,O=client,L=test,C=de
OpenSearch Version: 1.0.0
OpenSearch Security Version: 1.0.0.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: ebrain-opensearch-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists,
 so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig
Will update '_doc/config' with ../securityconfig/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with ../securityconfig/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with ../securityconfig/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with ../securityconfig/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with ../securityconfig/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with ../securityconfig/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with ../securityconfig/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with ../securityconfig/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with ../securityconfig/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Done with success
[opensearch@941f8cb2a77a tools]$

A further point raised by Jacob (on 17/Nov/2021) is that metricbeat (beyond 7.10.2) specifically version 7.12.1 has a setting in metricbeat.yml named:
setup.dashboards.enabled: false
which should be set to false.
(Reference: https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-dashboards.html)

This setting ensures metricbeat won't attempt to do a version check on Dashboards during startup. The check determines if the necessary artefacts have been loaded in to OpenSearch. In Jacob's case this failed without this setting, preventing startup because Dashboards will return a version of 1.0.0 triggering the error and aborting startup.