Configuration¶
The Elasticsearch DXL Python Service application requires a set of configuration files to operate.
This distribution contains a config
sub-directory that includes the configuration files that must
be populated prior to running the application.
Each of these files are documented throughout the remainder of this page.
Application configuration directory:
config/ dxlclient.config dxlelasticsearchservice.config logging.config (optional)
DXL Client Configuration File (dxlclient.config)¶
The required
dxlclient.config
file is used to configure the DXL client that will connect to the DXL fabric.The steps to populate this configuration file are the same as those documented in the OpenDXL Python SDK, see the OpenDXL Python SDK Samples Configuration page for more information.
The following is an example of a populated DXL client configuration file:
[Certs] BrokerCertChain=c:\\certificates\\brokercerts.crt CertFile=c:\\certificates\\client.crt PrivateKey=c:\\certificates\\client.key [Brokers] {5d73b77f-8c4b-4ae0-b437-febd12facfd4}={5d73b77f-8c4b-4ae0-b437-febd12facfd4};8883;mybroker.mcafee.com;192.168.1.12 {24397e4d-645f-4f2f-974f-f98c55bdddf7}={24397e4d-645f-4f2f-974f-f98c55bdddf7};8883;mybroker2.mcafee.com;192.168.1.13
Elasticsearch DXL Python Service (dxlelasticsearchservice.config)¶
The required
dxlelasticsearchservice.config
file is used to configure the application.The following is an example of a populated application configuration file:
[General] serverNames=es1 eventGroupNames=eventgroup1 apiNames=index,get,update,delete [es1] host=elasticserver1 user=elastic password=elasticpassword useSSL=yes verifyCertificate=yes verifyCertBundle=esCA.crt verifyHostName=yes clientCertificate=client1.crt clientKey=client1.key [eventgroup1] topics=/myservice/mytopic1,/myservice/mytopic2 documentIndex=opendxl-services documentType=myservice idFieldName=event_id transformScript=transform.pyGeneral
The
[General]
section is used to specify the list of Elasticsearch servers, Elasticsearch methods which should be available to invoke via DXL, and information for DXL event notifications which should be stored to Elasticsearch.
Name Required Description serviceUniqueId no An optional unique identifier used to identify the opendxl-elasticsearch service on the DXL fabric. If set, this unique identifier will be appended to the name of each request topic added to the fabric. For example, if the serviceUniqueId is set to
sample
, the request topic names would start with the following:
/opendxl-elasticsearch/service/elasticsearch-api/sample/<method>
If serviceUniqueId is not set, request topic names would not include an id segment, for example:
/opendxl-elasticsearch/service/elasticsearch-api/<method>
serverNames yes The list of Elasticsearch servers to expose to the DXL fabric, delimited by commas.
For example:
es1,es2,es3
For each name specified, a corresponding section must be defined within this configuration file that provides detailed information about the server (see "Server Section" below).
eventGroupNames no The list of event groups for which any DXL events received are indexed to Elasticsearch, delimited by commas.
For example:
eventgroup1,eventgroup2,eventgroup3
For each name specified, a corresponding section must be defined within this configuration file that provides detailed information about the event group (see "Event Group Section" below).
apiNames no The list of Elasticsearch APIs for which corresponding request topics should be exposed to the DXL fabric.
For example:
index,get,update,delete
With this example and the
serviceUniqueId
setting set tosample
, the request topics exposed to the DXL fabric would be:
/opendxl-elasticsearch/service/elasticsearch-api/sample/index
/opendxl-elasticsearch/service/elasticsearch-api/sample/get
/opendxl-elasticsearch/service/elasticsearch-api/sample/update
/opendxl-elasticsearch/service/elasticsearch-api/sample/delete
The total list of available API method names and parameters is at:
https://elasticsearch-py.readthedocs.io/en/master/api.html#elasticsearch
For each name specified, a corresponding section must be defined within this configuration file that provides detailed information about the event group.
reloadTransformScriptsOnChange no Controls whether or not changes made to event group transform scripts while the service is running can be reloaded dynamically. Setting this to
yes
can be helpful to reduce service restarts while developing a transform script.Defaults to
no
, load transform scripts only at service startup time.Server Section (1 per Elasticsearch server)
Each server specified in the
serverNames
property of the[General]
section must have a section defined which contains details about the particular Elasticsearch server.The section name must match the name of the server in the
serverNames
property (for example:[es1]
).
Name Required Description host yes The Elasticsearch server hostname or IP address. port no The Elasticsearch server HTTP API port. Defaults to 9200
.urlPrefix no The URL prefix under which the Elasticsearch server's HTTP API is hosted. For example, if this were set to
api
, requests that the OpenDXL Elasticsearch service makes to the Elasticsearch server would start with the following:
http(s)://<host>:<port>/api
Defaults to no prefix, making requests to the root path.
user no The name of the user used when making requests to the Elasticsearch server. Defaults to no user name. Required if password
is specified.password no The password associated with the user used when making requests to the Elasticsearch server. Defaults to no password. Required if user
is specified.useSSL no Whether or not to use SSL/TLS for requests made to the Elasticsearch server. If set to yes
, SSL/TLS is used. Defaults tono
.verifyCertificate no Whether to verify that Elasticsearch server's certificate was signed by a valid certificate authority when SSL/TLS is being used. Defaults to yes
.verifyCertBundle no A path to a CA Bundle file containing certificates of trusted CAs. The CA Bundle is used to ensure that the Elasticsearch server being connected to was signed by a valid authority. Only applicable if verifyCertificate
isyes
.verifyHostName no Controls how the name in the Elasticsearch server's certificate is validated if SSL/TLS is being used. If set to yes
or not specified, the name must match the value in thehost
setting. If set tono
, the hostname is not validated. If set to a different value, the hostname must match the value in the setting. For example, if the value is set tomyserver
, the name in the Elasticsearch server certificate must bemyserver
in order for the connection to be allowed.clientCertificate no A path to a client certificate supplied to the Elasticsearch server for TLS/SSL connections. Defaults to not using a client certificate. clientKey no A path to a client private key used for TLS/SSL connections made to the Elasticsearch server. Defaults to not using a client private key. Event Group Section (1 per event group)
Each event group specified in the
eventGroupNames
property of the[General]
section must have a section defined which contains details about the particular event group.The section name must match the name of the event group in the
eventGroupNames
property (for example:[eventgroup1]
).
Name Required Description topics yes The list of topics to subscribe to the DXL fabric for event notifications, delimited by commas.
For example:
es1,es2,es3
For each event notification received, corresponding documents with the event payload are indexed to Elasticsearch.
documentIndex yes Index to use in storing the event document to Elasticsearch. See the ElasticSearch Index API Documentation for additional details. documentType yes Type of the document to store to Elasticsearch. See the ElasticSearch Type API Documentation for additional details. idFieldName no Name of the field in the event payload whose corresponding value should be used as the ID of the document stored to Elasticsearch. See the Elasticsearch Id API Documentation for additional details. Defaults to using an unique ID that Elasticsearch automatically generates.
For example, an event payload may have the following content:
{ "myid": "12345", "mytext": "hello world" }If idFieldName is set to
myid
, the value12345
would be extracted from the payload and used as the document ID. If idFieldName is set but the value cannot be found in the event payload, the document will not be stored.NOTE: The idFieldName can currently only refer to a key which exists at the top-level of the event payload. In order to map the ID from a field which appears in a nested structure in the event payload, a "transformScript" would need to be used.
transformScript no Path to a Python script which will receive the event payload and optionally transform it into documents to be stored into Elasticsearch. The transform script must define an on_event
function which accepts two parameters: thedxlclient.message.Event
object received for the event callback and a dictionary containing a default set of parameters for a corresponding document to be stored to Elasticsearch.
Logging File (logging.config)¶
The optionallogging.config
file is used to configure how the application writes log messages.