Data Sources: Reach Out to External Data

The Data Sources view contains configurations to the tools your network uses to collect data. Each data source configured in the view has a corresponding step used to perform a query to that source. Separating the configuration of the source from the step allows you to update the connection information without having to go into each of the playbooks that use it.

The Data Sources View is on the PRO Playbooks and Data perspectives by default but can be added to any perspective by choosing Window → Show View and choosing the Data Sources view.

  1. Click on the Create Data Source icon or choose Create Data Source from the drop-down menu
  2. In the dialog that appears, choose the type of source to configure. Once this choice is made, it cannot be changed for this configuration.
    1. Elastic: The "E" in the ELK stack. Used by Step: Query Elasticsearch
    2. JMS: Java Message Service. Used by Step: Send To JMS
    3. Kafka: Message queuing and distribution system. Used by Step: Send To Kafka
    4. MS Teams: Chat room service in Microsoft Office. Used by Step: Post To Teams
    5. RDF: Resource Description Framework Graph Database (e.g. Stardog). Used by any step or feature that publishes or uses SPARQL queries.
    6. Slack: Chat room service (Searchable Log of All Communication and Knowledge). Used by Step: Post to Slack
    7. Splunk: Used by Step: Query Splunk
    8. Web: Any URL that can receive data. Used by Step: Web Request
  3. The new configuration will appear as UNTITLED in the list. Click on the name in the list and fill out the details for the connection as needed.
  4. Check the box next to the name in the list to activate the data source. DarkLight will initiate a test connection to the source with your given information and try to connect.
    1. The green cloud icon indicates a successful connection was made
    2. The orange asterisk icon indicates a validation error in the configuration. Hover your pointer over the icon to see the validation error message.
    3. The orange triangle icon indicates the source was connected at one time but is not responding. Hover your pointer over the icon to see the error message.
  • Tip: Using an https connection? Certs are automatically downloaded to the server (show me)

Elastic Configuration

  • Name: The name you will choose in the Query Elasticsearch step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The IP address or hostname of the Elasticsearch host. "http" is not required.
  • Port: The port number of the Elasticsearch host.
  • Username: (optional) The username if configured for the Elasticsearch host.
  • Password: (optional) The password for the username. Will be stored securely.
  • Secure Connection (optional) Check this box if your connection to ElasticSearch is via https
  • See Also: About Passwords

Note: This is to make a connection with Elasticsearch to do a query from within a playbook. If you want to bring data in from Logstash automatically, use the JMS or Kafka Data Feed.

Java Message Service (JMS) Configuration

  • Name: The name you will choose in the Send to JMS step
  • Description: (optional) free-text description you can use to make notes about the connection
  • Address: The protocol://domain:port of the JMS host. Typical protocols include ssl, tcp, stomp.
  • Topic Name: The name of the topic or queue used on the JMS service.
  • Type: Choose between Topic (messages remain after a client receives them) or Queue (messages are removed after a client receives them)
  • Username: (optional) The username if configured for the JMS host.
  • Password: (optional) The password for the username. Will be stored securely.
  • See Also: About Passwords
  • Send/Receive to/from JMS: Each source can only send or receive. Use the Receive option when sending a package from one DarkLight to another.

Note: This is to make a connection with a JMS endpoint to send data from within a playbook. If you want to bring data in from a JMS endpoint automatically, use the JMS Data Feed.

Kafka Configuration

  • Name: The name you will choose in the Send to Kafka step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The IP address or hostname of the Kafka host. "http" is not required.
  • Topic Name: The name a Kafka Consumer will use to subscribe to the data sent by this source.
  • Client ID: A unique name for this Kafka Producer that shows up on Kafka dashboards.
  • Secure Connection: Check this box to indicate that the connection should use an ssl connection.
  • Username: (optional) The username if configured for the server.
  • Password: (optional) The password for the username. Will be stored securely.
  • See Also: About Passwords

Note: This is to make a connection with Kafka to send data from within a playbook. If you want to bring data in from Kafka automatically, use the Kafka Data Feed.

LDAP Configuration

  • Name: The name you will choose in the Query LDAP step
  • Description: (optional) free-text description you can use to make notes about the connection
  • Hostname: The IP or hostname of your ldap server. Will start with either ldap: or ldaps:
    • If you use the secure version, DarkLight will retrieve the cert from the LDAP server so you can import it. See: Certificate Manager
  • Port: Standard ports are 389 for TCP, and 636 or 3269 for secure SSL
  • Bind DN or user: The user that will be authenticating to your LDAP server
  • Bind password: The password for the account

Microsoft Teams Configuration

  • Name: The name you will choose in the Post to Teams step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The Teams WebHook URL you set up for your company's Teams channel. Details are on the Post to Teams step page.

RDF Configuration

  • Name: The name to refer to this Data Source in the Publish or Query Knowledge Base steps.
  • Description: (optional) free-text description you can use to make notes about the connection
  • Address: The URL (including http/https) of the location and the port number
  • Repository Id: The name of the database or repository on the server
  • Username: Used to authenticate with the server, if needed.
  • Password: Used to authenticate with the server, if needed.

Warning: Don't Delete the Built-in RDF Sources

Many of the playbook steps rely on the pre-made default databases. Don't delete, uncheck, or change the repository ID of these items. The default RDF Data Sources are: "Working Memory" (cyber-working) and "Contextual Memory" (cyber-context)

These sources can be restored by importing this file: defaultrdfdatasources.dlx

Slack Configuration

  • Name: The name you will choose in the Post to Slack step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The Slack WebHook URL you set up for your company's Slack channel. Visit the Slack Incoming WebHooks page to create one.
  • Use this image in the "Customize Icon" section to have a DarkLight logo show up. dl_bug-black512px.png

Splunk Configuration

  • Name: The name you will choose in the Query Splunk step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The IP address or hostname of the Splunk host. "http" is not required.
  • Port: The port number of the Splunk host.
  • Username: (optional) The username if configured for the Splunk host.
  • Password: (optional) The password for the username. Will be stored securely.
  • See Also: About Passwords

Note: This is to make a connection with Splunk to do a query from within a playbook. If you want to bring data in from Splunk automatically, use the HTTP POST Data Feed.

Web Configuration

  • Name: The name you will choose in the Web Request step
  • Description: (optional) free-text description you can use to make notes about the connection
  • URL: The IP address or hostname of the source host. The rest of the URL is configured in the step's options. (We highly recommend https://httpbin.org for testing requests.)
  • Username: (optional) The username if configured for the Elasticsearch host.
  • Password: (optional) The password for the username. Will be stored securely.
  • See Also: About Passwords

Note: This is to make a connection with a web host to send a request from within a playbook. If you want to bring data in from an HTTP source, use the HTTP POST Data Feed.

To use a Data Source in a Playbook, add either the Query Elasticsearch or Query Splunk step to your playbook. If the results coming back from the search might include more than one result, you will likely want to follow that with a Split Package step and split on the variable name you used to store the result.