Tutorial: Send Data from One DarkLight to Another

One strategy you may choose to use is to distribute your data collection and processing between multiple instances of DarkLight and then send only the events you would like to explore deeper to a single instance of DarkLight. This tutorial describes the steps, feeds, and sources necessary to send a DarkLight Package from one instance to another.

In the above example, DarkLight Servers 1-3 are each processing a different kind of incoming data feed. They each have their own playbooks. Each of them includes one or more playbooks that use the steps in this tutorial to send a package (i.e. a processed event) using HTTP POST to DarkLight Server 4 which has playbooks to process, sort, and publish the packages to its memory. The DarkLight Client can then be used to view the results. (The DarkLight client can be used to connect to the other servers, too, but that's not shown in the diagram.)

The DarkLight that will be receiving the package from the other DarkLight needs to have its internal web server configured, a POST Data Feed created, and a playbook that uses the Ingest step.

Internal Web Server

DarkLight comes with a web server built in to it, and by default it has ports assigned for SSH (secure) and TCP communications. To see the current setting, or change the configuration, go to Window→Preferences and click on Web Server Settings.

The default ports are 48443 for SSH (https) connections, and 48080 for TCP (http) connections. If you want the server to only receive on the secure port, check the Secure Connections Only box. Otherwise, the server will receive on both ports. The port numbers can also be changed in this dialog.

Note: If you are using the secure connection option, the DarkLight client comes pre-loaded with a certificate that the server trusts.

POST Data Feed

In order for this DarkLight to know what to do with the data the web server receives, we need to set up a Data Feed.

In the Data Feed view, click the icon and choose "Post" as the Connection Type. DarkLight will use the information available to it to guess the hostname and create the URLs that you will need the other DarkLights to send to. (If you have checked the Secure Connections Only box then you will only see the https version of the URL.) You can also use the server's IP address when referencing it.

Check the Is Package Only Feed box to tell DarkLight not to further process the incoming data and treat like a DarkLight package.

For other options, including how Roles and security works, see: Data Feeds: Getting Data into DarkLight

Receiving Playbook Ingest

Any playbook that you want to receive the package from the other DarkLight needs to start with an Ingest step which references the Data Feed you just configured.

DarkLight Sender

Any Playbook that wants to send a DarkLight package to another DarkLight needs to use the Web Request step, which references a Web Data Source.

Web Data Source

Click the icon and choose "Web" as the Source Type. Give the Source a name that you will use in the Web Request step. Use the URL (either the domain name or IP address) of the Data Feed you set up on the receiving server. Be sure to include "https" if you intend on sending to the secure port.

Web Request Step

In any playbook that you want to use to send a package to the receiving DarkLight, add a Web Request step. (If you will be sending from several playbooks, you might want to use a Run-Playbook Step so you only have to configure this once.)

Set the Data Source to the one you just created, set the Request Type to POST, and in the Message Body, enter the FreeMarker built-in function ${packageAsJSON()}. This tells DarkLight to include the entire package from the current playbook, including all of its variables and graphs. Set Content-Type to application/json.

  • tutorial/dl2dl
  • Last modified: 2018/05/01 21:46