Running DarkLight in Client/Server Mode

DarkLight can be run in either Standalone mode as a self-contained executable on one system, or in Client/Server mode where the server executable runs on one machine and the client executable runs on a different machine. The workspace and all data is stored on the server and the client is used to configure the playbooks and and view results.

New! We now also offer DarkLight Server as a Docker container. See the Running DarkLight as a Docker Container in CentOS page for details.

This page assumes you will be using a Linux system for the server. DarkLight Server does not run as a service, so we typically run the server using the screen application on Linux so the session can remain active even while disconnected from a terminal client.

Notes about Client/Server

  • Single-User Client/Server: The current version of DarkLight should be considered a single-user application. While multiple clients can connect to the same server simultaneously, and clients can be assigned names and passwords, the server does not keep track of which client has performed which actions. The client and server communicate securely over port 4167 and exchange certificates, and you can set names and passwords to restrict access.
  • Single-System Server Architecture: The current version of the DarkLight server is intended to run on a single computer system. It is multi-threaded and can take advantage of multiple cores. The roadmap has plans to expand the capabilities to use cluster-based computing.

Tip: Use 'Screen' when Running on Linux with ssh

Linux has a very helpful utility called Screen that lets you pop in and out of a second shell. This lets you leave DarkLight running in the screen while you pop out and do other things in the original shell. You can even then connect back up to the named screen from a different connection.
  • Install with apt-get install screen or yum install screen
  • Create a new named screen: screen -S darklight
  • Resume a previously-running screen: screen -r darklight (or try just screen -r)
  • To pop out of a screen: ctrl+a then d (processes will remain running)
  • See if any screens are active: screen -list
    • If there is more than one, you'll need to use the number (like screen -r 3207)
  • If the screen is marked "(Attached)" that means it's already attached in another session somewhere. You'll need to detach it first: screen -d or detach and connect in one step: screen -d -r darklight
  • If you need to get rid of a screen, get attached to it then do: ctrl+a then k (Note: this stops any processes running in that screen)
  • Best practice is to create a user that will run DarkLight. Avoid installing and running as the root user (why?). Use the lsblk (list block) command to see the partition structure of your disk and make sure DarkLight and the workspace are installed in the largest partition.
  • The Linux distribution is a bin file. To install, open a Terminal window to your file location and make the bin file an executable by using chmod a+x.
  • The installer requires the full path to your install location and does not use the home (~) shortcut. Use /home/<username>/apps/darklight instead of ~/apps/darklight.
  • Choose the "Server" option in the installer
  • Choose "Don't Create Links" in the installer
  • We do not recommend running Client or Standalone on Linux graphical interface systems.

Set Your Hostname

Some virtual machines use "localhost" as the default name of the machine. This will cause issues when DarkLight launches as it needs to know the hostname of the system it is on.

  • Use the command: hostnamectl set-hostname YOUR_HOSTNAME to set your hostname on your system.
  • If you are running DLServer on a network that has both a public and private IP (like AWS) you will need to only use the fully qualified domain name (FQDN) for the server.
    • Edit your ~/champion/server/champion.settings file and add dl.server.address=yourFQDN as a new line in the file

Open Ports in the Server Firewall

Firewall should allow the following ports:

  • 41617/tcp Connection between client/server (Active MQ)
  • 49143/tcp Connection between client/server
  • 8282/tcp "P2" update server so the server can update the client version
  • 2222/tcp (optional - configurable) SSH Connection into the Server to send commands (requires change to ini file)

These ports are all between the client and the server so you could additionally only open these ports to the IP addresses your clients are running on. The server will also attempt to reach out to the internet to check with DarkLight's servers for updates.

Opening a port

sudo firewall-cmd --permanent --zone=public --add-port=41617/tcp
sudo firewall-cmd --permanent --zone=public --add-port=49143/tcp
sudo firewall-cmd --permanent --zone=public --add-port=8282/tcp
sudo firewall-cmd --permanent --zone=public --add-port=2222/tcp

then

sudo systemctl restart network
sudo systemctl restart firewalld

Install libsecret and configure master password

Install libsecret for secure password saving to work.

  1. On CentOS, this can be installed with yum -y install libsecret
  2. Create a master password file somewhere on your system the user running DLServer has access to
    1. This is just a text file with any word in it. One suggestion is to call it "build" and put a number in it
  3. Edit the DLServer.ini file (in the same directory as the DLServer executable) and add
    -eclipse.password
    /full/path/to/password/file

    after the line that says "DarkLightServer"

Set up Remote SSH Access to the Console (Optional)

The standard way to send commands to DLServer is via the console that you used to launch the server. This requires the use of a console manager like 'screen' (see note on this page) to prevent the process from exiting when the session is closed. Another option available starting in DL 3.7.2 is to run DLServer in screen and then use an SSH connection from a second system to send commands to the server. This option does not show all of the console output, but it will allow you to send commands such as 'shutdown' and 'workspace load'. (This feature will become the only way to send commands to DLServer in the near future when DLServer runs as a Docker container.)

To enable this feature, edit your DLServer.ini file (in the same directory as DLServer) and add

-Dosgi.console.ssh=2222

to the end of the file (somewhere below "-vmargs"). You can choose any available port (it doesn't have to be 2222).

Set the SSH Account and Password (One Time)

When DLServer is running, open a terminal from another system and enter

ssh -l equinox server.address -p 2222

(where "server.address" is your server's name or IP address, and "2222" is the port number you chose in the DLServer.ini file). The built-in user and password is named "equinox" and once you log in the first time with that combination you will be prompted to create a new account name, password (8+ characters), and role (leave blank). Once this is done the equinox user will be removed and no longer able to log in.

Note that for some reason the Enter key will not advance you to the next step in the process, so after pressing Enter, press any letter key followed by the delete key to get the next prompt to show up and then delete the character you entered (we didn't write it - we hate it, too :-/ )

If you want to add another user (or something went wrong during the initial prompts) you can add a new account by entering addUser at the osgi> prompt (see bug about Enter key)

Where the data is stored

Nerdy Note: Files created in ~/champion/server from this process are org.eclipse.equinox.console.authentication.config, hostkey.ser, and store. If these files are deleted, you will need to log in again as equinox/equinox to set up a new user.

Connecting to the SSH Session Console

Once you have completed the setup process and created a new user account, you'll use that user name in the ssh command instead of equinox (something like ssh -l mynewuser server.address -p 2222)

Exiting the SSH Session without Killing DLServer

Once connected to the server (you'll have an osgi> prompt) any commands you type will be sent to DLServer. For example, if you enter exit you'll kill the DLServer process.

To cleanly exit the SSH session and leave DLServer running, type disconnect

The server is located in your install location, in a directory called /server. The application to run is called DLServer. On Linux, run it from the Terminal (e.g. ./DLServer). On Windows and Mac, follow the instructions about running DarkLight with the console.

An example new Linux session might look something like this:

[darklight@DEV-DLServer ~]$ screen -S darklight
 
[darklight@DEV-DLServer ~]$ cd ~/apps/darklight/server
[darklight@DEV-DLServer server]$ ./DLServer

Exit the screen with ctrl-a followed by d (ctrl-a means your next key will be a command to the screen process)

To return to a running DLServer in a screen:

[darklight@DEV-DLServer ~]$ screen -r darklight

Choose a Workspace

When the server runs for the first time, it will not have a workspace selected. The console will report: "A workspace has not been set."

Enter workspace load into the console and press Enter

The server will prompt you for a full path (home directory shortcuts like ~/ cannot be used) to the workspace you want to use. Entering a path that does not exist will create the path if the user account has permission.

The next time the server starts up it will use the last workspace chosen.

Where the data is stored

Nerdy Note: This path is stored in a text file in your home directory: ~/champion/server/champion.settings

Server Console Commands

  • workspace load Allows you to enter in a path to a workspace. The next time the server starts up it will automatically use the last-loaded workspace. This can also be changed from the client using File→Switch Workspace.
  • workspace active Prints the path to the currently-loaded workspace. Also shown in the title bar of the client.
  • remote start Allows the workspace to receive connections from clients. The next time the server loads this workspace it will automatically start remote services. This setting is on for new workspaces by default.
  • remote stop Turns off the remote services for the currently-loaded workspace to prevent clients from connecting.
  • shutdown Turns off the server and exits the Apache AMQ and Stardog Database processes.
    • ALWAYS try to shut down the server with the shutdown command. If you force-quit the server, you might leave other processes running which will prevent a future startup of DLServer. If you do force-quit DLServer, use ps a | grep java to list any other java processes that might be running and then kill them before running DLServer again.

Configuring Client Authentication

Starting in version 3.6.0, the client must authenticate to the server with a name and password. There are two methods for configuring authentication: name/password, and LDAP Groups.

Authentication with Names and Passwords (Default)

  1. Navigate to the server's user folder (e.g. cd ~/champion/server/)
  2. Edit the users.ini file and add any new users and a SHA256 password as needed
    1. The example below shows a new user called "sample" added with the password "password"
    2. The built-in username is dluser and the password is moon-silver-star
      1. We recommend you leave this username and password in the file until you know you can connect the client. Then create a new one and delete this default one.
    3. The role after the SHA256-encoded password must be "DarkLightUser"

Show a sample ini file

Authenticating with LDAP Groups

  1. Navigate to the server's user folder (e.g. cd ~/champion/server/)
  2. Replace the default users.ini file with the one below.
  3. Edit the highlighted rows 3,4, and 5 to match your LDAP server and authenticating user account (most likely a service account)
  4. In the highlighted rows 10 and 11, edit the "DC" entries to match your domain
  5. The ini file currently uses a group called "DarkLightUser" whose common name is "DarkLight Users". Either add this group to your Active Directory settings or change the config to match your group name. Any users assigned to that group would then be able to authenticate to DarkLight using their network login.
  6. Users must include the active directory domain with their username. For example, sampleuser@SECURECO.local
  7. If you are using ssl to encrypt passwords in transit, you will need to add the cert for the ldap system to DarkLight's workspace. See: Certificate Manager

Show a sample ini file

  1. Install the client onto a Mac or Windows machine
  2. Run the client from the install location, in the /client folder
    1. What if I don't have a "/client" folder?
  3. When prompted for a server, enter either the hostname of the server or IP address of the server. This field also contains a list of previous successful server names.
  4. Enter the name and password as defined above (either name and password or network LDAP name@domain and password).
    1. The default username is dluser and the password is moon-silver-star (you will want to change this)
    2. Check the Remember username box to store the username in your system user folder in <userfolder>/champion/client/champion.settings

The client will attempt to connect to the server, and will automatically retry a connection every 4 seconds if it cannot establish a connection. The server's console will report a successful connection.

If the server shuts down, all clients will be notified and will automatically try to reconnect. Many of the views will be disabled while disconnected from the server. Check the status lights in the bottom-right corner to see if the server is connected. If the client is trying to reconnect you'll see "Connect to Message Bus Job".

TIP for Running on Localhost

If you are running client and server on the same machine, you need to use the IP address or hostname of the machine (with the "Use Name for Server Connection" unchecked), or set the DL_COMPUTER_ADDRESS environment variable to "localhost" and then check the "Use Name…" box when connecting.

Troubleshooting Client/Server Connections

  • Enter remote start on the server's console. This allows connections to the currently loaded workspace.
    • This is enabled by default on newly-created workspaces. If it is already on a message will tell you.
  • Check the Firewall settings to make sure the correct ports are open. See the Firewall section above.
    • A quick test is to use a web browser from your client machine and connect to http://yourserver:8282. If the port is open you will see a directory listing of files (this is how the server sends updates to the client)
  • Linux servers will need the "libsecret" library loaded. On Centos, install with yum -y install libsecret
  • Ensure that the server's /etc/hostname file has your actual hostname and not "localhost"
  • If your DarkLight console is referencing your system as 127.0.0.1 you will need to add your hostname and IP address manually to your system.
    • shutdown DarkLight server
    • Option 1: hostnamectl set-hostname YOUR_HOSTNAME
    • Option 2 if that doesn't work:
      • echo 'export DL_COMPUTER_NAME="<YOUR SERVER NAME HERE>"' » $HOME/.bashrc
      • echo 'export DL_COMPUTER_ADDRESS="<YOUR SERVER IP/FQDN HERE>"' » $HOME/.bashrc
Anywhere you see a reference to :41617 should be preceded by your IP address or FQDN, not by 127.0.0.1 or other IP address internal to the server only (e.g. with AWS)
  • start/client-server
  • Last modified: 2019/08/15 18:53