DarkLight Release Notes

Released September 13, 2019

New Features

  • New widget in the bottom corner shows the disk usage of the server. Colors and warning levels are configurable in Preferences. (#425)
  • Improvements to SPARQL auto-complete (#565)
  • Improvements to Playbook Foldering, including a new option to Enable/Disable all playbooks in a folder
  • Improvements to Results Graph, including new layout and pinned arrangements (#647)
    • The Results Graph V2 has been integrated into the standard Results Graph view (#677)
    • Graph literals (string data properties) can be edited via right-click in the graph. The data property can also be edited. (#696)
  • Graphs in the Results Table (loaded from Contextual or Working Memory) can be deleted via right-click (#702)
  • New Automatic Date Patterns added (#657)
  • Secure Data Sources and Feeds with missing certificates now allow you to download the cert locally so you can add it to the TrustStore. (#599, #649)
  • Data Feeds can be optionally throttled to an average number of events per second (#315)

Bugs Fixed

  • The SWRL Editor view errors incorrectly when an existing rule is edited (#509)
  • The SWRL Editor view errors on restart if it is left open. (#538)
  • Query Results Tree sorts by the Order By value (#483)
  • Select the object class options button in the table reifier error fixed (#662)

Known Issues

  • Playbook foldering sometimes has unexpected behaviors
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released August 13, 2019

New Features

  • Sub-playbooks can be set to run optionally without the parent playbook failing validation (#516)
  • Results Summary shows Object Types using the Class ontology hierarchy when the Show Types (tag icon) mode is on (#548)
  • Improvements to the Results Graph layout (in beta view Results Graph V2) (#270)
  • Results Summary linked objects show their ID/IRI as a link to view the object's details directly (#583)

Bugs Fixed

  • Playbook contextual menu no longer has a menu item of Assign to Cyber Effects Matrix (do that from the CEM view) (#465)
  • Renaming an untitled playbook no longer leaves behind an empty "Untitled" playbook (#492)
  • Import/Export of Collections is fixed (#519, #529, #535)
  • The console no longer prints a debugging list of all playbooks on playbook save (#521)
  • Refreshing default queries in the Query Results Manager no longer throws errors (#533)
  • Fixes to text box rendering in Results Summary view (#575)
  • Results Summary view no longer cuts off URL values (#582)
  • New Reify configurations that configure an object to use a JSONPath in the IRI won't save. (#588)
  • Brand new workspaces can show an error on startup attempting to load Contextual and Working Memory views. (#590)
  • LDAP validation message improvements (#593)
  • Results Summary view handles larger sets of data (#597)
  • Reify improvements with Arrays (#598)
  • Existing Reifiers with object IRIs set to "Generated" are properly reifiying (#612)
  • Reify configuration "Empty Object" works correctly again (#616)

Known Issues

  • The SWRL Editor view errors incorrectly when an existing rule is edited (#509)
  • The SWRL Editor view errors on restart if it is left open. Close the view and open it again from Window → Show View
  • Playbook foldering sometimes has unexpected behaviors
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released July 31, 2019

New Features

  • Results Summary view has been redesigned for improved readability
  • Reify Configurations can now create an IRI using multiple data property values
  • Reify Configurations array handling is improved
  • New Query LDAP allows you to pull data from your LDAP service and use it to do things like populate user accounts and devices
    • LDAP Data Source to support the new step

Bugs Fixed

  • Deleting playbooks no longer leaves the deleted name in the list (#250)
  • SWRL Rules view filter uses all data in the rules instead of just the title (#417)
  • Create New Object step now allows the IRI to be set fully from FreeMarker (#493)
  • Syntax highlighting now responds to arrow keys during text selection (#514)
  • Auto-opening sub-playbooks will now return to the Playbook perspective before they open (#517)
  • Playbooks can be added to the Cyber Effects Matrix cells again (#537)

Known Issues

  • Brand new workspaces can show an error on startup attempting to load Contextual and Working Memory views.
  • The SWRL Editor view errors on restart if it is left open. Close the view and open it again from Window → Show View
  • New Reify configurations that configure an object to use a JSONPath in the IRI won't save.
  • Collections exported via the Import/Export mechanism (.dlx files) do not import.
  • Playbook foldering sometimes has unexpected behaviors
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released July 17, 2019

Bug Fixes

  • Results Graph view loads graphs
  • MISP Step loads its step editor

Released July 15, 2019

New Features

  • Improves connections with AWS by adding a setting to specify the server's FQDN (Fully Qualified Domain Name)
    • The checkbox on the client was removed because it is no longer needed if the setting is configured
  • Automatic P2 updates can be disabled on startup in the settings file (useful for Docker installs)
  • Playbook Metrics list now includes sub-playbooks
  • When running a package through manually in debug mode, all sub-playbooks that were used now open and have debug information that can be used in the Inventory view automatically

Bugs Fixed

  • Newly created classes are available to the reasoner without a restart
  • Error fixed when adding an icon to a class
  • Table reifier list updates correctly when new reify configs are added
  • Date Normalized load error fixed
  • Playbook folder nesting algorithm prevents looping of parent/child relationships
  • Default queries in Query Results Manager (beta) only show RDF Data Sources now
  • Post to Teams and Post to Slack text areas now have scroll bars

Known Issues

  • The MISP Step does not load its interface
  • The Results Graph does not load on some launches
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released July 5, 2019

New Features

  • New Playbook Manager can organize playbooks into named folders, and playbooks can be sorted
  • New Global Variable feature to create FreeMarker templates that are available to any playbook
  • New Playbook Metrics view to show a table of recently-run playbooks and steps, with timing information
  • New Query SQL Database step to reach out to a SQL database
  • New view to show Data Property Ranges (string, date, etc)
  • New view to show SWRL Rules used in ontologies
  • Syntax highlighting improvements (FreeMarker coloring, line numbers)
  • New Results Graph visual design and features
  • Beta release of the Query Results Manager to let you write queries into your data and show the results in combinations of trees, tables, numbers, and pie charts

Bugs Fixed

Known Issues

  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released June 18, 2019

New Features

  • Updated Eclipse platform from version 4.7 "Oxygen" to version 4.11 "2019-03"
  • Many stability improvements on startup and shutdown
  • Client/Server connection is improved. The client will wait and automatically reconnect to the server when necessary.
  • In addition to the server console to input commands to the server, an optional SSH connection can be made to the server.
  • Reification speed has been improved, along with pipeline improvements, resulting in higher throughput of events through playbooks.
  • Cyber Effects Matrix updated to use the April 2019 MITRE ATT&CK update, which includes new Tactic of "Impact"
  • Updated connections between the CIS20 and Security Regulations Mappings View
  • Export dialog now automatically checks sub-playbooks of the selected parent playbook, and automatically checks ontologies that are imported by checked ontologies
  • FreeMarker variable added to the Reason step to directly reference the named graph being sent to the reasoner database. The Reason step default query is updated and uses the new variable.
  • Added new date pattern yyyy-MM-dd'T'HH:mm:ssz to the list of automatic Date Codes (MM/dd/yyyy)
  • Metrics collection for Playbooks and Steps returns, this time as a BYOPSQL (Bring Your Own PostgreSQL) solution that allows you to store data in a location of your choice.
    • Playbook Editor Views have a checkbox to allow metrics to be collected over time. Collected metrics for a playbook are shown as a heatmap to see how often each step was activated in the time range specified.
    • Console commands (and soon a new view) let you see how many playbooks and steps have been running and how long each took to execute.
    • DarkLight Preference lets you choose how long to keep metrics data and specify the connection details to your instance of PostgreSQL.
    • The Calculate Statistics step still uses the built-in RDBMS database for data storage

Bugs Fixed

  • In the Inventory View, values with \r or \n line breaks in table cells no longer get converted into real line breaks. Their native text characters are shown.
  • Ontology manager correctly handles loading an ontology with a duplicate namespace to one already loaded. The duplicate prefix will be renamed "ns1"
  • Importing ontologies with missing imports no longer prevents DarkLight from starting up
  • Creating a Reify Configuration with the same name as an existing one and choosing to overwrite the previous one now correctly shows the new sample json instead of the previous sample.
  • Deleting a Collection no longer throws an error in the console
  • Ontology Detail views always show the RDF Label text, even when not in Friendly Label mode.
  • New workspaces get the Working and Contextual Data Sources created automatically
  • When running in client/server mode, the Folder Data Feed "Browse" button correctly shows the file system of the server instead of the client.
  • Subplaybooks that exit early (i.e. not on a yellow node) no longer prevent the calling playbook from finishing
  • Changing a preference in Window→Preferences that requires a system restart will fully restart the server
  • Some step editors better support horizontal scrolling if the panel size is too narrow

Known Issues

  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released March 28, 2019

New Features

  • DarkLight installer for Windows and MacOS no longer warns about the installer not being trusted.
  • SSL ElasticSearch Data Source
  • New Font (DejaVu) and syntax coloring for SPARQL and JSON editors
  • New Step: Clear Package Variables - Reduce the size of the package; especially useful with the Split step.
  • New Step: Set Global Graph - Allow split packages to share the same graph instead of duplicating it.
  • New Step: FreshService - Communicate with the FreshService ticketing API
  • Load Sample Input/Package now lets you run without debug mode to reduce resources when running manually
  • Ontology Detail views: moved the edit/delete buttons to the left side
  • Results Graph layout changed for better readability
  • CIS20 Labels updated to match latest version
  • Combine Step lets you choose to merge the graphs but clear specific variables
  • Reify JSON Configuration lets you specify a property value of an object array to be used in the object's IRI
  • On startup, any ontologies added manually will have their prefix added to the prefix.mapper file automatically.
  • Client/Server connections have been improved to allow for servers that have internal and external addresses (like Amazon Web Services AWS)
  • Web Request Step is now listed in the Query section of steps (formerly was in Output)
  • New icons added to graphs in the Data perspective
  • MS Teams step uses the monospace font and syntax coloring
  • Publish to Database is now called Publish to Knowledge Base (no functionality changes)
  • Taxii 1 (XML) view has been deprecated and replaced by Taxii 2 (JSON)

Bugs Fixed

  • Fixes to creating Ontology items (Class, Data Property, Object Property)
  • File→Export dialog will not let you click OK until a file path has been set
  • Reify Multiple JSON Objects step protects you from reifying into the _default_ graph
  • Added vertical scroll bars to E-mail and Graph String Filter steps
  • Add Type step no longer throws errors when adding additional types
  • Value String Filter bug fixes, and clarification for "has length" comparator
  • Value Number Filter bug fixes with saving values
  • Fixed Schedule Feed bug where clicking on a second schedule feed would use the same value as the first
  • Fixed Results Summary view "E-mail Report" feature
  • Ontology views prevent "punning" - naming an object property and a data property the same IRI

Known Issues

  • Brand new empty workspaces do not automatically create the built-in Data Sources for Working Memory and Contextual Memory. This can cause an error on startup when they are accessed. Use File→Import and import this file to create the default Data Sources: defaultrdfdatasources.dlx
  • When running in client/server mode, the Folder Data Feed "Browse" button will show the file system of the client instead of the server. Type the path from the server in the box instead.
  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released November 15, 2018

New Features

  • DarkLight installer for Windows no longer warns about the installer not being trusted. (Mac users still need to right-click on the installer and choose "Open")
  • New View for Taxii2 data subscriptions. Taxii2 data arrives as STIX2 JSON data and is fed to Cognitive Playbooks via a new Taxii 2 Ingest step.
  • RDF Graph (SPARQL) Data Sources can now be configured via the Data Sources view and additional databases can be referenced in addition to the built-in Working Memory and Contextual Memory.
  • RDF Graph database Stardog has been updated to version 5.3.4 bringing more efficient query plans.
  • New Step: Cortex enables using FreeMarker variables to send in requests to external Cortex analyzers configured on your network.
  • Cyber Effects Matrix was updated to use the October 2018 MITRE ATT&CK release.
  • Schedule Data Source added an option to countdown a number of seconds in addition to Cron patterns.
  • New visual theme applied to views

Bugs Fixed

  • After doing a File→Import that involves ontologies, the DarkLight (Standlone/Server) must be restarted before the new ontologies can be used by the reasoner.
  • The FreeMarker built-in of ?url to add HTML escaping to a string no longer needs the (UTF-8) condition on it.
  • The Collection Manager will now show the first 5,000 items in a large collection instead of only warning that it was large.
  • "Replace Input Value" option now works with more variations of FreeMarker syntax
  • Table Reify IDs can now be assigned to object names.
  • Table Reify Load dialog will now correctly recognize pipe | and tab separated values
  • Date Normalize step correctly uses "epoch" option as one of the "automatic" choices.
  • JSON Reifier no longer fails to reify an object's ID does not exist in the incoming data.
  • Reify JSON step will error stop the playbook instead of going down the False output
  • Cyber Effects Matrix correctly shows the names of playbooks on hover
  • Class and Property Selection dialogs correctly sort by any of the columns
  • Cyber Effects Matrix now correctly shows MITRE Mitigation information for selected Techniques

Known Issues

  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released October 3, 2018

New Features

  • The Reify Configuration for JSON adds an Empty Object to the top of the sample to allow an easy way to create a sub-object in the reified graph.
  • The CIS 20 has been updated to the latest order

Bugs Fixed

  • The Reify Multiple JSON step now works correctly
  • When creating a new property or class from a dialog, the newly-created item is pre-selected in the list

Known Issues

  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • After doing a File→Import that involves ontologies, the DarkLight (Standlone/Server) must be restarted before the new ontologies can be used by the reasoner.

Released September 25, 2018

Bugs Fixed

  • The Reify Table Row step now correctly reifies CSV/tabular data.
  • When specifying the range for a new ontology data property, the full list was not being shown. Now, the ranges that DarkLight understands by default are listed first, then any other ranges declared by third-party ontologies are shown.
  • In step editors with a large text box (Queries, Set Value, Slack & Teams, etc.), smart quotes (curly quotes) are replaced with straight quotes at runtime. This prevents the smart quote characters from causing problems in JSON and SPARQL statements.
  • The Schedule Data Feed type can once again specify a custom Cron pattern in addition to the built-in presets.
  • The Graph String Filter can now use FreeMarker expressions as a comparator.
  • Lists of Collections have been alphabetized in steps that show them.
  • The Import dialog now shows the label of the assets that are going to be overwritten instead of their internal ID.
  • Cancelling an Import actually cancels the import now.
  • The Calculate Statistics step "purge" options no longer purge data from keys other than the one specified in the step.
  • The Replace Text step will output on the False side is no match is made.

Known Issues

  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • After doing a File→Import that involves ontologies, the DarkLight (Standlone/Server) must be restarted before the new ontologies can be used by the reasoner.

Released September 15, 2018

Notes:

  • The FreeMarker output of Replace Text, Text Operations, and Normalize Date has been updated, so if you have any existing playbooks using these steps, you will want to review them. See the step pages for more information.
  • The Edit Collection step needs to be updated if you are specifying a full IRI in the collection field. See the step page for more information.
  • Version 3.6.0 and 3.6.1 were internal-only releases - You didn't miss any updates 8-)

New Features

  • Client/Server communications has been rewritten and improves communication between the client and server to reduce the number of times the client loses its connection. Note that there is a new firewall port required.

Bugs Fixed

  • Several step interfaces have had bug fixes related to their restructuring in the previous release.
  • On the Mac, clicking on a text field will no longer replace the previous text with the first field in the step. (#1755)

Known Issues

  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • After doing a File→Import that involves ontologies, the DarkLight (Standlone/Server) must be restarted before the new ontologies can be used by the reasoner.
  • The Calculate Statistics Purge options delete more data then they should. Use "Don't Purge" for now.

Released August 28, 2018 Notes:

  • The FreeMarker output of Replace Text, Text Operations, and Normalize Date has been updated, so if you have any existing playbooks using these steps, you will want to review them. See the step pages for more information.
  • The Edit Collection step needs to be updated if you are specifying a full IRI in the collection field. See the step page for more information.
  • Version 3.6.0 and 3.6.1 were internal-only releases - You didn't miss any updates 8-)

New Features

  • Ontology information is now stored on the client, resulting in significantly faster loading of lists of classes and properties in Client/Server mode.
  • Client authenticates with a name and password to gain access to the server using either a config file or LDAP groups.
  • Step Editors are more consistent visually with each other, and now show what kind of FreeMarker output a variable will create.
  • Step Editors that use SPARQL or JSON editors have syntax highlighting and a new font.
  • Error messages from steps in the Inventory view are more detailed.
  • A new Manage Statistics view shows the current contents of the data being collected by the Calculate Statistics step. Individual entries and keys can be deleted from this view as well.
  • The Results Chart view has been rebuilt and shows a bar chart for the data that is in the Results table.
  • In client/server mode, the client can no longer switch workspaces from the File menu. Use workspace load from the server console instead.
  • Certificates can now be loaded into the DarkLight server via command line.
  • DarkLight has a new public key that can be used to make secure connections to other tools.
  • DarkLight installer graphics updated.

Bugs Fixed

  • Sometimes viewing a Reify configuration would prevent it from correctly using its Match Pattern on live data. This has been corrected. (#2062)
  • Regular Expressions (regex) no longer need to match the entire string in some steps. (#1744)
  • When steps were copied and pasted into a new playbook, changes to steps did not trigger a save notification. (#2106)
  • The Date Normalize step checkbox of "Exit step on False if no match" was not working. It has been fixed. (#2064)
  • The Choose Workspace dialog now uses the native operating system file list instead of a generic one. (#2028)

Known Issues

  • Client/Server connection issues can sometimes become interrupted which disconnects the client. The server continues to run even if the client is disconnected.
  • Starting up a Linux server for the very first time has some order-of-operations issues dealing with certificates. If you are creating a new Linux server, please contact support first.
  • The Reify JSON step does not correctly set the "Use Match Pattern" option when first created. A workaround is to use the copy/paste step feature to use an existing step from another playbook.
  • Collecting metrics on the throughput of playbooks and steps has been removed from this version because it was inefficient and slowing down the number of events that could be processed. The metrics capability is being reworked to use an external database for greater speed. The "Show Metrics" feature will return in a future version.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)

Released July 12, 2018

Automatic update in 3.4.1 and 3.5.0 has a bug that prevents it from working. Version 3.5.1 must be installed from the installer, and can be installed over your existing version.

New Features

  • New Step: ThreatConnect Use this step to query information from and post information to your ThreatConnect subscription.
    • Each cell can be rated (High, Medium, Low) to reflect your confidence in your company's ability to defend against that technique for each of the stages. Ratings for Techniques are averaged together into Tactic ratings, and each column has a cumulative score.
    • Cells that contain both a note and a playbook now show both icons.
    • Hovering over an icon shows a list of selected playbooks and a preview of the note.
  • Step: NSLookup Improvements:
    • If you send in a host, it returns an IP address. If you send in an IP address, it returns a host.
    • Exits down the False side of the step if the IP/host cannot be resolved
  • New Step: Generate OpenC2 Message Use this step to create a message in Open C2 JSON format that can be sent to other systems.

Bugs Fixed

  • Pasting a large number (~ >10) steps no longer causes a Concurrent Modification error (#2026)
  • Automatic update checking has been restored (certificate error) (#2040)

Known Issues

  • In Client/Server mode, access to the ontology information (e.g., choosing a Class or Data Property) is very slow (~2 minutes to populate a list). We suggest authoring playbooks in Standalone mode and using Import/Export to move playbooks to production in client/server mode. The entire ontology-management system is currently being overhauled to address this issue.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)

Released June 22, 2018

Automatic update in 3.4.1 has a bug that prevents it from working. Version 3.5.0 must be installed from the installer, and can be installed over your existing version.

New Features

  • New: Step: Send To Kafka Use the Data Source and Step to send data to a Kafka server.
  • New: Step: Post To Teams Use the Data Source and Step to send a message in text or as a formatted card to Microsoft Teams chat service.
  • New: Step: Base64 Encode or Decode a string of text to/from Base64.
  • New: Step: SHA-256 Encode a string to text to SHA-256 in either Hex or Base64.
  • Updates to Cyber Effects Matrix New order of defensive tactics and resiliency information from NIST; Topic Details from MITRE.
  • When creating Playbooks, steps can be copied from one playbook and pasted into another playbook. Even on a different system
  • DarkLight keeps statistics about playbooks and steps when they run. This data was not stored efficiently and used a lot of physical disk space. This version stores much less information and takes up very little disk space. Note that if you are upgrading from a previous version, you will need to delete the rdbms folder from your workspace.
  • As the full set of metrics are no longer kept, the Show Metrics feature of playbooks is now on-demand and controlled per playbook.
  • Inventory View updates (multiple packages, arrays as tables, select cell for freemarker syntax)
  • Newly-created workspaces are set to remote automatically
  • Web Request step improvements (Freemarker in header, custom content-type, 404 error handling)
  • Combine Packages step keeps variables and puts them in an array
  • Icons have been updated to have high DPI versions

Bugs Fixed

  • Web Request Step improvements
  • Pressing the Enter key in a list filter no longer closes the dialog
  • Edit Collection step correctly adds a new collection (but does not use FreeMarker)
  • JSONPath step no longer throws error when fields are empty
  • Reify configs can use xsd:long properties, and can use integers in IRIs
  • Typos fixed in Cyber Terrain view

Known Issues

  • In Client/Server mode, access to the ontology information (e.g., choosing a Class or Data Property) is very slow (~2 minutes to populate a list). We suggest authoring playbooks in Standalone mode and using Import/Export to move playbooks to production in client/server mode. The entire ontology-management system is currently being overhauled to address this issue.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)

Released May 23, 2018

New Features

  • The Cyber Effects Matrix now shows information about Groups and Software for each technique.
  • The Cyber Effects Matrix can be filtered down to show only the Techniques related to specific Groups, Software, Operating System Platform, and Stage.

Bugs Fixed

  • The Web Request step no longer fails to load when it is used as a new step. (#1900)
  • The Execute Task step now allows empty arguments (#1845)

Known Issues

  • In Client/Server mode, access to the ontology information (e.g., choosing a Class or Data Property) is very slow (~2 minutes to populate a list). We suggest authoring playbooks in Standalone mode and using Import/Export to move playbooks to production in client/server mode. The entire ontology-management system is currently being overhauled to address this issue.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the <workspace>/rdbms/metadata/ folder can be safely deleted (quit DarkLight first) if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)

Released May 3, 2018

New Features

  • New Step: A Playbook can now directly send its package round-trip to another playbook with the Run Playbook step. This is useful for "utility" playbooks that do a specific function. (#1628)
  • New Step: Calculate Statistics lets you perform basic statistical calculations like mean, median, and standard deviation, on incoming numeric values. Values can be collected by a custom key and many options exist for when to trigger the statistics and what to do with the collected data. (#1651)
  • The Cyber Effects Matrix perspective has been completely updated to allow you to assess your response to the Tactics and Techniques from MITRE's ATT&CK across the defensive spectrum. Playbooks and custom notes can be assigned to any cell
  • The Execute Task step has been made more secure by requiring that the script live in the server's workspace directory instead of allowing the client to specify a full path to the script. (#1677)
  • Web Request step allows for the specification of content-type and other headers (#1686)
  • The Reason step can now be used without reasoning. This is useful when you want to query information from the ontology without spending the computational resources to reason over the graph. (#1697)
  • The Reify Configuration can now use more complicated JSON array patterns for configuring objects and properties. (#1643)
  • The Reify Configuration Match Pattern Configuration can now use options like "contains, starts with, and regex" (#1444)
  • The Reify Configuration attempts to match an incoming date with a list of date formats. The list includes both Unix (epoch) seconds and milliseconds time. (#1753)
  • When viewing results, clicking on an object in the Results Graph will show that objects properties in the Results Summary view. (#1712)
  • The Stardog database has been updated to version 5.2.3 (#1648)
  • The Inventory view now turns a step red when it has an error, and shows the error message in a new tab. (#1723)
  • HTTP Post Data Feed now shows the full URL path as options are entered. (#1733)

Bugs Fixed

  • Small files caused problems on import. (#1805)
  • The Reify Configuration no longer lets you save a configuration without a data sample (#1752)
  • Playbooks using the Post to Slack step will not activate if the Slack Data Source is not available or active (#1734)
  • Occasionally on startup, the internal AMQ server would not initialize correctly. It will now automatically retry. (#1761)
  • Preferences: Web Server Settings → Secure Connections Only checkbox was not correctly staying checked. (#1747)
  • The ElasticSearch step would sometimes deselect the Data Source. (#1756)
  • On macOS, clicking a checkbox in a list no longer inadvertently toggles other checkboxes. (#1814)
  • The Web Request step now shows `&` correctly in the URL preview (#1824)

Known Issues

  • The Web Request step will fail to load when it is used as a new step. If it is already in an existing playbook it will work correctly. (#1900)
  • The Execute Task step does not allow empty arguments (#1845)
  • In Client/Server mode, access to the ontology information (e.g., choosing a Class or Data Property) is very slow (~2 minutes to populate a list). We suggest authoring playbooks in Standalone mode and using Import/Export to move playbooks to production in client/server mode. The entire ontology-management system is currently being overhauled to address this issue.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the <workspace>/rdbms/metadata/ folder can be safely deleted (quit DarkLight first) if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • If you are going to use a workspace from a version previous to 3.2, you need to delete the following directories
    • workspace/config/application/keystores
    • workspace/store
    • workspace/xmldb

Released January 26, 2018

New Features

  • XML Queries can be saved and named. Used in the XML Viewer view and Query XML/STIX step.
  • Playbooks can be manually triggered. Useful for playbooks that use a scheduled data feed as an input.
  • Data Sources behind-the-scenes improvements which will allow for future enhancements
  • New Freemarker Expression pkgdata() allows one-step access to a property value of a graph object.
  • Results view graph objects can dynamically toggle their properties on and off, reducing clutter.
  • Results view graph nodes are 50% wider, showing more of their label.
  • Steps that let you add a variable name now validate the name before saving.
  • Stardog graph database updated to version 5.0.5.1
  • New step: Web Request allows a playbook to GET, PUT, POST, DELETE to a remote server
  • Normalize Date step allows for 'epoch' as an output pattern
  • Export and Import Playbooks and More and other artifacts can now also be initiated from the File menu.

Bugs Fixed

  • Improvements to the Inventory system
  • Regex Step allows the capture of multiple segments (#1647)
  • Clicking Add Type in the Add Type step no longer causes errors (#1625)
  • Steps that use "Replace Input Variable" checkbox now work correctly (#1620)
  • Client/Server Export Data command improvements (#1586)
  • Send E-Mail step uses a plain text editor instead of RTF to allow for advanced FreeMarker usage.(#1170)
  • JMS Step now uses a Data Source to manage its connection, including Name and Password. (#1473)

Known Issues

  • The Web Request step does not correctly send POST message due to an issue with headers.
  • In Client/Server mode, access to the ontology information (e.g., choosing a Class or Data Property) is very slow (~2 minutes to populate a list). We suggest authoring playbooks in Standalone mode and using Import/Export to move playbooks to production in client/server mode. The entire ontology-management system is currently being overhauled to address this issue.
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted (quit DarkLight first) if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • If you are going to use a workspace from a version previous to 3.2, you need to delete the following directories
    • workspace/config/application/keystores
    • workspace/store
    • workspace/xmldb

Released December 6, 2017

New Features

Bugs Fixed

  • User interface fixes to various step editors (Query Package #1573, Send E-mail #1590)
  • Improvements to referencing nested array objects in Reify: Turning Data into a Graph (#1589)
  • Non-functiong checkbox in the Subscribing to TAXII Data (XML) view has been removed (#1583)
  • Fixed an issue where loading a sample file into the Inventory view caused a crash (#1604)

Known Issues

  • Step: Query Elasticsearch will reset the Data Source field back to the "Choose" setting when the step is edited. (#1619)
  • Step: Replace Text does not correctly replace the input variable when that option is checked. Save the output into a new variable name instead. (#1620)
  • Adding more than one blank type in Step: Add Type causes an error. Configure each one as they are added.
  • JMS Step does not make use of an entered Name and Password. (#1473)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • If you are going to use a workspace from a version previous to 3.2, you need to delete the following directories
    • workspace/config/application/keystores
    • workspace/store
    • workspace/xmldb

Released November 1, 2017

New Features

  • DarkLight will automatically check for updates and notify you when one is available. Standalone versions will update themselves. Server updates via the console, and Client updates from its server.
  • Step interfaces that require FreeMarker Expressions now show a icon next to them.
  • Several other updates to step editor user interfaces
  • New Steps:
  • The Inventory View: Testing Playbooks input shows the previously-loaded sample when opened again
  • Improvements in Client/Server connections and feedback
  • The internal AMQ no longer runs on its default port to prevent collisions with an external queue running on the same device.
  • Improvements to json reifiers with arrays
  • A system-wide certificate manager allows ssl connections for data sources and the Taxii client
  • DarkLight can receive data via HTTP POST as a Data Feed

Bugs Fixed

  • Imported Reify Configurations did not become available in the current session (required DarkLight restart) (#1469)
  • Step: Execute Task now accepts FreeMarker Expressions as arguments. (#1528)
  • (Client/Server Only) Export to a path on the client now works. (#1525)

Known Issues

  • JMS Step does not make use of an entered Name and Password. (#1473)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours shows the total number instead. The time-restrained version will return in a future update. (#1440)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • If you are going to use a workspace from a version previous to 3.2, you need to delete the following directories
    • workspace/config/application/keystores
    • workspace/store
    • workspace/xmldb

Released September 26, 2017

New Features

  • Core Java Eclipse framework upgraded from "Neon" to "Oxygen"
  • Linux server can now save passwords (Requires libsecret-1.so.0 to be installed) (#1461)
  • New Data Feed to access STOMP protocol (e.g. Apache Apollo) (#1464)
  • Console installer (-i console) reports if no license file is present before installing. (#1480)
  • Reifier configurations created with version 3.0.4 should be opened and saved with 3.1.0 to update to the new format.

Bugs Fixed

  • JSON Reify incorrectly used to set the path of properties of secondary objects (e.g. $.event_data.SubjectUserName became $.SubjectUserName) when saved. (#1474)
  • Renaming a Reify Configuration used to create a duplicate version with the original name. (#1468)
  • JSON Reify correctly handles variable names with spaces or other characters used in JSON, like $ [ ] . (#1450)
  • ElasticSearch Step query clears queries that do not contain JSON (e.g. Starting a query with an API call) (#1427)

Known Issues

  • JMS Step does not make use of an entered Name and Password. (#1473)
  • Imported Reify Configurations do not become available in the current session (requires DarkLight restart) (#1469)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours has been disabled due to performance issues. It will return in a future update. (#1440)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • Step: Execute Task does not accept FreeMarker Expressions as arguments. (#1528)
  • (Client/Server Only) Export to a path on the client fails. Enter in a path on the server. (#1525)

Released September 13, 2017

Bugs Fixed

  • Some date formats were incorrectly getting their times set to midnight.
  • JSON Reify had issues with setting the ID of a class, and incorrectly saved literal arrays. (#1453)

Known Issues

  • JSON Reify incorrectly sets the path of properties of secondary objects (e.g. $.event_data.SubjectUserName becomes $.SubjectUserName) when saved. (#1474)
  • JMS Step does not make use of an entered Name and Password. (#1473)
  • Imported Reify Configurations do not become available in the current session (requires DarkLight restart) (#1469)
  • Renaming a Reify Configuration creates a duplicate version with the original name. (#1468)
  • JSON Reify does not correctly handle variable names with spaces or other characters used in JSON, like $ [ ] . (#1450)
  • ElasticSearch Step query should only contain JSON. Starting a query with an API call, (e.g. GET winlogbeat-) results in a playbook crash. (#1427)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours has been disabled due to performance issues. It will return in a future update. (#1440)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)

Released September 12, 2017

Bugs Fixed

  • Reify JSON Step: changing the Input Data field after saving the playbook resulted in an error. (#1434)
  • Ingest Step: a file that consists of a list of JSON objects (JSON Lines) was not correctly ingested. (#1436)
  • Some valid xsd:dateTime formats threw errors during the Reify step (#1428)
  • Reify Table Step always reverted back to Match Pattern (#1441)
  • Ontology Import failed if it tried to overwrite an existing one. (#1362)
  • Export did not export ontologies unless the export tree is expanded (#1424)
  • Export dialog did not uncheck checkboxes once checked (#1448)
  • Regex Step "replace input variable" was not actually replacing if the FreeMarker template was in the form of ${variable}. (#1425)
  • JSON Path step would throw an error if the path was not found instead of sending the package down the False side of the step. (#1382)
  • Split Package step sample no longer shows FreeMarker example (#1445)

Known Issues

  • Some date formats were incorrectly getting their times set to midnight.
  • JSON Reify has issues with setting the ID of a class, and incorrectly saves literal arrays. (#1453)
  • JSON Reify does not correctly handle variable names with spaces or other characters used in JSON, like $ [ ] . (#1450)
  • ElasticSearch Step query should only contain JSON. Starting a query with an API call, (e.g. GET winlogbeat-) results in a playbook crash. (#1427)
  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours has been disabled due to performance issues. It will return in a future update. (#1440)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)

Released September 5, 2017

Improvements

  • Major improvements to the connection between the Client and Server.
  • Improvements to the installer
  • Data Feeds that reference message queues can use usernames and passwords.
  • Data Feeds and Data Sources are allowed to use blank names and passwords.
  • Improvements to the Reify Configuration view
  • Step interface improvements

Bugs Fixed

  • Import dialog remembers the last location it used (#1304)
  • Splunk Query output variable honors the configuration instead of always saving as "output-var" (#1325)
  • Query Splunk Step interface allows for time bound selection (#1159)
  • DarkLight correctly stores e-mail sending information so it can send e-mail from playbooks (#1306)
  • Replace Text Step allows an empty New Text field. (#1331)

Known Issues

  • The Results Chart view has been disabled due to performance issues. It will return in a future update. (#1439)
  • The top portion of the Dashboard that shows the number of Favorites in the past 24 hours has been disabled due to performance issues. It will return in a future update. (#1440)
  • The RDBMS folder stores information about each package seen by any playbook. On systems processing several thousand packages a minute this folder can get quite large (tens of gigabytes). A future update will automatically thin this data out. In the meantime, the folder can be safely deleted if it gets too large for your system, but the Metrics feature of playbooks will need to wait for new data to arrive before it has enough information to show. (#1364)
  • Reify JSON Step: changing the Input Data field after saving the playbook results in an error. (#1434)
  • Ingest Step: a file that consists of a list of JSON objects is not correctly ingested. A workaround is to wrap the file in square brackets and then use the Replace Text Step to remove the square bracket. (#1436)
  • Some valid xsd:dateTime formats throw errors during the Reify step (#1428)
  • Reify Table Step always reverts back to Match Pattern (#1441)
  • Ontology Import fails if it tries to overwrite an existing one. (#1362)
  • Export does not export ontologies unless the export tree is expanded (#1424)
  • start/release_notes
  • Last modified: 2019/09/13 22:51