As has already been highlighted in previous posts, Retrospective is not trying to compete with the fully blown SIEM (Security Information and Event Management) products. SIEMs are of course powerful but their provisioning, configuration and maintenance is tedious and time consuming. Retrospective can be configured in minutes and no special agent-like software has to be installed on servers where log files are to be processed. Moreover, besides avoiding any agent-like components, Retrospective guarantees that not a single file (even in the user home directory or /tmp) will be modified on the server.
Therefore, we can say with no hesitation that Retrospective is a very lightweight solution, which allows the user to peek into remote servers in an entirely transparent manner.
So we know that Retrospective is definitely lightweight, but what about its power? For sure providing some powerful features with the explained limiting constraints is not easy. Let’s go over the technical challenges which had to be overcome:
- Retrospective logic related to dividing data to log entries, parsing date/time information and applying filters, should be performed on the server side because:
- Transferring log files to Retrospective would consume too much time and bandwidth (especially when connecting to servers over Internet);
- It is more efficient to process log files with the use of the available server side resources.
- No special tooling should be required on the server side. Only the flagship *NIX tools such as grep and awk can be used. Additionally, the following should be noted.
- Flagship *UNIX tools on different operating systems such as Linux, Solaris, AIX, HP-UX, Freebsd or iMac could work slightly differently in terms of both functionality and performance. For example, grep on FreeBSD is slow as hell .
- To invoke these tools, there is a need for scripting logic. Different possible shells (sh, dash, bash, ksh, zsh, csh) have different features and the script syntax sometimes differs quite significantly.
- No temporary state can be written to a file on the server side. Everything has to be placed in a pipeline which processes the data, as it goes, in the most functional and efficient manner possible.
In order to overcome the above-mentioned challenges, we have come up with the following architectural setup which presents the logic related to searching.
As can be seen, processing of log data is divided into separate thread workers and each worker is concerned with a single file. Processing of local log data is different than the server side processing, which is performed by a specially crafted SSH command pipeline implemented by means of advanced POSIX shell scripting. The whole approach guarantees the following benefits.
- Resources (CPU, memory) of the node on which Retrospective is launched are not consumed.
- Different OSes, shells and tools flavors (POSIX, GNU, Solaris, other) are supported.
- When the file is searched with given search criteria, then only the data matching the criteria is actually transferred through the network to Retrospective.
- The above is also true for date/time filtering. In this case, a sophisticated polymorphic script executed as SSH commands is able to interpret and correctly filter dates in all possible formats and locales.
- In many cases, Retrospective scripts used for searching are faster than regular *NIX tools (awk, grep) thanks to the usage of special optimizations.
- In cases of profiles with many data sources and files, thanks to a parallel file processing, Retrospective could be compared to multiple instances of grep and awk tools executed simultaneously. This results in a significant performance boost.
- By assuming an adaptive approach, Retrospective exploits the resources of the remote server in an optimal manner. If servers respond quickly, then more simultaneous search/monitor SSH commands are allowed. If servers respond slowly, then the amount of simultaneous SSH commands is reduced. In the end, we get the results only as quickly as it is possible.
By facing the challenges with a solid piece of software engineering we have ensured that the latest Retrospective release is still lightweight but a definitively more powerful tool, whose functionality and performance could easily compete with some of the sophisticated SIEM components.
Furthermore, the following features and bug-fixes have been incorporated:
- Date/time is now filtered remotely on hosts accessed through SSH
- Autofind Improvements
- Optimized encoding detection for directories and filters
- Performance optimization through asynchronous processing, file group analyzis etc.
- Displaying autofind configuration mismatch in Profile Manager
- New File Browser tab for quick access to search and monitoring
- Improved layout and functionality of Host Manager
- Window and tab state is now saved at program exist and re-established at program restarts
- Tab position can now be changed by drag-and-drop
- Host panel: Save button should be active only when changes have been made
- Let user limit the number of analyzed files for automatically find the strategy
- Improved performance when working with result data stored on disk
- Improved main menu, toolbar and short cuts
2318 Encoding mismatch in Filter/Directory causes exception during search
2356 NumberFormatException while search in progress
2407 SWTException: Widget is disposed after exploding/imploding
2410 New release can be activated with expired license
2416 Locale is not correctly identified in autofind
2417 Manual activation email xml is truncated to approx 1'500 bytes
2441 Negative item counter after removing row from result table
2468 Defining manual split strategy does not clear 'mismatch' information from autofind
2491 Activation through proxy does not work with NTLM
2497 Search in a file without dates can raise an NullPointerException using Date/Time Filter
2510 NPE related to empty date/time on search tab
2516 AssertionFailedException when loading view that uses non existing profile
2523 Editing a profile used in ongoing search or tail must not be allowed
2544 Re-activation with new license is not possible
2550 Misleading result when searching in binary file
2552 Problem when displaying preferences -> Result Options on Linux
2561 Changing the path of a data source resets its encoding
2562 Unacceptable long time needed to display log entry details