Date search - file modification timestamp

ilogulog
Posts: 1
Joined: 10 Oct 2013, 07:10

Date search - file modification timestamp

Postby ilogulog » 10 Oct 2013, 07:27

Hi

I am evaluating the Retrospective product and one of the first things I have noticed, apart from being "a great bit of kit", is that the timestamp search function does not take into account the file modification date. It seems pointless to me to have to create separate profiles based on file dates before performing a search based on timestamps. Here is my use case:

    X number of hosts in profile1
    X number of source files in profile1 based on filename filter
    Each host specified "logrotates" specified source files, so the amount of source files returnable is indeterminate at profile creation.
Required result: return only lines between selected timestamps in the fastest time possible.

When starting a "grep" search using timestamps, the function pointlessly searches in source files with a file modification date outside of the search parameters thereby increasing the processing time. It would be better to reduce the source files to be searched based on timestamps selected. Granted, the function must search in archive files for individual file dates so this has to be done at execution.

Now I haven't verified this behaviour 100% because I gave up watching Retrospective after 7 mins searching through 127 source files, but please confirm if my analysis is true. If it is, I would expect this to be improved before I purchase.

Yours
ilogulog
markus
centeractive people
Posts: 16
Joined: 18 Jun 2012, 09:58
Contact:

Re: Date search - file modification timestamp

Postby markus » 15 Nov 2013, 18:16

Hiho,

thanks for your post, and sorry for not anwsering so long.
We had the new release 3.1.0 going on and loads of hectic round here.

Your are right: it would be cumbersome to define different profiles in order to filter for time ranges.

In release 3.1.0, there is the "Enable search optimization" option in the preferences under "Search/Tail(Monitor)".
With this option activated, and if a time range is given in the search filter:

Retrospective will fetch a small amount of log data from the head and from the tail of each file (when the file is > 100k),
and only search though the file if the first and the last lines of the file are within the time range.

This optimization is only done for the first time range filter given, if you use several time range filters.
And the optimization is not done for gz, tar, etc. archives, it would take to long to decompress each archive file.
As you mentioned, it would be at execution time, and we think its a too big risk to get really slow here.

Instead, we are working now on a feature of our backlog to directly filter for time ranges in the search itself on the remote host.
This will also work with archives and will result in overall faster searches for time ranges.

The existing functionality of using the head/tail content of a file to exclude it from the search without completely reading through it
will be included into this new feature.

As you also mentioned the file timestamps. We are also thinking to create an option allowing the user to exclude files from time range searches
based on their file write timestamp. We have some customers where this does not work, e.g. the timestamps of their archives does not match the contained logs at all.
But it might be useful for other customers, so it may go as a configurable option into a future release.

Yours,
Markus

Return to “General Retrospective Topics”

Who is online

Users browsing this forum: No registered users and 1 guest