Prev: FAQ 4.34 How do I extract selected columns from a string?
Next: checksum calculation for file offsets
From: Dr.Ruud on 27 Apr 2010 02:22
GRP wrote:

> I would like to write a script to grep some pattern in few set of log
> files and alert it...
>
> In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
> LINE-NUMBER:FILE-CREATION-TIME" in a flat file (colon delimiter) , so
> that next time if the same pattern found in same line it will check
> against the flat file and won't alert. At the same time if it happens
> in different line number it wud check in flat file & the 2nd filed
> would be different so it will alert.
>
> Since i can't find a way to get ctime in unix (i tried with perl as
> well), would be there any other way to achieve this or better logic?

You can use git.

--
Ruud
From: sln on 27 Apr 2010 10:46
On Mon, 26 Apr 2010 21:26:21 -0700 (PDT), GRP <rengaprasath(a)gmail.com> wrote:

>On 27 Apr, 11:40, John Bokma <j...(a)castleamber.com> wrote:
>> GRP <rengapras...(a)gmail.com> writes:
>> > On Apr 27, 10:17�am, John Bokma <j...(a)castleamber.com> wrote:
>> >> GRP <rengapras...(a)gmail.com> writes:
>> >> > hi,
>>
>> >> > I would like to write a script to grep some pattern in few set of log
>> >> > files and alert it...
>>
>> >> > In order to avoid duplicate alert i plan to store the "FILENAME:ERR-
>> >> > LINE-NUMBER:FILE-CREATION-TIME" �in a flat file (colon delimiter) , so
>> >> > that next time if the same pattern found in same line it will check
>> >> > against the flat file and won't alert. At the same time if it happens
>> >> > in different line number it wud check in flat file & the 2nd filed
>> >> > would be different so it will alert.
>>
>> >> It might help if you describe when a clash can occur, i.e. why do you
>> >> need the inode creation time, and why is filename:err-line-no not unique
>> >> (I can think of reasons, but want to know the ones in your situation.)
>> > For example lets say i run my script at 10am where i found the error
>> > pattern in the line number 100, i will store in a flat file in this
>> > format "FILENAME:LINE-NUMBER" and the script will alert a msg to
>> > tivoli. In the next run at 10.30am, assume there is no error found
>> > other than line number 100, my script should not alert it since it's
>> > already alerted.
>>
>> Clear, but FILENAME:LINE-NUMBER is already sufficient to stop that from
>> happening, unless you forgot to mention something.
>>
>> > In order to achieve this along with "FILENAME:LINE-
>> > NUMBER" i need to store some other value which is constant (ex. file
>> > creation time , not change time), so that i can compare against it and
>> > if found different i assume error happened in different line.
>>
>> Do the lines in the file have a time stamp?
>>
>> > Since there is no way to capture file-creation-time i;m looking for
>> > some other way to achieve this or probably you can suggest better
>> > logic to achieve this...
>>
>> I am still missing important information. Give an example of a
>> possible clash.
>>
>> --
>> John Bokma � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � � j3b
>>
>> Hacking & Hiking in Mexico - �http://johnbokma.com/http://castleamber.com/- Perl & Python Development
>
>here it goes....
>
>Script will check for pattern in few log files (lets say its looking
>for pattern "Out-of-Memory" in /u1/app/log/WL.log & /u2/app/log/WL.log
>etc.. ).
>
>These WL.log files from different directories will delete & overwrite
>once it reach 10mb. It may happen in a day or in few days.
>
>Since i need a maintain a flat file or Database like below
><directory-and-file-name>:<err-line-number>:<some-other-constant-value-
>example.filecreation-time>
>
>the reason being maintaining a flat/database file because if the same
>pattern "Out-of-Memory" found in the same line, it should check in the
>flat file & if the entry found it wont alert. Whereas the same pattern
>found in a different line , it will not be available in the flat file
>and alert will trigger.
>
>Since the log files may rotate often i need to tag someother value
>along with filename and linenumber in the flat file which is key
>value. I can't simply store <directory-and-file-name>:<err-line-
>number> in flat file , becos once the file recreated and if the error
>comes in the same line, it wont get alerted , which is wrong. In order
>to overcome i taught of storing file creation time, which is not
>possible.
>
>Hope i clarified now.... :)

Are the log files continually appended to after and up until
the next 10 mb limit, where it is deleted?

If thats the case, then you need some other item for distinction.
The odds of this hapening should be very low though, unless the logs
are mechanical, and predictably repeatitive.

Its hard to believe there is no timestamp in the log. Why is that?

Finally, if you could timestamp the file in some way, how would
you age the entries in the flat file?
You either need the creation time of the log, or need to have whoever
creates the log to include a timestamp in the log of each entry or just
a single timestamp at the top.

-sln
From: sln on 27 Apr 2010 10:59
On Tue, 27 Apr 2010 07:46:02 -0700, sln(a)netherlands.com wrote:

>On Mon, 26 Apr 2010 21:26:21 -0700 (PDT), GRP <rengaprasath(a)gmail.com> wrote:
>>Hope i clarified now.... :)
>
>Are the log files continually appended to after and up until
>the next 10 mb limit, where it is deleted?
>
>If thats the case, then you need some other item for distinction.
>The odds of this hapening should be very low though, unless the logs
>are mechanical, and predictably repeatitive.
>
>Its hard to believe there is no timestamp in the log. Why is that?
>
>Finally, if you could timestamp the file in some way, how would
>you age the entries in the flat file?
>You either need the creation time of the log, or need to have whoever
>creates the log to include a timestamp in the log of each entry or just
>a single timestamp at the top.
>

One more thing. If the logs are being deleted, starting fresh,
how do you know when that happens?
If you have your alert script running on a timed interval,
how do you know that you didn't miss some information between the
last time it checked and when the log was deleted? How do you
get a last chance check?

Sounds like problems to me.

-sln
From: Keith Thompson on 27 Apr 2010 17:58
GRP <rengaprasath(a)gmail.com> writes:
[...]
> Script will check for pattern in few log files (lets say its looking
> for pattern "Out-of-Memory" in /u1/app/log/WL.log & /u2/app/log/WL.log
> etc.. ).
>
> These WL.log files from different directories will delete & overwrite
> once it reach 10mb. It may happen in a day or in few days.
>
> Since i need a maintain a flat file or Database like below
> <directory-and-file-name>:<err-line-number>:<some-other-constant-value-
> example.filecreation-time>
>
> the reason being maintaining a flat/database file because if the same
> pattern "Out-of-Memory" found in the same line, it should check in the
> flat file & if the entry found it wont alert. Whereas the same pattern
> found in a different line , it will not be available in the flat file
> and alert will trigger.
>
> Since the log files may rotate often i need to tag someother value
> along with filename and linenumber in the flat file which is key
> value. I can't simply store <directory-and-file-name>:<err-line-
> number> in flat file , becos once the file recreated and if the error
> comes in the same line, it wont get alerted , which is wrong. In order
> to overcome i taught of storing file creation time, which is not
> possible.

Do you have any control over what's written to the files? If so, it
would be easy enough to write a timestamp and/or other information on
the first line.

If not, checking the inode number (stat("filename"))[1] should detect
when a new copy of the file is created. The combination of device and
inode number (elements 0 and 1 of the result of stat()) should uniquely
identify the file, but just inode number is probably enough for your
purposes.

Complex file system setups can mess this up, with the same file
appearing on different devices, but that's probably not going to be an
issue for you.

--
Keith Thompson (The_Other_Keith) kst-u(a)mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"
From: Alan Curry on 27 Apr 2010 18:39
[the question boiled down to "how do I know if a log file has been rotated
since the last time I looked at it?"]

In article <lnmxwoblsk.fsf(a)nuthaus.mib.org>,
Keith Thompson <kst-u(a)mib.org> wrote:
|
|Do you have any control over what's written to the files? If so, it
|would be easy enough to write a timestamp and/or other information on
|the first line.

That's the best answer (other than "switch to FreeBSD so you can use
st_birthtime")

|
|If not, checking the inode number (stat("filename"))[1] should detect
|when a new copy of the file is created. The combination of device and
|inode number (elements 0 and 1 of the result of stat()) should uniquely
|identify the file, but just inode number is probably enough for your
|purposes.

device.inode is unique at a single point in time, but the same combination
can reappear later, after the first file is gone.

Rotated log files can get deleted eventually, and before that they can get
gzipped, which will result in a .gz file with a new inode number and the
original inode number becoming available for reuse.

It wouldn't be surprising to see /var/log/mail get the same inode number it
had 2 weeks ago, when it had the contents now present in /var/log/mail.2.gz

--
Alan Curry
First  |  Prev  | 
Pages: 1 2 3 4
Prev: FAQ 4.34 How do I extract selected columns from a string?
Next: checksum calculation for file offsets