Questions tagged [log-files]

A log file is a file, often in clear text, which is generated by a software process to record some specific details about the workings of this process.

A log file is a file which is generated by a software process to record some specific details (events) about the workings of this process. Most logfiles are plain, human-readable text files, some however are using a binary format to conserve space and need specific treatment for post-processing. (Example: Unix login history in utmp/wtmp format)

A log file can be categorized into several types:

  1. Access log which records information related to the usage of the service provided by this process.
  2. Error log which records error conditions.
  3. Info/Debug log which records more detailed information as compared with other logs.
  4. Transaction log to document a chain of events, this often is required to be immutable to avoid later tampering.
  5. redo logs, containing the complete information to replay a chain of actions, often used where incomplete data transactions would lead to inconsistent data, for example in file systems or databases.

The main two goals of creating and maintaining log files are:

  1. Finding and debugging problems.
  2. Gathering usage information.

Depending on the frequence of events and the level of detail, logfiles may be growing very quickly, and usually require a strategy for managing their growth.

Depending on special reqirements, log-files may go to specialized devices to meet criteria of speed or immutability.

1108 questions
13
votes
9 answers

Anyone know of a tool to detect and report on repeating patterns in a log file?

I need to monitor some large noisy log files (500m/day) from a Java application (log4j). Right now I manually look at the files, grep for "ERROR" and so on. However it should be possible for a tool to spot repeating patterns in the file, count them…
7
votes
2 answers

Cloud mapping experiment. Contact research@pdrlabs.net in the access logs

54.204.131.75 - - [09/Jul/2014:17:53:43 +0100] "HEAD / HTTP/1.1" 200 283 "-" "Cloud mapping experiment. Contact research@pdrlabs.net" A new line in my access log the other day. As far as i can tell this is most likely a phishing scam. Does anyone…
John
  • 897
5
votes
2 answers

Using multitail to display all new log files recursively

I'm trying to get multitail (6.0) on Ubuntu to tail -f any log files (even new ones) in certain directories recursively by doing: multitail -Q 1 /home/default/storage/accounts/**/*.log This just produces a blank screen. So I tried being more…
eComEvo
  • 1,041
3
votes
1 answer

Clearing a log file owned by root

I need to clear a log file with: cat /dev/null > error.log The problem is that the file is owned by root -rw-r--r-- 1 root root 15505 Feb 9 15:08 error.log and this doesn't work $ sudo cat /dev/null > error.log -bash: error.log: Permission…
davidhq
  • 215
3
votes
6 answers

Thoughts on Apache log file sizes?

Do you place any limits on the size of Apache log files - access.log and error.log? Specifically, can you give: Reasons to limit log file sizes Disk space Any other? Reasons NOT to limit log file sizes Research into performance issues or…
Nathan Long
  • 1,545
  • 7
  • 22
  • 38
2
votes
4 answers

sensible logs policy & tools

What is a sensible logs policy? On one hand I would like to keep everything forever. On the other hand I don't want to waste time in administrative tasks and must avoid disks getting full in production servers. What is a sensible logs policy? What…
flybywire
  • 617
2
votes
4 answers

best way to traverse long logs on server

I am ssh'd into a server and want to traverse through long log (5000 lines). what is the best command to do this? I tried tail -2000 server.log but then I have to scroll up my SSH window....which only goes up a little. I cant install anything new…
Omnipresent
1
vote
1 answer

Do I have to restart my server every time I clear my nginx log file?

On nginx: if I open the error log, select all (in emacs) and cut, save, php errors stop logging to that file completely. If I restart nginx, they begin logging again. If I delete the files line-by-line (Control+K), the errors still save as expected.…
1
vote
2 answers

understand apache error log (maybe an attack?)

anyome can help to understand the apache error log? my website has benn recently attacked (iframe at bottom page). my website in ona dedicated server with centos ad directadmin than you all. [Tue Jan 31 01:08:11 2012] [error] [client 157.55.16.230]…
1
vote
2 answers

Request for recommendations for logfile analyer / troubleshooting tool

I generally use google analytics, but occassionally I need to examine log files, for example to see how many 301 redirects from an old website are issued to real user-agents (rather than robots) Can anyone recommend a good tool for doing ad hoc…
Myster
  • 223
  • 3
  • 9
1
vote
1 answer

Is there a HTOP log

is there a log for HTOP - i think I accidentally killed something and things are going weird
P P
  • 11
  • 1
  • 3
1
vote
2 answers

How can filter certain columns in my logfile in multitail not to output?

I recently learned about awk which has the simply ability to filter or output certain columns of a logfile with the print method. Is there also an option how to suppress the output of certain columns in multitail and if yes how to do? I need it to…
NES
  • 215
1
vote
1 answer

Log file for outgoing PHP requests (like access.log)

Is there a way to log all outgoing requests of PHP to a log file? I'm not looking for a way to debug a single script with tcpdump but to continuously log all PHP requests, so I can review them later - i.e., very similar to the web servers access.log…
Philipp
  • 128
0
votes
2 answers

Are there any regularly updated IP databases of Bots?

I am looking for a database of IP Addresses of known bots, so I can identify them in web site traffic. I have useragent data, but there seem to be plenty that don't give useragents to identify themselves. These tend to be the spambots. I know…
0
votes
1 answer

How to understand these lines in apache.log

I just get 19000 lines like these in the apache.log file for my site example.com. My hosting provider shut down the hosting and notified me that I need to avoid to activate my hosting again. I understand that I got a big amount of visits but I…
1
2