Win a copy of Svelte and Sapper in Action this week in the JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Paul Clapham
  • Bear Bibeault
  • Junilu Lacar
Sheriffs:
  • Jeanne Boyarsky
  • Tim Cooke
  • Henry Wong
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • salvin francis
  • Frits Walraven
Bartenders:
  • Scott Selikoff
  • Piet Souris
  • Carey Brown

logging for a multithreaded app which writes log billions of times a day may make disk full

 
Ranch Hand
Posts: 1665
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In multithreaded applications same run method would run so many times. If I provide loggin in it , it would write to log billions of time. Is it ok do give logging to it using log4j. I am asking this question because when it will write to logs so many times logs will grow at a very fast rate.Soon the log folder would be in GBs size.Disk will get full everyday.Please advice
 
Rancher
Posts: 43016
76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Billions of times a day implies more than 10000 entries per second. That sounds either like you're logging way too much, or that you have a seriously high end application. In the former case I advise to revisit the events you're logging, and to determine whether those log entries are actually needed. For example, for a method that's invoked thousands of times per second, is it worthwhile logging every single invocation? In the latter case you'll just have to grow your hardware with it, and decide how to handle log files. Typically you compress the log files at the end of a day and -if they are too large to store locally- move them to secondary storage, maybe Amazon S3 or Amazon Glacier. You cold also extract anything that you may want to keep, and discard the rest. And you may want to consider storing what you do want to keep in a DB instead a file system. In a case like this -where there are presumably many very similar log entries- compression may be able to reduce file size by about 2 orders of magnitude, which may move it into the realm of keeping them on primary storage long term.
 
Bartender
Posts: 689
17
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What exactly do you plan to do with billions of logs? It's likely to be unreadable and any useful information will be swamped in a sea of unwanted information.

Also consider the effect on performance. As always with performance don't optimism prematurely, but with a billion logs a day make sure you plan enough time for performance tuning.
 
author
Posts: 23883
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Monica. Shiralkar wrote:In multithreaded applications same run method would run so many times. If I provide loggin in it , it would write to log billions of time. Is it ok do give logging to it using log4j. I am asking this question because when it will write to logs so many times logs will grow at a very fast rate.Soon the log folder would be in GBs size.Disk will get full everyday.Please advice



Loggers, like log4j, has the concept of log levels. This allows you to control the amount of logging. For testing environments, you might set it to a more verbose level like DEBUG or INFO, so that you can get the lots of information. For production systems, you might want to run with ERROR or FATAL log levels, as you don't care about the majority of information.

Log4j also support the concept of a roll file. This way when the file gets too big, it rolls over. In effect, you only get the latest information -- and you don't run out of disk space as old stuff is automatically detected.

Henry
 
Monica Shiralkar
Ranch Hand
Posts: 1665
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

What exactly do you plan to do with billions of logs? It's likely to be unreadable and any useful information will be swamped in a sea of unwanted information.



I am deploying my application on production server. So I feel now I cannot keep System.out.println messages and I thought I should replace them with log.info. Is it possible that my application runs and does what it is supposed to do and there is no where to check if anything goes wrong. If anything goes wrong wont it that I will be asked that where is your logs to check what went wrong?

thanks
 
author & internet detective
Posts: 40200
816
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Monica. Shiralkar wrote:

What exactly do you plan to do with billions of logs? It's likely to be unreadable and any useful information will be swamped in a sea of unwanted information.



I am deploying my application on production server. So I feel now I cannot keep System.out.println messages and I thought I should replace them with log.info. Is it possible that my application runs and does what it is supposed to do and there is no where to check if anything goes wrong. If anything goes wrong wont it that I will be asked that where is your logs to check what went wrong?

thanks


Which means you should be more discerning about what you log. If there is a problem, you aren't going to go through billions of lines of information to figure out what went wrong. Look at which messages add value. Also, see if you can turn the logging on just when there is an error rather than always.
 
Mike. J. Thompson
Bartender
Posts: 689
17
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I wasn't suggesting that you shouldn't have logs. I was suggesting that billions of logs a day will not be useful because they will be hard to understand.
 
Monica Shiralkar
Ranch Hand
Posts: 1665
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Actually it is a multithreaded program which works on lakhs of files multiple times in a day.So the run method code will be called millions of times a day. If there are statements in it to log it will be called so many times. Please suggest that how to have a mechanism to know what went wrong if logging is not feasible?

thanks
 
Jeanne Boyarsky
author & internet detective
Posts: 40200
816
Eclipse IDE VI Editor Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What might go wrong that logging with help with? I'm serious. Try to write the answer to that question as it might suggest the answer.

Possibilities are to only log when an exception is thrown. Or to only log when certain data conditions occur.
 
Bartender
Posts: 4107
72
Hibernate Fedora Chrome
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
As already pointed out you should first decide what exactly you need to log (and will be useful in getting a solution in if something fails in production). And then you can decide what levels of logging they belong. Apparently it is not the case as you mention you have all those Sys.out to log different things.
 
Ulf Dittmer
Rancher
Posts: 43016
76
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Monica. Shiralkar wrote:So the run method code will be called millions of times a day. If there are statements in it to log it will be called so many times.


If a method runs that many times, then the information that it did so is likely of little value - so it should not be logged. Only if something goes wrong should you log the relevant information. So you need to have exception handlng in place that is is triggered if something goes wrong, and which logs all the pertinent information that you need to have in that case.
 
Monica Shiralkar
Ranch Hand
Posts: 1665
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think I will log only the e.printStacktrace information of each Catch statement.
Thanks
 
I didn't do it. You can't prove it. Nobody saw me. The sheep are lying! This tiny ad is my witness!
the value of filler advertising in 2020
https://coderanch.com/t/730886/filler-advertising
    Bookmark Topic Watch Topic
  • New Topic