How to keep a log file from growing too big

[henryvuong]

I have written a Windows Service program with C# .NET. For every task this program executes, it will write an entry into a log file (.txt format) with a datetime stamp. Everything works fine, but my concern is that the log file can eventually grow too big without my attention. I need to find a way to keep this log file at certain size. Some of the methods I am thinking are:

• Every time the program writes a new entry, it will check and delete entries that are older than a certain number of days (say 90 days). This is my prefer method.
• OR: When the log file grows to a certain size (say 100 MB), chop off half of the entries from the oldest first.
• OR: When the log file grows to a certain size, erase all entries and start fresh. This is not my prefer method, but if nothing else works, it can be a solution.

 

[Skydiver]

Your first choice should be to use a pre-built logging solution. Log4Net would be my recommendation, but some people like SeriLog, Elmah, or others.

If you really must stick with your own implementation, the most efficient solution is to just start a new file, and leave the old file alone. Or rename the current file, and then start a new file. This still leaves you with the problem of eating up disk space, but you can efficiently delete the older files if they are older than a particular age, by simply looking at the file stamps.

The first two of your options are going to be very inefficient because you would need to read in the entire file, parse each line, and the rewrite the entries that you want to keep. (Recall that you can't just delete lines from a text file without having to rewrite the file.) Imagine that happening when you are trying to log as fast as you can because you are calling your logging function from within a tight loop.

The last of your options, can be done quickly and efficiently by using the file operation mode to open a file for writing and truncate the existing contents. But as you said, this isn't really a great option since you'll lose half of your log data on average.

 

[jmcilhinney]

In my office, we've used Elmah and Log4Net before but we currently use NLog by default. As described above, it automatically uses the current date in the file name, so it will automatically create a new file each day. You can the create a scheduled task to delete files older than a particular age or do it manually from time to time.

 

[kareninstructor]

With SeriLog, write to a new file each day and if desire remove old folders after a period of time. Also, SeriLog supports writing to a database so as with writing to a file you can write a job to remove old records. See the following GitHub repository for various samples and best practices.

 

[jmcilhinney]

Also, SeriLog supports writing to a database so as with writing to a file you can write a job to remove old records.

It is a good idea to use a third-party, tried-and-tested logging tool for this reason. You can add extra or alternative targets with little change. Using NLog, we usually log to files but we recently deployed a project that logged to Azure blob storage. All it required was a line or two changed in startup code or a config file.

 

[henryvuong]

Thanks for all of your suggestion. I will look into each solution and find the best fit for me.

 

[jmcilhinney]

Note that most (all?) of those logging frameworks can be installed from NuGet packages.