26

I want to log some information of every single request sent to a busy http server in a formatted form. Using the logging module would create some thing I don't want to:

[I 131104 15:31:29 Sys:34]

I thought of the CSV format, but I don't know how to customize it. Python has the csv module, but I read in the manual

import csv
with open('some.csv', 'w', newline='') as f:
    writer = csv.writer(f)
    writer.writerows(someiterable)

Since it would open and close a file each time, I am afraid this way would slow down the whole server performance. What could I do?

Improve this question
1
  • 3
    You should use a logging.Formatter instance with a format that outputs csv lines.
    – Pedro Werneck
    Commented Nov 4, 2013 at 10:00

4 Answers 4

Reset to default
28

Just use python's logging module.

You can adjust the output the way you want; take a look at Changing the format of displayed messages:

To change the format which is used to display messages, you need to specify the format you want to use:

import logging
logging.basicConfig(format='%(levelname)s:%(message)s', level=logging.DEBUG)
logging.debug('This message should appear on the console')
logging.info('So should this')
logging.warning('And this, too')

and Formatters:

Formatter objects configure the final order, structure, and contents of the log message.

You'll find a list of the attribtus you can use here: LogRecord attributes.

If you want to produce a valid csv-file, use python's csv module, too.

Here's a simple example:

import logging
import csv
import io

class CsvFormatter(logging.Formatter):
    def __init__(self):
        super().__init__()
        self.output = io.StringIO()
        self.writer = csv.writer(self.output, quoting=csv.QUOTE_ALL)

    def format(self, record):
        self.writer.writerow([record.levelname, record.msg])
        data = self.output.getvalue()
        self.output.truncate(0)
        self.output.seek(0)
        return data.strip()

logging.basicConfig(level=logging.DEBUG)

logger = logging.getLogger(__name__)
logging.root.handlers[0].setFormatter(CsvFormatter())

logger.debug('This message should appear on the console')
logger.info('So should "this", and it\'s using quoting...')
logger.warning('And this, too')

Output:

"DEBUG","This message should appear on the console"
"INFO","So should ""this"", and it's using quoting..."
"WARNING","And this, too"

Improve this answer
2
  • 1
    It took me 2 hours to realize that logging.basicConfig(level=logging.DEBUG) is necessary for logging for instance a INFO level log... even when you set propagate=False to the logger.
    – Niloct
    Commented Jan 25, 2020 at 5:37
  • How do you add asctime to the CsvFormatter?
    – ramiwi
    Commented Jan 6, 2023 at 18:23
7

As sloth suggests, you can easily edit the delimiter of the log to a comma, thus producing a CSV file.

Working example:

import logging

# create logger
lgr = logging.getLogger('logger name')
lgr.setLevel(logging.DEBUG) # log all escalated at and above DEBUG
# add a file handler
fh = logging.FileHandler('path_of_your_log.csv')
fh.setLevel(logging.DEBUG) # ensure all messages are logged to file

# create a formatter and set the formatter for the handler.
frmt = logging.Formatter('%(asctime)s,%(name)s,%(levelname)s,%(message)s')
fh.setFormatter(frmt)

# add the Handler to the logger
lgr.addHandler(fh)

# You can now start issuing logging statements in your code
lgr.debug('a debug message')
lgr.info('an info message')
lgr.warn('A Checkout this warning.')
lgr.error('An error writen here.')
lgr.critical('Something very critical happened.')
Improve this answer
3
  • 3
    is there a way to add a CSV Header Row? (i.e. a first row in the CSV Text File containing the names of the columns?)
    – ycomp
    Commented Sep 10, 2015 at 6:27
  • Aye, this might be what you're looking for: stackoverflow.com/questions/27840094/…
    – ecoe
    Commented Sep 11, 2015 at 11:48
  • 4
    This solution is not sufficiently robust—what happens when a comma is output in asctime? What if the message itself contains a comma or a newline? The CSV file is then broken. Anything writing CSV data should be doing so through a csv.writer instance, as shown in some of the other answers.
    – Alex Peters
    Commented Jul 18, 2019 at 11:40
6

I would agree that you should use the logging module, but you can't really do it properly with just a format string like some of the other answers show, as they do not address the situation where you log a message that contains a comma.

If you need a solution that will properly escape any special characters in the message (or other fields, I suppose), you would have to write a custom formatter and set it.

logger = logging.getLogger()

formatter = MyCsvFormatter()

handler = logging.FileHandler(filename, "w")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(level)

You'll obviously have to implement the MyCsvFormatter class, which should inherit from logging.Formatter and override the format() method

class MyCsvFormatter(logging.Formatter):
    def __init__(self):
        fmt = "%(levelname)s,%(message)s" # Set a format that uses commas, like the other answers
        super(MyCsvFormatter, self).__init__(fmt=fmt)

    def format(self, record):
        msg = record.getMessage()
        # convert msg to a csv compatible string using your method of choice
        record.msg = msg
        return super(MyCsvFormatter, self).format(self, record)

Note: I've done something like this before, but haven't tested this particular code sample

As far as doing the actual escaping of the message, here's one possible approach: Python - write data into csv format as string (not file)

Improve this answer
0
0

I don't think that is the best idea, but it is doable, and quite simple. Manually buffer your log. Store log entries in some place, and write them to file from time to time. If you know that your server will be constantly busy, flush your buffer when it reaches some size. If there may be big gaps in usage, I'd say that new thread (or better process, check yourself why threads suck and slow down apps) with endless (theoretically of course) loop of sleep/flush would be better call. Also, remember to create some kind of hook that will flush buffer when server is interrupted or fails (maybe signals? or just try/except on main function - there are even more ways to do it), so you don't lose unflushed buffer data on unexpected exit.

I repeat - this is not the best idea, it's the first thing that came to my mind. You may want to consult logging implementations from Flask or some other webapp framework (AFAIR Flask has CSV logging too).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.