Join lines that are encoded in quoted-printable format

Question

I have a giant mbox file (8.8 GB) that contains many quoted-printable attachments, consisting of 75-character lines terminated by the "soft linebreak" sequence (equal sign, carriage return, linefeed).

I would like to search for all occurrences of a particular regular expression in the mbox content. However, any particular match may be split among multiple lines, therefore I need to delete each soft linebreak sequence before performing the regex search.

I'm having trouble figuring out the best way to do this. The solution for GNU sed that I found here fails because it apparently instructs sed to treat the entire file as one line. That one line has length is greater than INT_MAX, which causes sed to exit with an error message. I also see that there is a solution here that uses ripgrep. However, this fails to actually join the lines; the equal signs are removed but the line with the equal sign remains separate from the following line.

Maybe you can open it with `mutt` and search in it with `l` (lower case L to `limit` the display) and search with `~b your-regexp` to search on the decoded contents? — Stéphane Chazelas, Apr 26 '23 at 16:23
What exactly means "search for all occurrences of a particular regular expression"? Display the lines that match the expression? Please add a piece of example input, the regular expression and the expected result. — Bodo, Apr 26 '23 at 16:23
google the Unix tools `formail` and `procmail` - one of them can probably help you. — Ed Morton, Apr 26 '23 at 21:33

score 2 · Accepted Answer · answered Apr 26 '23 at 17:28

In the end, what you're describing that you're doing is that you parse the mbox into email contents (towards which I count their attachments), and then look through those. Proper approach!

So, do that: parse the mbox file actually as mails, and you'll be somewhat happy. Here's my top-of-head, cobbled-together, untested but not red in my editor code:

#!/usr/bin/env python3
import mailbox
import re as regex

pattern = regex.compile("BRIANSREGEX")

mb = mailbox.mbox("Briansmails.mbox", create=False)
for msg in mb:
    print(f"{msg['Subject']}")
    for part in msg.walk():
        print(f"| {part.get_content_type()}, {part.get_content_charset()}")
        print("-"*120)
        payload = part.get_payload(decode=True)
        charset = part.get_content_charset
        if type(payload) is bytes:
            content = ""
            try:
                content = payload.decode(charset)
            except:
                # print("failed to decode")
                try:
                    content = payload.decode() # try ascii
            match = pattern.search(content)
            # do what you want with that match…
            if match:
                print(f"| matched at {match.start()}")
        print("-"*120)

There are some errors here (`get_content_charset` is missing the `()` needed to call it, and we must take into account that it might return `None`) but this is a good approach overall; I realized that it also matches inside base64 attachments, not just quoted-printable ones. — Brian Bi, Apr 27 '23 at 01:54
In addition, if `part.is_multipart()` is true, then attempting to get the payload should just be skipped (it will return `None` anyway). — Brian Bi, Apr 27 '23 at 13:48
@BrianBi as said, that was just quickly written down without any testing or sense. — Marcus Müller, Apr 27 '23 at 21:03

score 0 · Answer 2 · answered Apr 29 '23 at 18:18

I've given Marcus Müller credit for his answer, but here's the version that I ultimately ended up using, which is based on his answer:

#!/usr/bin/env python3
import mailbox
import re
import sys

byte_pattern = re.compile(b"https?://[^/]*imgur.com/[a-zA-Z0-9/.]*")
str_pattern = re.compile("https?://[^/]*imgur.com/[a-zA-Z0-9/.]*")

mb = mailbox.mbox(sys.argv[1], create=False)
for msg in mb:
    for part in msg.walk():
        if part.is_multipart():
            continue
        payload = part.get_payload(decode=True)
        if type(payload) is bytes:
            # first, search it as a binary string
            for match in byte_pattern.findall(payload):
                print(match.decode('ascii'))
            # then, try to decode it in case it's utf-16 or something weird
            charset = part.get_content_charset()
            if charset and charset != 'utf-8':
                try:
                    content = payload.decode(charset)
                    for match in str_pattern.findall(content):
                        print(match)
                except:
                    pass
        else:
            print('failed to get message part as bytes')

Note that part may be a multipart message, i.e., not a leaf node in the tree, because walk performs a depth-first traversal that includes non-leaf nodes.

Only if it's a leaf node, I first search for my pattern as a byte string, then attempt to decode it as text using the indicated charset if it is not UTF-8. (If it is UTF-8, which is most common, it will have been found as a byte string already.)

Join lines that are encoded in quoted-printable format

2 Answers2