25 Feb
25 Feb
4:23 p.m.
So, after staring at much Python and sending dozens of e-mail messages
to myself, here's what I've learned about sending non-ASCII characters
through Mailman:
The fundamental problem is that Python's smtplib.sendmail() function
throws an exception if non-ASCII characters are present in the e-mail
body.
Mailman therefore executes msgtext = msgtext.encode('ascii',
'replace').decode('ascii') before passing anything to smtplib. This is
what (I surmise) converts right-apostrophes, Æ's, and the like into
question marks.
Normal e-mail clients don't trigger the conversion because they
variously encode non-ASCII characters using quoted printable encoding
(=36=86 etc), convert the non-ASCII characters into HTML entities, or
encode the message as a base64 MIME attachment. I think Mailman is
trying to do the last of these but my e-mail client (at least) displays
the ASCII-ified part of the message rather than the attachment.
On the other hand, PHP's mail() function simply sends whatever bytes the
programmer gives it. This works as long as your SMTP server accepts
UTF-8 characters, but, as already discussed, Python doesn't.
I sent the message below using
$to = "selenetest(a)lochac.sca.org".org";
$from = "aelfred(a)nps.id.au"d.au";
$subject = "Sea Dragons Picnic";
$body = "Testing <i>Sea Dragon’s Picnic</i> with an apostrophe, from
Ælfred.";
$header = "From: $from\r\n" .
"Content-Type: text/html; charset=utf-8\r\n" .
"Content-Transfer-Encoding: quoted-printable";
mail($to, $subject, quoted_printable_encode($body), $header);
where quoted_printable_encode() is part of the standard PHP library.
When we first tested Mailman3, I recall that some webmail clients (e.g.
Gmail) had trouble with the quoted-printable characters. The message
below appears correctly in Roundcube, though.
The other two approaches I know of are:
Convert non-ASCII characters to HTML entitites. This is what Roundcube
does, taking advantage of a function provided by the TinyMCE editor
(Javascript). I don't know of any ready-made PHP function that will do
this.
Encode the message body with Base64 and set Content-Transfer-Encoding:
base64. I haven't tested this.
Finally, all of this applies only to the e-mail body. The subject line
can't contain raw UTF-8 characters because most SMTP servers reject
them. Messages I've sent from my desktop client use a kind of
quoted-printable encoding that looks like this:
Subject: [Selenetest]
=?utf-8?q?Message_with_an_=C3=86_in_it?=
...but I haven't looked into what the relevant standard is or what PHP
functions might produce such a thing.
Ælfred
------ Original Message ------
From "aelfred--- via Selenetest" <selenetest(a)lochac.sca.org>
To selenetest(a)lochac.sca.org
Date 25/02/2023 3:41:20 PM
Subject [Selenetest] Sea Dragons Picnic
Testing Sea Dragon’s Picnic with an apostrophe, from
Ælfred.
8:33 p.m.
Great detective work!
I had a look into the subject encoding, and it seems to align with
https://www.rfc-editor.org/info/rfc2047 - as you pointed out, similar to
quoted-printable.
Looks like PHP has both
https://www.php.net/manual/en/function.iconv-mime-encode.php and
https://www.php.net/manual/en/function.mb-encode-mimeheader.php, though the
latter looks a little easier to use.
I will give this a test!
Cheers,
Dylan
On Sat, 25 Feb 2023, 06:24 Ælfred se leof via Selenetest, <
selenetest(a)lochac.sca.org> wrote:
So, after staring at much Python and sending dozens of
e-mail messages to
myself, here's what I've learned about sending non-ASCII characters through
Mailman:
1. The fundamental problem is that Python's smtplib.sendmail()
function throws an exception if non-ASCII characters are present in the
e-mail body.
2. Mailman therefore executes msgtext = msgtext.encode('ascii',
'replace').decode('ascii') before passing anything to smtplib.
This is what
(I surmise) converts right-apostrophes, Æ's, and the like into question
marks.
3. Normal e-mail clients don't trigger the conversion because they
variously encode non-ASCII characters using quoted printable encoding
(=36=86 etc), convert the non-ASCII characters into HTML entities, or
encode the message as a base64 MIME attachment. I think Mailman is trying
to do the last of these but my e-mail client (at least) displays the
ASCII-ified part of the message rather than the attachment.
4. On the other hand, PHP's mail() function simply sends whatever
bytes the programmer gives it. This works as long as your SMTP server
accepts UTF-8 characters, but, as already discussed, Python doesn't.
I sent the message below using
$to = "selenetest(a)lochac.sca.org".org";
$from = "aelfred(a)nps.id.au"d.au";
$subject = "Sea Dragons Picnic";
$body = "Testing <i>Sea Dragon’s Picnic</i> with an apostrophe, from
Ælfred.";
$header = "From: $from\r\n" .
"Content-Type: text/html; charset=utf-8\r\n" .
"Content-Transfer-Encoding: quoted-printable";
mail($to, $subject, quoted_printable_encode($body), $header);
where quoted_printable_encode() is part of the standard PHP library.
When we first tested Mailman3, I recall that some webmail clients (e.g.
Gmail) had trouble with the quoted-printable characters. The message below
appears correctly in Roundcube, though.
The other two approaches I know of are:
1. Convert non-ASCII characters to HTML entitites. This is what
Roundcube does, taking advantage of a function provided by the TinyMCE
editor (Javascript). I don't know of any ready-made PHP function that will
do this.
2. Encode the message body with Base64 and set
Content-Transfer-Encoding: base64. I haven't tested this.
Finally, all of this applies only to the e-mail body. The subject line
can't contain raw UTF-8 characters because most SMTP servers reject them.
Messages I've sent from my desktop client use a kind of quoted-printable
encoding that looks like this:
Subject: [Selenetest] =?utf-8?q?Message_with_an_=C3=86_in_it?=
...but I haven't looked into what the relevant standard is or what PHP
functions might produce such a thing.
Ælfred
------ Original Message ------
From "aelfred--- via Selenetest" <selenetest(a)lochac.sca.org>
To selenetest(a)lochac.sca.org
Date 25/02/2023 3:41:20 PM
Subject [Selenetest] Sea Dragons Picnic
Testing *Sea Dragon’s Picnic* with an apostrophe, from Ælfred.
_______________________________________________
Selenetest mailing list -- selenetest(a)lochac.sca.org
To unsubscribe send an email to selenetest-leave(a)lochac.sca.org
or manage your subscription via the web interface at
https://mailman.lochac.sca.org/postorius/lists/selenetest.lochac.sca.org
430
days inactive
430
days old
2 comments
3 participants
participants (3)
-
aelfred＠lochac.nps.id.au -
Dylan Kerr -
Ælfred se leof