[Mimedefang] MIME::Entity not handling Charset => 'utf-8' correctly?

[Philip Prindeville]

On Feb 21, 2013, at 3:36 AM, Steffen Kaiser <skmimedefang at smail.inf.fh-bonn-rhein-sieg.de> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Wed, 20 Feb 2013, Philip Prindeville wrote:
> 
> I suppose that violates the internals of the MIME:: and Mail:: namespace functions. They are tied together very closely.


Not sure I follow.  MIME:: can't have external dependencies?


> 
> Actually, I looked into a UTF8 MIMEtools a few years back to overcome character set problems when storing header data into a postgres database. I thought that everything the MIME:: functions should return would be in Perl utf8, any character set information already decoded. Anything the functions get passed into is Perl internal utf-8 as well. I think one would need to rewrite the whole framework anew.


It seems a reasonable goal to have all strings be stored in internal format, except for functions that explicitly generate "on-the-wire" formatted strings.

Could this also be controlled by either a global gating variable that controls a semantic, or by detecting if the caller has "utf8" loaded into this space?

Of course, things get complicated when MIME:: is used by some intermediate module (Foo:: for instance) which uses utf8 but the user's main' doesn't…



> 
> Encode is a tricky module by its own, perldoc Encode:
> 
> […]
> 
> Some encodings modify the $str argument to return the characters NOT decoded. So you'd call Encode::decode($charset, "".$str) to enforce a copy - - but have the performance penalty.


That seems to be an issue with Encode not having consistent semantics.  That might be fixed separately.

Encode is part of perl core, isn't it?


> 
> […]
> 
> I pass all strings through a function now, that looks terrible, but since then Web, Postgres, LDAP and text files play together.


And would fixing the semantics of Encode to be more uniform fix that, or is this an orthogonal issue?

-Philip