offlineimap - unicode decode errors
My main system is currently running Ubuntu 21.04. For e-mail I’m relying on neomutt together with offlineimap, which both are amazing tools. Recently offlineimap was updated/moved to offlineimap3. Looking on my system, offlineimap reports itself as OfflineIMAP 7.3.0
and dpkg tells me it is version 0.0~git20210218.76c7a72+dfsg-1
.
Unicode Decode Error problem
Today I noticed several errors in my offlineimap sync log. Basically the errors looked like this:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 1299: invalid start byte
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xeb in position 1405: invalid continuation byte
Solution
If you encounter it as well (and you use mutt or neomutt), please have a look at this great comment on Github from Joseph Ishac (jishac) since his tip solved the issue for me.
To “fix” this issue for future emails, I modified my .neomuttrc and commented out the default send encoding charset and omitted the iso-8859-1 part:
#set send_charset = "us-ascii:iso-8859-1:utf-8"
set send_charset = "us-ascii:utf-8"
Then I looked through the email files on the filesystem and identified the ISO-8859 encoded emails in the Sent folder which are causing the current issues:
$ file * | grep "ISO-8859"
1520672060_0.1326046.desktop,U=65,FMD5=7f8c0215f16ad5caed8e632086b81b9c:2,S: ISO-8859 text, with very long lines
1521626089_0.43762.desktop,U=74,FMD5=7f8c02831a692adaed8e632086b81b9c:2,S: ISO-8859 text
1525607314.R13283589178011616624.desktop:2,S: ISO-8859 text
That left me with opening the files with vim and saving them with the correct encoding:
:set fileencoding=utf8
:wq
Voila, mission accomplished:
$ file * | grep "UTF-8"
1520672060_0.1326046.desktop,U=65,FMD5=7f8c0215f16ad5caed8e632086b81b9c:2,S: UTF-8 Unicode text, with very long lines
1521626089_0.43762.desktop,U=74,FMD5=7f8c02831a692adaed8e632086b81b9c:2,S: UTF-8 Unicode text
1525607314.R13283589178011616624.desktop:2,S: UTF-8 Unicode text