Arabic OTRS encoding

List overview All Threads
Download

newer

older

Re: [Wikitech-l] [MediaWiki-CVS]...

Captcha readibility

Muhammad Alsebaey

30 Sep 2007 30 Sep '07

5:54 p.m.

Hi All, I just joined the OTRS team a short time ago but I noticed a chronic problem with the Arabic OTRS system, most of the messages in Arabic come to us encoded as Windows-1256 and not UTF-8 unfortunately (probably because MS Outlook, the prevalent client in Arab-speaking countries encodes them as such) , therefore to read them, we have to switch to plain view then change the encoding from the browser to read the mail, this is a minor inconvenience, the big problem however is that when we send them back a response through the system, users receive it as gibberish (I am guessing it is UTF-8 encoded). I have seen a lot of users complain in the short time I have been on OTRS that we are sending out unintelligible messages as response to them. Does this have a solution? how does OTRS handles encoding in general? do volunteers for other languages have the same problem? -- Best Regards, Muhammad Alsebaey

Show replies by date

jf＠mormo.org

30 Sep 30 Sep

6 p.m.

On Sun, Sep 30, 2007 at 07:54:20PM +0200, Muhammad Alsebaey wrote:

...

The problem is that the problematic messages that I've seen in the past were encoded as Windows-1256, but had a Content-Encoding: UTF-8 mail header. Please send me a few ticket IDs, so that I can check whether that's the case there. It would further be helpful to get in contact with a user who sees our responses as gibberish, so that we can investigate the reason for the problems. Regards, jens

Mido

7:27 p.m.

as Cary noted to me before, the problem is that the messages being sent from Yahoo mail it's sent in charset=iso-8859-1 and OTRS system can't resolve it as it seems like this one #2007092610005435 I couldn't find other mail servers than Yahoo that has this problem when mail is sent from another server even with charset="windows-1256" encoding like this one #2007093010019191, it render correctly in the queue view (fails in plain version however) so, the main problem lies within the iso-88589-1, will this be difficult to solve? Thanks, Mido On 9/30/07, Jens Frank <jf(a)mormo.org> wrote:

...

On Sun, Sep 30, 2007 at 07:54:20PM +0200, Muhammad Alsebaey wrote:

Hi All, I just joined the OTRS team a short time ago but I noticed a chronic

problem

with the Arabic OTRS system, most of the messages in Arabic come to us encoded as Windows-1256 and not UTF-8 unfortunately (probably because

Outlook, the prevalent client in Arab-speaking countries encodes them as such) , therefore to read them, we have to switch to plain view then

change

the encoding from the browser to read the mail, this is a minor inconvenience, the big problem however is that when we send them back a response through the system, users receive it as gibberish (I am

guessing it

is UTF-8 encoded). I have seen a lot of users complain in the short time

have been on OTRS that we are sending out unintelligible messages as response to them. Does this have a solution? how does OTRS handles

encoding

in general? do volunteers for other languages have the same problem?

<http://meta.wikimedia.org/wiki/Wikimania_2008/Bids/Alexandria>

Muhammad Alsebaey

9:18 p.m.

Actually, I just found out you are correct, I just scanned the messages and almost all problematic ones come from Yahoo mail :) , however it seems like more than 75% of our traffic on the Arabic queue comes from Yahoo? that's why I had the impression it is a general problem. The ticket Mido posted has someone who replied back that he received gibberish. Another one who actually mailed in English and I replied in Arabic and he received gibberish is 2007091010002735, I had to resend the reply in English to close. Here are some others (as I said, most of the queue is from Yahoo): 2007092710012265 2007092810013833 2007092910019264 (this one is from yahoo.fr) One note though, when I change the encoding to Windows-1256 in the plain view I can read it, that shouldnt be the case if the encoding is ISO-8859-1? On 9/30/07, Mido <mido.architect(a)gmail.com> wrote:

...

On Sun, Sep 30, 2007 at 07:54:20PM +0200, Muhammad Alsebaey wrote:

Hi All, I just joined the OTRS team a short time ago but I noticed a chronic

problem

with the Arabic OTRS system, most of the messages in Arabic come to us encoded as Windows-1256 and not UTF-8 unfortunately (probably because

MS > Outlook, the prevalent client in Arab-speaking countries encodes them

such) , therefore to read them, we have to switch to plain view then

change > the encoding from the browser to read the mail, this is a minor > inconvenience, the big problem however is that when we send them back

response through the system, users receive it as gibberish (I am

guessing it > is UTF-8 encoded). I have seen a lot of users complain in the short

time

have been on OTRS that we are sending out unintelligible messages as response to them. Does this have a solution? how does OTRS handles

encoding

in general? do volunteers for other languages have the same problem?

<http://meta.wikimedia.org/wiki/Wikimania_2008/Bids/Alexandria> _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Best Regards, Muhammad Alsebaey

Brion Vibber

1 Oct 1 Oct

2:42 a.m.

Muhammad Alsebaey wrote:

...

[snip]

...

One note though, when I change the encoding to Windows-1256 in the plain view I can read it, that shouldnt be the case if the encoding is ISO-8859-1?

There are two parts to the problem: 1) The original mailer marks the mail with the wrong encoding 2) OTRS believes the incorrect header and thus converts the text to Unicode incorrectly for web output. As a result, you can't change the encoding setting of the web browser to fix it. However, OTRS also does its 'plain view' incorrectly by outputting the raw original bytes instead of converting to Unicode to match the surrounding page -- which has the convenient property that you *can* change the encoding setting in the browser. So I'd recommend using that as a workaround for the moment until we figure out a way to patch up OTRS to handle it better. [snip]

...

The ticket Mido posted has someone who replied back that he received gibberish.

That might be harder to deal with... :( -- brion vibber (brion @ wikimedia.org)

Muhammad Alsebaey

7 Oct 7 Oct

8:17 p.m.

I can live with changing the encoding myself to read the message, is there anything that can be done though about the garbled reply that the user receives? Do I need to post a bug somewhere? on otrs.org maybe? On 10/1/07, Brion Vibber <brion(a)wikimedia.org> wrote:

...

Muhammad Alsebaey wrote:

Actually, I just found out you are correct, I just scanned the messages

and

almost all problematic ones come from Yahoo mail :) , however it seems

more than 75% of our traffic on the Arabic queue comes from Yahoo?

that's

why I had the impression it is a general problem.

[snip]

One note though, when I change the encoding to Windows-1256 in the plain view I can read it, that shouldnt be the case if the encoding is

ISO-8859-1? There are two parts to the problem: 1) The original mailer marks the mail with the wrong encoding 2) OTRS believes the incorrect header and thus converts the text to Unicode incorrectly for web output. As a result, you can't change the encoding setting of the web browser to fix it. However, OTRS also does its 'plain view' incorrectly by outputting the raw original bytes instead of converting to Unicode to match the surrounding page -- which has the convenient property that you *can* change the encoding setting in the browser. So I'd recommend using that as a workaround for the moment until we figure out a way to patch up OTRS to handle it better. [snip]

The ticket Mido posted has someone who replied back that he received gibberish.

That might be harder to deal with... :( -- brion vibber (brion @ wikimedia.org) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Best Regards, Muhammad Alsebaey

Brion Vibber

9 Oct 9 Oct

2:49 p.m.

Muhammad Alsebaey wrote:

...

Muhammad Alsebaey

10 Oct 10 Oct

5:42 p.m.

I made a dummy test account on yahoo: shipmaster3 @ yahoo.com , we can use that. On 10/9/07, Brion Vibber <brion(a)wikimedia.org> wrote:

...

Muhammad Alsebaey wrote:

I can live with changing the encoding myself to read the message, is

there

anything that can be done though about the garbled reply that the user receives? Do I need to post a bug somewhere? on otrs.org maybe?

We might need to add a little hack in OTRS; it'll help if someone with an account on the affected e-mail service can confirm the issue w/ replies and confirm a fix. Can you provide a contact for testing? -- brion vibber (brion @ wikimedia.org) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Best Regards, Muhammad Alsebaey

Muhammad Alsebaey

12 Oct 12 Oct

12:25 a.m.

Ok testing using the newly created yahoo account, ticket 2007101210000472, I replied and received an unreadable email on Yahoo mail. The mail I rec On 10/10/07, Muhammad Alsebaey <muhammad.alsebaey(a)ontometrics.com> wrote:

...

I made a dummy test account on yahoo: shipmaster3 @ yahoo.com , we can use that. On 10/9/07, Brion Vibber < brion(a)wikimedia.org> wrote:

Muhammad Alsebaey wrote:

I can live with changing the encoding myself to read the message, is

there

anything that can be done though about the garbled reply that the user receives? Do I need to post a bug somewhere? on otrs.org maybe?

We might need to add a little hack in OTRS; it'll help if someone with an account on the affected e-mail service can confirm the issue w/ replies and confirm a fix. Can you provide a contact for testing? -- brion vibber (brion @ wikimedia.org ) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/wikitech-l

-- Best Regards, Muhammad Alsebaey

Mohamed Magdy

1 Oct 1 Oct

3:50 a.m.

Will otrs change the encoding of the outgoing message if we change the encoding to win56 (or whatever) in the send form?

6055

days inactive

6067

days old

wikitech-l@lists.wikimedia.org

Manage subscription

9 comments

6 participants

tags (0)

participants (6)

Brion Vibber
jf＠mormo.org
Mido
Mohamed Magdy
Muhammad Alsebaey
Muhammad Alsebaey