Unserialize problem on charset conversion

jaytraxx 2009-10-13 11:48:56 UTC #1

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in setup with a database export and re-import it with the new system charset.

It’s working mostly fine. The only problem comes up with serialized arrays (e.g. preferences) which include special characters, like e.g. the € (EUR) sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid changing all the serialized fields by myself. Maybe there is a possibility to adjust the value length somehow on re-import time.

Greetings
christian

Leithoff_Klaus 2009-10-13 12:41:41 UTC #2

in class config, around line 218 you see that value is unserialzed.
if it is empty, you may want to try an unserialize of an "backconverted"
string.
unserialize(iconv(“UTF-8-”,“ISO-8859-1”,$value))
that way the array may survive the first conversion.
after that, you may try an array_map and convert back to utf8 (I never tryed
that, but …).
Thats only for config, but in general that may work for preferences as well.
I dont have that in running code only for upgrade/conversion needs

jaytraxx 2009-10-13 12:50:55 UTC #3

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’, “‘s:’.strlen(’$2’).’:”$2";’", $serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving the string to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in setup with a database export and re-import it with the new system charset.

It’s working mostly fine. The only problem comes up with serialized arrays (e.g. preferences) which include special characters, like e.g. the € (EUR) sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid changing all the serialized fields by myself. Maybe there is a possibility to adjust the value length somehow on re-import time.

Greetings
christian

Leithoff_Klaus 2009-10-13 12:53:09 UTC #4

Sounds better, try.

Ralf_Becker_6 2009-10-13 12:58:39 UTC #5

If you include that in the restore, I think many users will be happy.

Ralf

jaytraxx schrieb:

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’, “‘s:’.strlen(’$2’).’:”$2";’",
$serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving the string
to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx wrote:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in
setup with a database export and re-import it with the new system charset.

It’s working mostly fine. The only problem comes up with serialized arrays
(e.g. preferences) which include special characters, like e.g. the € (EUR)
sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed
on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t
change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid
changing all the serialized fields by myself. Maybe there is a possibility
to adjust the value length somehow on re-import time.

Greetings
christian

–
Ralf Becker
Director Software Development

Stylite GmbH
[open style of IT]

Morschheimer Strasse 15
67292 Kirchheimbolanden

fon +49 (0) 6352 70629-0
fax +49 (0) 6352 70629-30
mailto: rb@stylite.de

www.stylite.de
www.egroupware.org

Geschäftsführer Andre Keller,
Gudrun Müller, Ralf Becker
Registergericht Kaiserslautern HRB 30575
Umsatzsteuer-Id / VAT-Id: DE214280951

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

jaytraxx 2009-10-13 13:01:48 UTC #6

tried, but it doesn’t work
I’ll try further… maybe it’s getting better …

christian

Leithoff:

Sounds better, try.

----------------ursprüngliche Nachricht-----------------
Von: “jaytraxx” christian@jaytraxx.de
An: egroupware-developers@lists.sourceforge.net
Datum: Tue, 13 Oct 2009 05:50:56 -0700 (PDT)

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.strlen(’$2’).’:"$2";’",
$serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving the
string
to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx wrote:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in
setup with a database export and re-import it with the new system
charset.

It’s working mostly fine. The only problem comes up with serialized
arrays

(e.g. preferences) which include special characters, like e.g. the €
(EUR)

sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed
on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t
change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid
changing all the serialized fields by myself. Maybe there is a
possibility

to adjust the value length somehow on re-import time.

Greetings
christian

–
View this message in context:
http://www.nabble.com/unserialize-problem-on-charset-conversion-tp25
871424s3741p25872316.html
Sent from the egroupware-developers mailing list archive at Nabble.com.

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

–
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951
SteuerNr. 44/667/1114/3

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

jaytraxx 2009-10-13 13:25:19 UTC #7

it works partially, maybe you have an idea:

in phpgwapi/inc/class.preferences.inc.php, LINE 250:
$value = unserialize($row[‘preference_value’]);
if($value === false)
{
// something went wrong while trying to unserialize (maybe a charset problem)
// retry it by manually generating the string lengths
$out = preg_replace(’!s:(\d+):"(.*?)";!se’, “‘s:’.strlen(’$2’).’:”$2";’", $row[‘preference_value’] );
$value = unserialize($out);
}

the values are now correctly read before I login to EGw. After login, they are vanished again. Maybe it’s something to do with writing into the session after login.

Greetings
christian

jaytraxx:

tried, but it doesn’t work
I’ll try further… maybe it’s getting better …

christian

Leithoff:

Sounds better, try.

----------------ursprüngliche Nachricht-----------------
Von: “jaytraxx” christian@jaytraxx.de
An: egroupware-developers@lists.sourceforge.net
Datum: Tue, 13 Oct 2009 05:50:56 -0700 (PDT)

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.strlen(’$2’).’:"$2";’",
$serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving the
string
to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx wrote:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in
setup with a database export and re-import it with the new system
charset.

It’s working mostly fine. The only problem comes up with serialized
arrays

(e.g. preferences) which include special characters, like e.g. the €
(EUR)

sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed
on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t
change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid
changing all the serialized fields by myself. Maybe there is a
possibility

to adjust the value length somehow on re-import time.

Greetings
christian

–
View this message in context:
http://www.nabble.com/unserialize-problem-on-charset-conversion-tp25
871424s3741p25872316.html
Sent from the egroupware-developers mailing list archive at Nabble.com.

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

–
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951
SteuerNr. 44/667/1114/3

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

jaytraxx 2009-10-19 09:55:10 UTC #8

Hi folks,

found a solution:

in phpgwapi/inc/class.preferences.inc.php, LINE 250:
$value = unserialize($row[‘preference_value’]);
if($value === false)
{
// manually retrieve the string lengths of the serialized array if unserialize failed
$value = unserialize(preg_replace(’!s:(\d+):"(.*?)";!se’, “‘s:’.mb_strlen(’$2’,‘8bit’).’:”$2";’", $row[‘preference_value’]));
}

… will soon commit that to /trunk.

christian

jaytraxx:

it works partially, maybe you have an idea:

in phpgwapi/inc/class.preferences.inc.php, LINE 250:
$value = unserialize($row[‘preference_value’]);
if($value === false)
{
// something went wrong while trying to unserialize (maybe a charset problem)
// retry it by manually generating the string lengths
$out = preg_replace(’!s:(\d+):"(.*?)";!se’, “‘s:’.strlen(’$2’).’:”$2";’", $row[‘preference_value’] );
$value = unserialize($out);
}

the values are now correctly read before I login to EGw. After login, they are vanished again. Maybe it’s something to do with writing into the session after login.

Greetings
christian

jaytraxx:

tried, but it doesn’t work
I’ll try further… maybe it’s getting better …

christian

Leithoff:

Sounds better, try.

----------------ursprüngliche Nachricht-----------------
Von: “jaytraxx” christian@jaytraxx.de
An: egroupware-developers@lists.sourceforge.net
Datum: Tue, 13 Oct 2009 05:50:56 -0700 (PDT)

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.strlen(’$2’).’:"$2";’",
$serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving the
string
to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx wrote:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do that in
setup with a database export and re-import it with the new system
charset.

It’s working mostly fine. The only problem comes up with serialized
arrays

(e.g. preferences) which include special characters, like e.g. the €
(EUR)

sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing is changed
on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I don’t
change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to avoid
changing all the serialized fields by myself. Maybe there is a
possibility

to adjust the value length somehow on re-import time.

Greetings
christian

–
View this message in context:
http://www.nabble.com/unserialize-problem-on-charset-conversion-tp25
871424s3741p25872316.html
Sent from the egroupware-developers mailing list archive at Nabble.com.

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

–
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951
SteuerNr. 44/667/1114/3

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Leithoff_Klaus 2009-10-19 09:57:08 UTC #9

If that works, we should put that to config as well (-> customfields!!!)

jaytraxx 2009-10-19 21:17:22 UTC #10

Hi Klausi,

my knowledge of customfields is close to NULL. I just committed it for
prefs. Maybe you want to apply it for customfields.

Thx
christian

Klaus schrieb:

If that works, we should put that to config as well (-> customfields!!!)

Von: “jaytraxx” christian@jaytraxx.de
An: egroupware-developers@lists.sourceforge.net
Datum: Mon, 19 Oct 2009 02:55:18 -0700 (PDT)

Hi folks,

found a solution:

in phpgwapi/inc/class.preferences.inc.php, LINE 250:
$value = unserialize($row[‘preference_value’]);
if($value === false)
{
// manually retrieve the string lengths of the serialized array if
unserialize failed
$value = unserialize(preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.mb_strlen(’$2’,‘8bit’).’:"$2";’", $row[‘preference_value’]));
}

… will soon commit that to /trunk.

christian

jaytraxx wrote:

it works partially, maybe you have an idea:

in phpgwapi/inc/class.preferences.inc.php, LINE 250:
$value = unserialize($row[‘preference_value’]);
if($value === false)
{
// something went wrong while trying to unserialize (maybe a charset
problem)
// retry it by manually generating the string lengths
$out = preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.strlen(’$2’).’:"$2";’", $row[‘preference_value’] );
$value = unserialize($out);
}

the values are now correctly read before I login to EGw. After login,
they

are vanished again. Maybe it’s something to do with writing into the
session after login.

Greetings
christian

jaytraxx wrote:

tried, but it doesn’t work
I’ll try further… maybe it’s getting better …

christian

Leithoff, Klaus wrote:

Sounds better, try.

Von: “jaytraxx” christian@jaytraxx.de
An: egroupware-developers@lists.sourceforge.net
Datum: Tue, 13 Oct 2009 05:50:56 -0700 (PDT)

I found one hint on php.net:
$out = preg_replace(’!s:(\d+):"(.*?)";!se’,
"‘s:’.strlen(’$2’).’:"$2";’",
$serialized );
$value = unserialize($out);

This re-calclulates every stringlen of every value before giving
the
string
to unserialize(), means:
a:17:{s:8:“currency”;s:1:“€” …
gets
a:17:{s:8:“currency”;s:3:“€” …

What do you think?
Greetings
christian

jaytraxx wrote:

Hi folks,

I’m currently migrating our EGw from iso charset to utf-8. I do
that
in
setup with a database export and re-import it with the new
system
charset.

It’s working mostly fine. The only problem comes up with
serialized
arrays

(e.g. preferences) which include special characters, like
e.g. the €
(EUR)

sign.

In ISO, the € (EUR) value is serialized like that:
a:17:{s:8:“currency”;s:1:“€” …

If I re-import the database now with utf-8 as charset, nothing
is
changed
on the serialized array on re-import. It remains with
a:17:{s:8:“currency”;s:1:“€” …
But, the € (EUR) sign in UTF-8 is three chars long and until I
don’t
change the value to
a:17:{s:8:“currency”;s:3:“€” …
all unserialize calls on the string end up with “false”.

Anyone out there having a fix for that is welcome. I just want to
avoid
changing all the serialized fields by myself. Maybe there is a
possibility

to adjust the value length somehow on re-import time.

Greetings
christian

–
View this message in context:

http://www.nabble.com/unserialize-problem-on-charset-convers
ion-tp25
871424s3741p25872316.html
Sent from the egroupware-developers mailing list archive at
Nabble.com.

Come build with us! The BlackBerry® Developer Conference in SF,
CA
is the only developer event you need to attend this year. Jumpstart
your
developing skills, take BlackBerry mobile applications to
market and
stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/egroupware-deve
lopers

–
Stylite GmbH
[ open style of IT ]
Morschheimer Strasse 15
67292 Kirchheimbolanden

fon 06352 . 70629-0
fax 06352 . 70629-30
www.stylite.de

Geschäftsführer: Andre Keller, Gudrun Müller, Ralf Becker
Handelsregister Kaiserslautern HRB 30575
USt-ID: DE214280951
SteuerNr. 44/667/1114/3

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart
your
developing skills, take BlackBerry mobile applications to market
and
stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net

https://lists.sourceforge.net/lists/listinfo/egroupware-develo
pers

–
View this message in context:
http://www.nabble.com/unserialize-problem-on-charset-conversion-tp25
871424s3741p25955739.html
Sent from the egroupware-developers mailing list archive at Nabble.com.

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Come build with us! The BlackBerry® Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

eGroupWare-developers mailing list
eGroupWare-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/egroupware-developers

Leithoff_Klaus 2009-10-21 09:00:31 UTC #11

I will have a look at it, when i am back
(have two weeks off)