Migration to Unicode (UTF-8)

• mishmash of external libraries
  1. PHP Multibyte String extention^[link13]
  2. iconv^[link14]
  3. Internationalization extension^[link15] -- ICU library wrapper
  4. PHP UTF-8^[link16]
  5. Patchwork-UTF8^[link17]
  6. Portable UTF-8^[link18]
  7. HTML Purifier^[link19]
  8. forceutf8^[link20]
• development culture means that people work on what they’re interested in

1. PCRE UTF-8 support: will not run if your PHP installation is not compiled with UTF-8 support in the PCRE extension.
2. 3rd party libraries used by WackoWiki also require UTF-8 support
3. utf8mb4: MySQL versions prior to 5.7.7 or MariaDB 10.2.2 do not have the innodb_large_prefix option enabled by default.

1. Unicode normalization

1. http://www.w3.org/TR/charmod-norm/

2. Steps

1. Convert to UTF-8 without BOM.
   • make sure that any included/required file is in either in ASCII or UTF-8 without BOM, as php doesn't handle non-ascii file very good
   • Tools: iconv

     •  find . -name "*.php" -exec iconv -f ISO-8859-1 -t UTF-8 {} -o {}.new \;	

3. search code base for non-UTF8 compatible functions and replace them^[link21]
   • add missing functions^[link7], not covered by PHP Multibyte String extention, to lib/mb_extends/mb_extends.php
4. add optional support to replace SafeHtml with HTMLPurifier^[link19]
5. write a database conversion script^[link22] that runs through the database and re-encodes everything as UTF-8
6. String access by character
7. remove unneeded functions notes^[link23]
   1. htmlentities
   2. html_entity_decode

2.1. MySQL – Migrating a database data that is already encoded in latin1 to UTF-8

• migrate all code and data in a single shot
• downtime needed
• trial runs needed
• watch for cp1252 vs. ISO-8859-1

1. utf8mb4^[link12]
2. Converting your MySQL database to UTF8^[link5]
3. Database Conversion Script^[link22]

3. Resources