Osamu Sayama's Weblog

Reduce locale shared object size

Current shared object size for UTF-8 locale is about 2Mbyte per a locale. This size is increasing because new unicode standard introduces new characters whenever it is released.

% ls -lah /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3 /usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3 /usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3
-r-xr-xr-x   1 root     bin         2.4M Sep 19 04:31 /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3
-r-xr-xr-x   1 root     bin         1.7M Aug  7 19:42 /usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3
-r-xr-xr-x   1 root     bin         1.7M Aug  7 19:42 /usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3

'locale -a|grep -i utf-8 | wc' shows 108 locales on nevada 100 with full locale support and more than 400MByte is used for UTF-8 locale shared objects. This was not so problem on the installed system (however, when creating a patch for the locale, huge size of patch will be created...). However, it is a problem for OpenSolaris Live CD because the size is more limited. So we should try to reduce this size as possible. The root cause of this size is that the weight tables in  _LC_collate_t lc_coll (ct_wgts* and subs_map) and qmask index table (qifx) in _LC_ctype_t lc_ctype. Since many of UTF-8 locales are sharing LC_CTYPE and LC_COLLATE definition between locales (ex, fr_FR.UTF-8 and fr_CA.UTF-8), spliting these tables from locale shared object and creates new shared object for ctype and collation tables can reduce the total disk size dramatically. It looks that the table size of LC_CTYPE and LC_COLLATE consists of 99% of the total size and 90% is LC_COLLATE on UTF-8 locale.

- en_US.UTF-8

[68]    |    982856|    974848|OBJT |LOCL |0    |11     |ct_wgts0
[67]    |      8008|    974848|OBJT |LOCL |0    |11     |ct_wgts1
[72]    |   1958144|    243713|OBJT |LOCL |0    |11     |subs_map
[81]    |   2202848|    243456|OBJT |LOCL |0    |11     |qidx

- fr_FR.UTF-8 and fr_CA.UTF-8

[72]    |   1120768|    607132|OBJT |LOCL |0    |11     |weightstr
[69]    |    793096|    262136|OBJT |LOCL |0    |11     |ct_wgts0
[68]    |    530960|    262136|OBJT |LOCL |0    |11     |ct_wgts1
[67]    |    268824|    262136|OBJT |LOCL |0    |11     |ct_wgts2
[66]    |      6688|    262136|OBJT |LOCL |0    |11     |ct_wgts3
[71]    |   1055232|     65535|OBJT |LOCL |0    |11     |subs_map
[80]    |   1728456|     65278|OBJT |LOCL |0    |11     |qidx

As a trial, I splited fr_FR.UTF-8.c, which is created by localedef command, to 3 parts. CLDR.UTF-8-ctype.c, CLDR.fr.UTF-8-collate.c  and fr_FR.UTF-8.c. Then compiled and linked like the following.

% cc -xO3 -K PIC -G -Xa  -h CLDR.UTF-8-ctype.so.3 -o CLDR.UTF-8-ctype.so.3 ./CLDR.UTF-8-ctype.c
% cc -xO3 -K PIC -G -Xa  -h CLDR.fr.UTF-8-collate.so.3 -o CLDR.fr.UTF-8-collate.so.3 ./CLDR.fr.UTF-8-collate.c
% cc -xO3 -K PIC -G -Xa  -h fr_FR.UTF-8.so.3 -o fr_FR.UTF-8.so.3 ./fr_FR.UTF-8.c  /usr/lib/locale/common/methods_unicode.so.3 ./CLDR.UTF-8-ctype.so.3 ./CLDR.fr.UTF-8-collate.so.3 -R /usr/lib/locale/common

Then copy CLDR.UTF-8-ctype.so.3 and CLDR.fr.UTF-8-collate.so.3 to /usr/lib/locale/common, copy  fr_FR.UTF-8.so.3 to /usr/lib/locale/fr_FR.UTF-8. Here is modified source.

% ldd /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3 /usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3 /usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3
/usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3:
        libc.so.1 =>     /lib/libc.so.1
        /usr/lib/locale/common/methods_unicode.so.3
        en_US.UTF-8-ctype.so.3 =>        /usr/lib/locale/common/en_US.UTF-8-ctype.so.3
        en_US.UTF-8-collate.so.3 =>      /usr/lib/locale/common/en_US.UTF-8-collate.so.3
        libm.so.2 =>     /lib/libm.so.2
/usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3:
        libc.so.1 =>     /lib/libc.so.1
        /usr/lib/locale/common/methods_unicode.so.3
        CLDR.UTF-8-ctype.so.3 =>         /usr/lib/locale/common/CLDR.UTF-8-ctype.so.3
        CLDR.fr.UTF-8-collate.so.3 =>    /usr/lib/locale/common/CLDR.fr.UTF-8-collate.so.3
        libm.so.2 =>     /lib/libm.so.2
/usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3:
        libc.so.1 =>     /lib/libc.so.1
        /usr/lib/locale/common/methods_unicode.so.3
        CLDR.UTF-8-ctype.so.3 =>         /usr/lib/locale/common/CLDR.UTF-8-ctype.so.3
        CLDR.fr.UTF-8-collate.so.3 =>    /usr/lib/locale/common/CLDR.fr.UTF-8-collate.so.3
        libm.so.2 =>     /lib/libm.so.2

% ls -lah /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3 /usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3 /usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3
-rwxr-xr-x   1 root     root         44K Oct 17 15:19 /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3
-rwxr-xr-x   1 root     root         14K Oct 19 08:59 /usr/lib/locale/fr_CA.UTF-8/fr_CA.UTF-8.so.3
-rwxr-xr-x   1 root     root         14K Oct 17 18:58 /usr/lib/locale/fr_FR.UTF-8/fr_FR.UTF-8.so.3
% ls -lah /usr/lib/locale/common/CLDR.* /usr/lib/locale/common/en_US.UTF-8-c*
-rwxr-xr-x   1 root     root         90K Oct 18 15:09 /usr/lib/locale/common/CLDR.UTF-8-ctype.so.3
-rwxr-xr-x   1 root     root        1.6M Oct 18 15:09 /usr/lib/locale/common/CLDR.fr.UTF-8-collate.so.3
-rwxr-xr-x   1 root     root        2.1M Oct 17 15:18 /usr/lib/locale/common/en_US.UTF-8-collate.so.3
-rwxr-xr-x   1 root     root        243K Oct 17 15:18 /usr/lib/locale/common/en_US.UTF-8-ctype.so.3

This simple modification works fine with current libc (no modification is needed in libc !) and meet our requirement. The number of current UTF-8 collation types are about 15 and ctype types are 2. So I expect that this change will reduce the size to 1/6 ((15 collation types + 2 ctype types) / 100 UTF-8 locales)... Now I'm thinking that localedef should add the option to produce 3 shared objects. I will try later...

Posted at 10:15午後 10 20, 2008 by sayama in English  |  投稿されたコメント[0]

unzip for non-ascii file name

Current unzip command in OpenSolaris cannot handle a non-ascii file name used in Windows world. If such a file is included in zip file, created file itself is garbled. This is inconvenient because localized file name is often used in Localized Windows. When searching the information from the web, I found that Ubuntu has already the patch for this problem (here). I tried to make build SUNWunzip with this patch in sfw workspace. A file encoding created by Windows zip can be specified by -O option (Also a file encoding created by Unix zip seems to be specified by -I option). For example, zip file created by Japanese windows can convert by -O cp932 (or PCK, SJIS) option like the following.

I filed this as 6719511 but not fixed yet. So I attached the pacakge I made build (here). Please use it until this CR is integrated.

Posted at 10:51午後 7 24, 2008 by sayama in English  |  投稿されたコメント[0]

Printing with BrandZ and Linux native driver - English

In the linux world, the inkjet printer drivers from the major printer manufacturers in Japan such as Canon or Epson can be downloaded from the web site like Canon (English version here, PIXMA iP4200 only?) or Epson (English verion here). So it's great if these drivers can be leveraged for Solaris with a minimum configuration. Fortunately, CentOS3 under BrandZ includes cups-1.1.17 and  ghostscript-7.05 which works fine with these drivers so that these would be used without a modification. Here is the configuration I tried. Since USB port cannot be accessed in BrandZ, the result of the conversion is backed to Solaris and then it is sent to a printer on Solaris if the USB printer is used. However, a network printer can be accessed in BrandZ, the result is sent directly.

o When a USB printer is used,

Solaris remote printer - > BrandZ Linux Printer - > Solaris USB local printer -> USB printer

o When a network attached printer is used,

Solaris remote printer - > BrandZ Linux Printer - > Network printer

I could use both cases with using Canon PIXUS iP4100 on nevada_b54 + b55 lpsched (See bug of 6502814). Also coud use Epson PM-A820 USB printer. The following is the step how to do. To do this, you must set up BrandZ and install CentOS from here. Please see here how to install.

1. Setting of Solaris USB local printer (Option USB printer only)

Since a spooled content is already converted to the native printer code of Canon/Epson, no more conversion is needed here. So a USB printer without a filter should be set up. Here is the example of this setting.

# lpadmin -p usbprt -v /dev/printers/0 -I any -i /usr/lib/lp/model/standard -o "stty=-opost" -o banner=never
# /usr/sbin/accept usbprt
# /usr/bin/enable usbprt

usbprt is printer name. Also open rfc1179 service to enable brandZ host to access global zone's spooler. Please try telnet <machine> 515 from a remote machine after enabled. If connected, it works fine.

# svcadm enable svc:/application/print/rfc1179:default

2. BrandZ Linux Printer

Install the driver rpms and set up cups.  I downloaded bjfilter-common-2.50-2.i386.rpm and bjfilter-pixusip4100-2.50-2.i386.rpm from Canon site. Also libxml-1.8.17-9.2.i386.rpm is downloaded from rpmfind because it is needed. In the case of Epson PM-A820, only pipslite-cups-1.0.0-1.i386.rpm is needed. (Also to use Epson pipslite, you must create a ppd file to connect USB printer through linux. See avasys FAQ for this if you use knoppix. Once created, copy /usr/share/cups/model/ekpma820.ppd to brandZ).

o Canon PIXUS iP4100

brandz# rpm -Uvh libxml-1.8.17-9.2.i386.rpm bjfilter-common-2.50-2.i386.rpm bjfilter-pixusip4100-2.50-2.i386.rpm

o Epson PM-A820

brandz# rpm -Uvh pipslite-cups-1.0.0-1.i386.rpm

brandz# /etc/init.d/cups start
brandz# chkconfig cups-lpd on       < - from global zone it can access with lpd,

Also /etc/{rc2.d,rc3.d}/lx_S55cups should be moved to S55cups to start cupsd automatically. After that, cups in BrandZ can configure with using mozilla. Here is the example.

% ssh -X brandz /usr/bin/mozilla http://localhost:631/

o USB printer

  Since the filter output is backed to global zone printer which is defined in 1), the printer defined in 1) is specified.

Printer name: lxprt
URI: lpd://<global IP>/usbprt

o Network printer

  The filter output is directly sent to the printer.

Printer name: lxprt
URI: socket://<printer IP>:9100

3. Solaris remote printer

Finally the remote printer is set on the global zone of Solaris. 

# lpadmin -p canon -s <brandz IP>\!lxprt
# lpadmin -d canon

After that, you can printer PS file to send this remote printer.

% lp t.ps

If you cannot print well, please check the following

• telnet <brandZ> 515 in global zone, telnet <global zone> 515 in brandZ work fine?
• How is the cusps status?
• Change /dev/printers/0 to a file and check whether a context is increased when printing.
  

Posted at 07:19午後 12 28, 2006 by sayama in English  |  投稿されたコメント[0]