Monday June 20, 2005
A Peek into Solaris Sustaining engineer's life...Thoughts of an engineer(Raja Gopal Andra) working on two worlds document for Native LDAP in Solaris[TM] Operating Environment. Finally we managed to come up with a share-able version of the document which I'm including below! My sincere thanks to following people who helped me in this effort: * Martin Reifenrath * Michael Haines * Serge Dussud * Stacey Marshall The Native LDAP PSD can also be accessed from http://blogs.sun.com/roller/resources/raja/ldap-psd.html Questions, error-fixes, additions and amendments to: Raja Gopal Andra rajagopal DOT andra AT sun DOT com ---------------------------------------------------------------------------------------- Date: June 20, 2005 Table of Contents 1.0 About Native LDAP 1.1. What is Native LDAP? 1.2. What is Native/Secure LDAP? 1.3. Why use LDAP? 1.4. What are the benefits of using Secure LDAP versus using NIS, NIS+, or files? 2.0 Common How Tos 2.1. How to Set Up a Solaris[TM] Secure LDAP Client on Solaris 8? 2.2. How to Set Up a Solaris[TM] Secure LDAP Client on Solaris 9? 2.3. How to Create Client Profiles Manually? 2.4. How to Setup and Implement printing? 2.5. How to Setup and Implement SSD's? 2.6. How to Setup and Implement RBAC? 2.7. How to Setup and Implement SMC? 2.8. How to Setup and Implement Automounter? 2.9. How to use ldapaddent(1M)? 2.10. How do I reinitialize a client? 2.11. How do I un-initialize a client? 2.12. How to setup Secure NFS?? 3.0 LDAP Client Security 3.1. How to Setup and Implement SASL DIGEST-MD5? 3.2. How to Setup and Implement TLSv1/SSL? 4.0 Debugging LDAP 4.1. How to start debugging, if something fails? What to do first? 4.2. How to use snoop to debug the LDAP Client? 4.3. How to debug PAM? 4.4. How to understand the Sun ONE Directory Server v5.2 access log? 4.5. How to use ssltap to debug TLSv1/SSL communication? 5.0 Some Frequently Asked Questions 5.1. Does the LDAP Client (Phase I/II) support password aging? 5.2. Is there a user name limit imposed on LDAP clients? 5.3. What schema is used by the LDAP (Phase I/II)? 5.4. What tools exist to Automate the LDAP Client setup? 5.5. Does the LDAP Client (Phase I/II) support TLSv1/SSL? 5.6. What is NS1 format?? How is the NS1 format converted/used to authenticate against the userPassword in CRYPT format in the LDAP server? 5.7. What TLSv1/SSL Functions are supported in the LDAP Client (Phase I/II)? 5.8. Are there any restrictions on where idsconfig can be run from? 5.9. Does ldapaddent(1M) have any requirements? 5.10. Can you use both pam_ldap and pam_unix to support the password policy? 5.11. How do I run ldaplist(1)? 5.12. Why would I want multiple profiles? 5.13. Can my Directory (LDAP) Server be it's own LDAP client? 5.14. What are the requirements for Directory server to support Solaris Native LDAP clients? 5.15. What commands are usually run on a Native LDAP Client? 5.16. What commands are usually run on a directory server? 5.17. Can I use SASL CRAM-MD5? 5.18. What is the ldap_cachemgr(1M) daemon? 5.19. What is RFC2307bis? 5.20. Why does /usr/bin/passwd fail, if root tries to change and end-users password in LDAP? 5.21. What Printer Schema is used by the Secured LDAP Client? 5.22. How to troubleshoot when users are not able to login using LDAP? 5.23. Can the client profile location be changed? 6.0 Information Gathering 6.1. Which files should I gather? 6.2. Which commands should I run to collect status/config information? 6.3. Which information does an Explorer (since version 4.1) collect and how can this become interpreted? 6.4. Which simple commands should I run for some basic testing? 6.5. How to troubleshoot when client does not seem to be able to contact server? 7.0 Packages 7.1. List the Sun ONE Directory Server 5.1 Packages? 7.2. List the Sun ONE Directory Server 5.2 Packages? 7.3 Secure LDAP client packages in Solaris 8 and 9 8.0 Naming Service Migration? 8.1. NIS+/LDAP 8.2. NIS/LDAP 9.0 Patches 10.0 Known Bugs & RFEs? 10.0. Bugs 10.1. RFE's 11.0 References? 1. Important Man Pages 2. Sun SRDBs / InfoDocs 3. Sun Educational Services 4. Solaris Documentation 5. Third Party Documentation 6. RFCs 12.0 Supportability 13.0 Additional Support 14.0 Index of LDAP documents 1. docs.sun.com 2. Manual Pages 3. Other 15 Contributors 1.0. About Native LDAP 1.1. What is Native LDAP? Native LDAP is the integration of LDAP as a name service for the Solaris Operating Environment (OE). First introduced in Solaris 8 OE, Native LDAP allows the name services library function calls to retrieve their information from an LDAP server (such as the Sun Java System Directory Server). These functions are known as the getXbyY functions and include, not limited to, gethostbyname(3NSL), gethostbyaddr(3NSL), getpwent(3NSL), and getservbyname(3NSL). For more details on the getXbyY functions please see the nsswitch.conf(4) man page, which contains the "Database" and "Used By" listing in the DESCRIPTION section. Most of the getXbyY and other calls that use name service are listed in the "Used By" column. The man pages for each of these calls have more calls listed in the NAME section. Once configured Native LDAP is another name service option within nsswitch.conf(4) designed to complement /etc files and DNS and is used in the same way as NIS or NIS+. Please note that LDAP clients do a hard lookup for getXbyY calls which means that the caller waits until it gets a response. 1.2. What is Native/Secure LDAP? Secure LDAP is Native LDAP client with security features included. A more robust security model, which supports strong authentication, TLS encrypted sessions. A client's proxy credentials are NO LONGER stored in a client's profile on the directory server 1.3. Why use LDAP? The answer to this question depends on your role and how you use the system. If you are a user, a system administrator, or a developer then the following apply: As a user, LDAP offers: * A single place to maintain personal information * A single source for information about others * A place to find what you need to access * Remote access as easy as local access * Mobility from your desktop * A way to ease every day communication and work As a system administrator, LDAP offers: * A single place to administer users and groups * A single place to administer enterprise configuration information * A way for authority to be distributed * A way to enable data to be distributed and replicated for reliability and performance As a developer, LDAP offers: * A place to get and store information about users * A place to get and store configuration information * Mobility to users of your application * A general attribute/value directory that is fast, replicated and reliable 1.4. What are the benefits of using Secure LDAP versus using NIS, NIS+, or files? LDAP offers an extensive and extensible standards based data repository. The Lightweight Directory Access Protocol provides the standard models and protocols used in directories today. One of the key advantages to LDAP, is it's inter-operability, and the fact that it is possible for a LDAP client developed by Novell for example to work with a server developed by Sun. LDAP provides better security through authentication and authorization and by utilizing secure transport layers. The LDAP standard has proposed ways (RFC 2251 and RFC 2829) in which LDAP clients can authenticate to LDAP servers. Note, that RFC 3377 - The LDAPv3 Technical Specification was published to list all RFCs that comprise the full specifications of LDAPv3. That is, RFC 2251-2256, RFC 2829 (authentication methods) and RFC 2830 (Extension for TLS). LDAP replication provides increased data availability, load balancing, enhanced performance (by replicating on a local server) and local data management. It also provides a mechanism for mix-and-matching data from different LDAP servers. LDAP is the preferred naming service for the Solaris Operating Environment. We have announced that NIS+ may be removed at some future (unspecified) time. In other words, we have announced the EOF of NIS+ http://docs.sun.com/app/docs/doc/806-4077/6jd6blbba?a=view gives a good comparison chart for different naming services. 2.0. Common How-to's 2.1. How to Set Up a Solaris[TM] Secure LDAP Client on Solaris 8? The following procedure requires that the following 2 patches are installed on the system: 108993-47 (or newer) for the software and configuration modules 108808-44 (or newer) for the man pages It also assumes that a LDAP server has been successfully configured for Native LDAP (e.g., using idsconfig(1M)). A Solaris 8 system can be configured as Native LDAP client using ldapclient(1M) (/usr/sbin/ldapclient) command. ldapclient(1M) needs to be run as root to configure it as an LDAP client. There are two ways to configure a system as LDAP client - using profiles stored on LDAP server and using 'manual' method. Using profiles is recommended as it's easy to make mistakes when configuring a system using manual option. Please refer to man page for different options with ldapclient(1M) command. There are multiple ways to configure a Solaris 8 system to become a native LDAP client. Before starting to configure the system, some planning need to be done to have a clear road map for proceeding with the configuration. One important question is how LDAP clients authenticate with the LDAP server? The two options are: o no authentication at all / anonymous access: This method requires to leave complete DIT (Directory Information Tree) that holds the Naming Services Data within the LDAP server's database completely open for read/write access from client systems. This method is not recommended due to security concerns. o authentication through a Proxyagent: This method requires a special user to be already defined in the LDAP database and to become used by all client systems when authenticating to the LDAP server. This is the recommended method. You may create a client profile in your LDAP server that may become commonly used by all (or multiple) clients. Such a profile is good for easily storing/administrating configuration parameters that will become used by all (or multiple) clients in common. Steps to be performed to configure an LDAP client: 1. Verify, if all required patches are installed on the client system and make sure the patches are latest. * Run either one of the commands "showrev -p" or "ls /var/sadm/patch" * ... then check, if the following patches appear in the output: 108827, 108993 and 112218 * ... then cross check with SunSolve to see, if the revision (the extension "-nn" in the patch number) is identical. 2. Create ACI to allow anonymous access. Please skip to step 3, if you decided to authenticate through a Proxyagent. When allowing to access the LDAP server with anonymous access, you will need to create an ACI (Access Control Information) in the LDAP server to permit READ access for User Passwords to anybody. * Create an LDIF-file that looks like follows: dn: dc=ldap,dc=example,dc=com changetype: modify add: aci aci: (target="ldap:///dc=ldap,dc=example,dc=com") (targetattr="userPassword") (version 3.0; acl "password read"; allow (compare,read,search) userdn = "ldap:///anyone"; ) * Load this file into your LDAP-database: # ldapadd -h <IP-address> -D "cn=Directory Manager" -w <password> -f <filename> This file will allow users that are stored in the LDAP database to login to LDAP client systems. Please keep in mind, that permitting users to change their passwords requires to define similar ACIs that allow even higher access levels to become opened for anonymous access. Skip to step 4. 3. Create Proxyagent If you decided to have the client authenticating through a proxyagent, you will need to create the proxyagent in your LDAP database. * Create an LDIF file that looks like follows: dn: cn=proxyagent,ou=profile,dc=ldap,dc=example,dc=com cn: proxyagent sn: proxyagent objectclass: top objectclass: person userpassword: test1234 * Load this file into your LDAP-database: # ldapadd -h <IP-address> -D "cn=Directory Manager" -w <password> -f <filename> 4. Create client Profile Please skip to step 5., if you do not want to create a client profile. If you decided to use a client profile for all (or multiple) native LDAP clients, you will need to create the profile and store it in your LDAP database. * Create an LDIF file that contains information matching to your environment: (example using a proxyagent) # ldap_gen_profile -P S8Profile -b dc=ldap,dc=example,dc=com \ -D "cn=proxyagent,ou=profile,dc=ldap,dc=example,dc=com" \ -w test1234 -a simple <IP-address> > S8Profile.ldif (example for anonymous access) # ldap_gen_profile -P S8Profile -b dc=ldap,dc=example,dc=com \ -a none <IP-address> > S8Profile.ldif * Edit the LDIF file. All lines must start with text in the first column. Set all parameters to meet your environment's requirements: dn: cn=S8Profile,ou=profile,dc=ldap,dc=example,dc=com SolarisBindDN: cn=proxyagent,ou=profile,dc=dc=ldap,dc=example,dc=com SolarisBindPassword: {NS1}4a3788e8c053424f SolarisLDAPServers: 10.16.50.134 SolarisSearchBaseDN: dc=ldap,dc=example,dc=com SolarisAuthMethod: NS_LDAP_AUTH_SIMPLE SolarisTransportSecurity: NS_LDAP_SEC_NONE SolarisSearchReferral: NS_LDAP_FOLLOWREF SolarisSearchScope: NS_LDAP_SCOPE_ONELEVEL SolarisSearchTimeLimit: 30 SolarisCacheTTL: 43200 cn: S8Profile SolarisBindTimeLimit: 30 ObjectClass: top ObjectClass: SolarisNamingProfile * Load this file into your LDAP database: # ldapadd -h <IP-address> -D "cn=Directory Manager" -w <password> \ -f S8Profile.ldif 5. Initialize the system to become a native LDAP client * for anonymous access without profile: # ldapclient -i -a none -d ldap.example.com <IP-address> * for proxyagent access without profile: # ldapclient -i -a simple \ -D "cn=proxyagent,ou=profile,dc=ldap,dc=example,dc=com" \ - w <password> \ -d ldap.example.com <Ldap_Server_IP_address> * for any access with profile: # ldapclient -P <profile_name> -d ldap.example.com <IP-address> 6. Verify that the LDAP client works * Run the command ldaplist to verify, if the LDAP client works: (the output MUST look like the following example) # ldaplist dn: cn=Directory Administrators, dc=ldap,dc=example,dc=com dn: ou=People, dc=ldap,dc=example,dc=com dn: ou=Special Users,dc=ldap,dc=example,dc=com dn: ou=Groups, dc=ldap,dc=example,dc=com dn: ou=group,dc=ldap,dc=example,dc=com dn: ou=rpc,dc=ldap,dc=example,dc=com ... 7. Prepare nsswitch.conf file Edit the file /etc/nsswitch.conf to use LDAP as the Naming Service for those tables that meet your environment. Please regard the file /etc/nsswitch.ldap as an example for this. 8. Reboot your client system It is recommended (but not required) to reboot the system, after it became configured as a native LDAP client, because some application programs (like automountd) need to become restarted to reload their naming service information. Example 1: Example for configuring a native LDAP client in Solaris 8 # /usr/sbin/ldapclient -P example_prof -D cn=proxyagent,ou=profile,dc=example,dc=com -w proxy12 -d example.com 192.168.1.1 Arguments parsed: domainName: example.com proxyDN: cn=proxyagent,ou=profile,dc=example,dc=com profileName: example_prof proxyPassword: proxy12 defaultServerList: 192.168.1.1 Handling init option About to configure machine by downloading a profile findBaseDN: begins findBaseDN: ldap not running findBaseDN: calling __ns_ldap_default_config() found 1 namingcontexts findBaseDN: __ns_ldap_list(NULL, "(&(objectclass=nisDomainObject)(nisdomain=example.com))" rootDN[0] dc=example,dc=com found baseDN dc=example,dc=com for domain example.com Proxy DN: cn=proxyagent,ou=profile,dc=example,dc=com Proxy password: {NS1}ecc423aad07c60 Credential level: 1 Authentication method: 1 About to modify this machines configuration by writing the files Stopping network services Stopping sendmail Stopping nscd Stopping autofs ldap not running nisd not running nis_cache not running nispasswd not running Stopping nis(yp) file_backup: stat(/etc/nsswitch.conf)=0 file_backup: (/etc/nsswitch.conf -> /var/ldap/restore/nsswitch.conf) file_backup: stat(/etc/defaultdomain)=0 file_backup: (/etc/defaultdomain -> /var/ldap/restore/defaultdomain) file_backup: stat(/etc/.rootkey)=-1 file_backup: No /etc/.rootkey file. file_backup: stat(/var/nis/NIS_COLD_START)=-1 file_backup: No /var/nis/NIS_COLD_START file. file_backup: nis domain is "sunicnc.France.Sun.COM" file_backup: stat(/var/yp/binding/sunicnc.France.Sun.COM)=0 file_backup: (/var/yp/binding/sunicnc.France.Sun.COM -> /var/ldap/restore/sunicnc.France.Sun.COM) file_backup: stat(/var/ldap/ldap_client_file)=-1 file_backup: No /var/ldap/ldap_client_file file. Starting network services start: /usr/bin/domainname example.com... success start: /usr/lib/ldap/ldap_cachemgr... success start: /etc/init.d/autofs start... success start: /etc/init.d/nscd start... success start: /etc/init.d/sendmail start... success System successfully configured # *NOTE* When ldapclient(1M) ends with "System successfully configured", it means that the Native LDAP client has been successfully configured. But it does NOT mean the Native Ldap client will also successfully work! For example: If for instance you did specify a non-existing name for the proxy DN, ldapclient(1M) will log the message "System successfully configured", but the system will not be able to work as a LDAP client (won't be able to bind to the LDAP server). It is hence strongly recommended to check that the Native LDAP client indeed works fine, for instance by running ldaplist(1). 2.2. How to Set Up a Solaris[TM] Secured LDAP Client on Solaris 9? This is described in detail at: http://docs.sun.com/app/docs/doc/806-4077/6jd6blbfk?a=view The LDAP client in Solaris 9 is configured with the ldapclient(1M) (/usr/sbin/ldapclient) command. ldapclient(1M) must be run as root to configure the system as an LDAP client, and also can be used to generate client profiles. Please refer to man page for different options available. Example 1: Setting up a system as Native LDAP client using 'example_prof' profile and using 192.168.1.1 as the directory server. # /usr/sbin/ldapclient -v init -a profilename=example_prof -a proxydn=cn=proxyagent,ou=profile,dc=example,dc=com -a proxypassword=proxy12 -a domainname=example.com 192.168.1.1 Parsing profilename=example_prof Parsing proxydn=cn=proxyagent,ou=profile,dc=example,dc=com Parsing proxypassword=proxy12 Parsing domainname=example.com Arguments parsed: domainName: example.com proxyDN: cn=proxyagent,ou=profile,dc=example,dc=com profileName: example_prof proxyPassword: proxy12 defaultServerList: 192.168.1.1 Handling init option About to configure machine by downloading a profile findBaseDN: begins findBaseDN: ldap not running findBaseDN: calling __ns_ldap_default_config() found 1 namingcontexts findBaseDN: __ns_ldap_list(NULL, "(&(objectclass=nisDomainObject)(nisdomain=example.com))" rootDN[0] dc=example,dc=com found baseDN dc=example,dc=com for domain example.com Proxy DN: cn=proxyagent,ou=profile,dc=example,dc=com Proxy password: {NS1}ecc423aad07c60 Credential level: 1 Authentication method: 1 About to modify this machines configuration by writing the files Stopping network services Stopping sendmail Stopping nscd Stopping autofs ldap not running nisd not running nis_cache not running nispasswd not running Stopping nis(yp) Removing existing restore directory file_backup: stat(/etc/nsswitch.conf)=0 file_backup: (/etc/nsswitch.conf -> /var/ldap/restore/nsswitch.conf) file_backup: stat(/etc/defaultdomain)=0 file_backup: (/etc/defaultdomain -> /var/ldap/restore/defaultdomain) file_backup: stat(/etc/.rootkey)=-1 file_backup: No /etc/.rootkey file. file_backup: stat(/var/nis/NIS_COLD_START)=-1 file_backup: No /var/nis/NIS_COLD_START file. file_backup: nis domain is "sunicnc.France.Sun.COM" file_backup: stat(/var/yp/binding/sunicnc.France.Sun.COM)=0 file_backup: (/var/yp/binding/sunicnc.France.Sun.COM -> /var/ldap/restore/sunicnc.France.Sun.COM) file_backup: stat(/var/ldap/ldap_client_file)=-1 file_backup: No /var/ldap/ldap_client_file file. Starting network services start: /usr/bin/domainname example.com... success start: /usr/lib/ldap/ldap_cachemgr... success start: /etc/init.d/autofs start... success start: /etc/init.d/nscd start... success start: /etc/init.d/sendmail start... success System successfully configured # *NOTE* When ldapclient(1M) ends with "System successfully configured", it means that the Native LDAP client has been successfully configured. But it does NOT mean the Native Ldap client will also successfully work! For example: If for instance you did specify a non-existing name for the proxy DN (option: "-a proxydn=..."), ldapclient(1M) will log the message "System successfully configured", but the system will not be able to work as a LDAP-client (won't be able to bind to the LDAP server). Hence it is strongly recommended to check that the Native LDAP client indeed works fine, for instance by running ldaplist(1). 2.3. How to Create Client Profiles Manually? On Solaris 9, use genprofile option of ldapclient(1M) and on Solaris 8, use ldap_gen_profile(1M) commands. Example 1: This example shows how to create a profile on a Solaris 8 system. $ldap_gen_profile -P s8profile -b dc=sun,dc=com -D "cn=proxyagent,ou=people,dc=sun,dc=com" -w test1234 -a simple 129.158.226.172 dn: cn=s8profile,ou=profile,dc=sun,dc=com SolarisBindDN: cn=proxyagent,ou=people,dc=sun,dc=com SolarisBindPassword: {NS1}4a3788e8c053424f SolarisLDAPServers: 129.158.226.172 SolarisSearchBaseDN: dc=sun,dc=com SolarisAuthMethod: NS_LDAP_AUTH_SIMPLE SolarisTransportSecurity: NS_LDAP_SEC_NONE SolarisSearchReferral: NS_LDAP_FOLLOWREF SolarisSearchScope: NS_LDAP_SCOPE_ONELEVEL SolarisSearchTimeLimit: 30 SolarisCacheTTL: 43200 cn: s8profile ObjectClass: top ObjectClass: SolarisNamingProfile $ Now you must edit the s8profile.ldif file to work around bug 4346889. Remove any white space from the beginning of the lines. The result should look like the following (except that it will have your suffix instead of dc=sun,dc=com): dn: cn=s8profile,ou=profile,dc=sun,dc=com SolarisBindDN: cn=proxyagent,ou=people,dc=sun,dc=com SolarisBindPassword: {NS1}4a3788e8c053424f SolarisLDAPServers: 129.158.226.172 SolarisSearchBaseDN: dc=sun,dc=com SolarisAuthMethod: NS_LDAP_AUTH_SIMPLE SolarisTransportSecurity: NS_LDAP_SEC_NONE SolarisSearchReferral: NS_LDAP_FOLLOWREF SolarisSearchScope: NS_LDAP_SCOPE_ONELEVEL SolarisSearchTimeLimit: 30 SolarisCacheTTL: 43200 cn: s8profile ObjectClass: top ObjectClass: SolarisNamingProfile Example 2: The following example shows how to create a profile using genprofile subcommand of ldapclient(1M) on a Solaris 9 system. $ldapclient genprofile -a profileName=s9profile -a credentialLevel=proxy -a authenticationMethod=simple -a bindTimeLimit=20 -a defaultSearchBase=dc=test,dc=com dn: cn=s9profile,ou=profile,dc=test,dc=com ObjectClass: top ObjectClass: DUAConfigProfile defaultSearchBase: dc=test,dc=com authenticationMethod: simple cn: s9profile credentialLevel: proxy bindTimeLimit: 20 $ 2.4. How to setup and implement printing? There is already an existing Info Doc on the subject: Info Doc 73358 How to administer the printers(4) database when using ldap(1) as Naming Service http://sunsolve.sun.com/search/document.do?assetkey=1-9-73358-1&searchclause=73358 2.5. How to Setup and Implement Service Search Descriptors(SSD's)? SSD's are explained in detail at http://docs.sun.com/app/docs/doc/806-4077/6jd6blber?a=view It is easy to configure SSD's - simple steps that can be used are: 1. Create a profile with SSD's 2. Add the profile to LDAP server 3. Configure a system as Native LDAP client using this profile 4. Verify everything works fine and SSD's are working properly. We are not covering 'manual' method of ldapclient for setting up SSD's as using profiles is the recommended and easy way to do this. Example 1: Following is an example on how to configure and use SSD's. step 1: create a profile with SSD Profile is created as explained in section 2.3 ldapsrv$cat ssd_profile.ldif dn: cn=ssd_profile,ou=profile,dc=example,dc=sun,dc=com ObjectClass: top ObjectClass: DUAConfigProfile defaultSearchBase: dc=example,dc=sun,dc=com authenticationMethod: simple preferredServerList: 129.158.233.109 cn: ssd_profile credentialLevel: proxy serviceSearchDescriptor: passwd:ou=people,dc=example,dc=sun,dc=com?one ldapsrv$ Step 2: Add the profile to LDAP server ldapsrv$ldapadd -v -D "cn=directory manager" -w nssecret -f ssd_profile.ldif add ObjectClass: top DUAConfigProfile add defaultSearchBase: dc=example,dc=sun,dc=com add authenticationMethod: simple add preferredServerList: 129.158.233.109 add cn: ssd_profile add credentialLevel: proxy add serviceSearchDescriptor: passwd:ou=people,dc=example,dc=sun,dc=com?one adding new entry cn=ssd_profile,ou=profile,dc==example,dc=sun,dc=com modify complete ldapsrv$ Step 3: Initialize a system as LDAP client using ssd_profile ldapclnt2/$ldapclient -v init -a proxyDN=cn=proxyagent,ou=profile,dc=example,dc=sun,dc=com -a domainname=example.sun.com -a profilename=ssd_profile -a proxypassword=nssecret 129.158.233.109 Parsing proxyDN=cn=proxyagent,ou=profile,dc=example,dc=sun,dc=com Parsing domainname=example.sun.com Parsing profilename=ssd_profile Parsing proxypassword=nssecret Arguments parsed: domainName: example.sun.com proxyDN: cn=proxyagent,ou=profile,dc=example,dc=sun,dc=com profileName: ssd_profile proxyPassword: nssecret defaultServerList: 129.158.233.109 Handling init option About to configure machine by downloading a profile findBaseDN: begins findBaseDN: ldap not running findBaseDN: calling __ns_ldap_default_config() found 2 namingcontexts findBaseDN: __ns_ldap_list(NULL, "(&(objectclass=nisDomainObject)(nisdomain=example.sun.com))" rootDN[0] dc=example,dc=sun,dc=com found baseDN dc=example,dc=sun,dc=com for domain example.sun.com Proxy DN: cn=proxyagent,ou=profile,dc=example,dc=sun,dc=com Proxy password: {NS1}6849886b43e612a6 Credential level: 1 Authentication method: 1 About to modify this machines configuration by writing the files Stopping network services Stopping sendmail Stopping nscd Stopping autofs ldap not running nisd not running nis_cache not running nispasswd not running Stopping nis(yp) file_backup: stat(/etc/nsswitch.conf)=0 file_backup: (/etc/nsswitch.conf -> /var/ldap/restore/nsswitch.conf) file_backup: stat(/etc/defaultdomain)=0 file_backup: (/etc/defaultdomain -> /var/ldap/restore/defaultdomain) file_backup: stat(/etc/.rootkey)=-1 file_backup: No /etc/.rootkey file. file_backup: stat(/var/nis/NIS_COLD_START)=-1 file_backup: No /var/nis/NIS_COLD_START file. file_backup: nis domain is "blr03-01.India.Sun.COM" file_backup: stat(/var/yp/binding/blr03-01.India.Sun.COM)=0 file_backup: (/var/yp/binding/blr03-01.India.Sun.COM -> /var/ldap/restore/blr03-01.India.Sun.COM) file_backup: stat(/var/ldap/ldap_client_file)=-1 file_backup: No /var/ldap/ldap_client_file file. Starting network services start: /usr/bin/domainname example.sun.com... success start: /usr/lib/ldap/ldap_cachemgr... success start: /etc/init.d/autofs start... success start: /etc/init.d/nscd start... success start: /etc/init.d/sendmail start... success System successfully configured ldapclnt2# 4. Verify ldapclnt2$ldapclient list NS_LDAP_FILE_VERSION= 2.0 NS_LDAP_BINDDN= cn=proxyagent,ou=profile,dc=example,dc=sun,dc=com NS_LDAP_BINDPASSWD= {NS1}6849886b43e612a6 NS_LDAP_SEARCH_BASEDN= dc=example,dc=sun,dc=com NS_LDAP_AUTH= simple NS_LDAP_SERVER_PREF= 129.158.233.109 NS_LDAP_PROFILE= ssd_profile NS_LDAP_CREDENTIAL_LEVEL= proxy NS_LDAP_SERVICE_SEARCH_DESC= passwd:ou=people,dc=example,dc=sun,dc=com?one ldapclnt2$ ldaplist passwd lists correct entries. Example 2: Following is an example profile using 2 way search. $cat ssd_prof1.ldif dn: cn=ssd_prof1,ou=profile,dc=example,dc=sun,dc=com ObjectClass: top ObjectClass: DUAConfigProfile defaultSearchBase: dc=example,dc=sun,dc=com authenticationMethod: simple preferredServerList: 129.158.233.109 cn: ssd_prof1 credentialLevel: proxy serviceSearchDescriptor: passwd:ou=people,dc=example,dc=sun,dc=com?one;ou=people,dc=sun,dc=com?one $ Follow the same procedure as above and verify that ldaplist passwd dumps user entries correctly. 2.6. How to Setup and Implement RBAC? There is already an existing Info Doc on the subject: Info Doc 70152 How to load RBAC configuration data into a Native LDAP datastore http://sunsolve.sun.com/search/document.do?assetkey=1-9-70152-1&searchclause=70152 2.7. How to Setup and Implement SMC? Here is a brief description of how to achieve this: - Creating the Name Service Domain Tool box - Register the LDAP administrative credentials with SMC as root - Verify communications with the Sun Java (LDAP) Directory Server - Create the tool box for managing LDAP within SMC using the scope - Start the console and select your new LDAP tool box. 2.8. How to Setup and Implement Automounter? There is already an existing internal Info Doc on the subject: Info Doc 47323 Solaris[TM] 8 and 9 Native LDAP clients and Automount Maps http://sunsolve.sun.com/search/document.do?assetkey=47323 and few things worth knowing: - case sensitive mount point not working, if LDAP server is Sun ONE DS 5.1: this is bug 4630941, fixed in Sun ONE DS 5.2 (too risky to fix in 5.1) - what to use: auto.sun or auto_sun? Below is an extract from a mail on internal mail alias that explains it all: For the /product mount point the customer has: automountKey=/product,automountMapName=auto_master,dc=cs,dc=xxxxx,dc=xxx,dc=xxx objectClass: automount objectClass: top automountKey: /product automountInformation: auto.sun4 -intr,nosuid Note the "." in auto.sun4. What the automounter will now look for is a container called automountMapName=auto.sun4 _not_ automountMapName=auto_sun4. She probably has done the same thing with /net in that instead of auto_net she has auto.net in the entry automountKey=/net, automountMapName=auto_master,dc=cs,dc=xxx.... Just fix the problem with the automountInformation attribute to all the entries under the automountMapName=auto_master container which have the problem. After this, issue the command /etc/init.d/autofs stop and start on one of the clients and I thing she should be OK. I do not believe she needed to restart anything else. Bottom line is that when /etc/auto_master is imported, make sure that the maps in the source files do have the auto_xxx format, rather than auto.xxx format. E.g.: /home auto_home -intr,nosuid,nobrowse /usr/pkg auto_pkg -intr,nosuid /usr/lang auto_lang -intr,nosuid 2.9. How to use ldapaddent(1M)? Before we take a look at how to use ldapaddent(1M), we need to understand a little behind what ldapaddent(1M) is. Probably one of the most important things to remember is that ldapaddent is not a completely stand-alone utility. If you are familiar with ldapaddent(1M) you may have noticed that it does not have an option to specify which server the data should be loaded to, i.e something like a -h option in ldapadd/modify is missing in ldapaddent. This is because ldapaddent(1M) has to access the configuration from an already setup Secure LDAP client in order to talk to a Directory Server. There is no doubt that it would be desirable to ldapaddent(1M) as a totally stand alone utility, but it was not designed in that way. After the Secure LDAP Client has been initialized, you can then use the ldapaddent(1M) command to create entries in LDAP containers from their corresponding /etc files. The following example uses the -a option, which specifies the authentication method and the -f option which indicates the input file to read: # /usr/sbin/ldapaddent -D "cn=Directory Manager" -w <password> -a simple -f /etc/hosts hosts The ldapaddent(1M) command functions by using the bind DN proxy agent to search for the ou=hosts directory entry. When the ou=hosts directory entry has been located, the Directory Manager is used to bind to the directory, add host entries, and attributes for each system beneath the ou=hosts directory entry. You can use the ldapaddent(1M) command to populate the other databases, such as passwd, shadow, rpc, services, protocols, and so on. For better performance, you should use the ldapaddent(1M) command to populate the passwd information before the shadow information, networks before netmasks, and bootparams before ethers. Valid databases to the ldapaddent(1M) command are: *passwd *shadow *networks *netmasks *bootparams *ethers *aliases *group *hosts *netgroup *protocols *publickey *rpc *services Valid databases also include auto_master, auto_home, or auto_name, where the name can be any automount map name such as the auto_direct file. A syntax error occurs when using the ldapaddent(1M) command if the /etc/auto_name file contains entries that begin with a plus (+). The resulting LDAP automount information functions, but the plus (+) entry is considered a syntax error by the ldapaddent(1M) command. Using ldapaddent(1M) with NIS If your naming service data is already in a NIS server and you want to move the data to a Directory Server (LDAP) for LDAP naming services, you can use the ypcat(1) command to dump the NIS maps into files and run the ldapaddent(1M) command against those files. For example to migrate the protocols map from NIS perform the following: # cd /var/tmp # /usr/bin/ypcat protocols > protocols.txt # /usr/sbin/ldapaddent -D "cn=Directory Manager" -w <password> -f /var/tmp/protocols.txt protocols When importing the passwd data from NIS, each entry contains the encrypted password. For example: # ypcat passwd | grep mh13749 mh13749:5dJCOKb8w.4R2:14749:10:.. To process this as one file i.e. no shadow file, use the -p option and the ldapaddent(1M) command as follows. For example: # /usr/sbin/ldapaddent -p -D "cn=Directory Manager" -w <password> -f /var/tmp/passwd.txt passwd ldapaddent(1M) Performance Firstly, ldapaddent(1M) does not perform any connection management by itself. Instead, ldapaddent(1M) uses interfaces from the libsldap library, so it basically relies on libsldap and ldap_cachemgr(1M) to do the connection management. (Remember, we have to run ldapaddent(1M) from a configured Secure LDAP Client). ldapaddent(1M) reads line by line from a supplied /etc format file and tries to add the entry to the server one by one. Its all sequential, which is probably one of the reasons why the performance of ldapaddent(1M) is slow. To improve ldapaddent(1M) performance, what is required is to make ldapaddent(1M) multi-threaded so that entries can be added in parallel. Right now, the best one can do is to just have multiple terminal windows and start a ldapaddent(1M) per window but for different container(s). 2.10. How do I reinitialize a client? There might be some cases, such as 4688752, where the Native LDAP II client is broken, and one needs to set it up again. In this particular case, as explained in bug workaround and evaluation fields, you might want to try to restore missing files (e.g., /etc/nsswitch.conf) from /var/ldap/restore, and then restart ldap_cachemgr(1M) and nscd(1M). However, running again ldapclient(1M) init (or equivalent on Solaris 8, see section 2.1 and 2.2 above) is probably the best thing. 2.11. How do I un-initialize a client? This is achieved with ldapclient(1M) run as root on both Solaris 8 (with 108993-47 or later revision of patch and 108808-43) and Solaris 9. The command line option is however different between Solaris 8 and later releases. Solaris 8: /usr/sbin/ldapclient [-v] -u Example: Un-initializing a Solaris 8 LDAP client #/usr/sbin/ldapclient -v -u Arguments parsed: Handling uninit option Restoring machine to previous configuration state Stopping network services Stopping sendmail Stopping nscd Stopping autofs Stopping ldap nisd not running nis_cache not running nispasswd not running nis(yp) not running recover: stat(/var/ldap/restore/defaultdomain)=0 recover: open(/var/ldap/restore/defaultdomain) recover: read(/var/ldap/restore/defaultdomain) recover: old domainname "sunicnc.France.Sun.COM" recover: stat(/var/ldap/restore/ldap_client_file)=-1 recover: stat(/var/ldap/restore/ldap_client_cred)=-1 recover: stat(/var/ldap/restore/NIS_COLD_START)=-1 recover: stat(/var/ldap/restore/sunicnc.France.Sun.COM)=0 recover: file_move(/var/ldap/restore/sunicnc.France.Sun.COM, /var/yp/binding/sunicnc.France.Sun.COM)=0 recover: stat(/var/ldap/restore/nsswitch.conf)=0 recover: file_move(/var/ldap/restore/nsswitch.conf, /etc/nsswitch.conf)=0 recover: stat(/var/ldap/restore/defaultdomain)=0 recover: file_move(/var/ldap/restore/defaultdomain, /etc/defaultdomain)=0 Starting network services start: /usr/bin/domainname sunicnc.France.Sun.COM... success start: /usr/lib/netsvc/yp/ypstart... success start: /etc/init.d/autofs start... success start: /etc/init.d/nscd start... success start: /etc/init.d/sendmail start... success System successfully recovered Solaris 9: /usr/sbin/ldapclient [-v] uninit Example: Un-initializing a Solaris 9 LDAP client # /usr/sbin/ldapclient -v uninit Arguments parsed: Handling uninit option Restoring machine to previous configuration state Stopping network services Stopping sendmail Stopping nscd Stopping autofs Stopping ldap nisd not running nis_cache not running nispasswd not running nis(yp) not running recover: stat(/var/ldap/restore/defaultdomain)=0 recover: open(/var/ldap/restore/defaultdomain) recover: read(/var/ldap/restore/defaultdomain) recover: old domainname "sunicnc.France.Sun.COM" recover: stat(/var/ldap/restore/ldap_client_file)=-1 recover: stat(/var/ldap/restore/ldap_client_cred)=-1 recover: stat(/var/ldap/restore/NIS_COLD_START)=-1 recover: stat(/var/ldap/restore/sunicnc.France.Sun.COM)=0 recover: file_move(/var/ldap/restore/sunicnc.France.Sun.COM, /var/yp/binding/sunicnc.France.Sun.COM)=0 recover: stat(/var/ldap/restore/nsswitch.conf)=0 recover: file_move(/var/ldap/restore/nsswitch.conf, /etc/nsswitch.conf)=0 recover: stat(/var/ldap/restore/defaultdomain)=0 recover: file_move(/var/ldap/restore/defaultdomain, /etc/defaultdomain)=0 Starting network services start: /usr/bin/domainname sunicnc.France.Sun.COM... success start: /usr/lib/netsvc/yp/ypstart... success start: /etc/init.d/autofs start... success start: /etc/init.d/nscd start... success start: /etc/init.d/sendmail start... success System successfully recovered 2.12. How to setup Secure NFS?? There is already an existing Info Doc on the subject: Info Doc 72141 How to configure secure RPC when using LDAP as Naming Service? http://sunsolve.sun.com/search/document.do?assetkey=1-9-72141-1&searchclause=72141 Secure RPC hasn't been fully implemented yet in the Solaris LDAP client, because the libnsl function netname2user(3NSL) doesn't yet support ldap as a source for the publickey database. In consequence of that, the keyserv daemon is unable to convert a netname (like "unix.1234@example.com") into a user name. Therefore NFS cannot become secured by Secure RPC yet in a LDAP environment. Please see bugs 4953916 and 5106725 for details. 3.0 LDAP client Security 3.1. How to Setup and Implement SASL DIGEST-MD5? There is already an existing Info Doc on the subject: Info Doc 70546 Configuring sasl/DIGEST-MD5 for Native LDAP in Sun[TM] ONE Directory Server http://sunsolve.sun.com/search/document.do?assetkey=1-9-70546-1&searchclause=70546 Steps: 1. Ensure the system can see it's own host name in as a fully qualified host name # grep hosts /etc/nsswitch.conf hosts: dns [NOTFOUND=continue] files OR (quick & dirty) # grep lab134 /etc/hosts 10.16.50.134 lab134.commslab.dns.com lab134 loghost # # getent hosts lab134 10.16.50.134 lab134.commslab.dns.com 2. setup the (S9 bundled DS) # directoryserver setup ... administrator ID [admin]: Password: nssecret Password (again): nssecret ... Suffix [dc=commslab, dc=dns, dc=com]: dc=example,dc=com ... Directory Manager DN [cn=Directory Manager]: Password: nssecret Password (again): nssecret ... Administration Domain [commslab.dns.com]: example.com ... 3. configure the DS for LDAP Naming Service # /usr/lib/ldap/idsconfig ... ... Answer the following with YES !!! ... Do you want to store passwords in "crypt" format (y/n/h)? [n] y ... Summary of Configuration 1 Domain to serve : example.com 2 Base DN to setup : dc=example,dc=com 3 Profile name to create : default 4 Default Server List : 10.16.50.134 5 Preferred Server List : 6 Default Search Scope : one 7 Credential Level : proxy 8 Authentication Method : sasl/DIGEST-MD5 9 Enable Follow Referrals : FALSE 10 iDS Time Limit : 11 iDS Size Limit : 12 Enable crypt password storage : TRUE 13 Service Auth Method pam_ldap : 14 Service Auth Method keyserv : 15 Service Auth Method passwd-cmd: 16 Search Time Limit : 30 17 Profile Time to Live : 43200 18 Bind Limit : 10 19 Service Search Descriptors Menu ... Enter DN for proxy agent: [cn=proxyagent,ou=profile,dc=example,dc=com] Enter passwd for proxyagent: nssecret Re-enter passwd: nssecret ... Now change the passwordstoragescheme into clear: # cat > passwordstoragescheme.ldif dn: cn=config changetype: modify replace: passwordstoragescheme passwordstoragescheme: clear # ldapmodify -h lab134 -D "cn=directory manager" -w nssecret \ -f passwordstoragescheme.ldif modifying entry cn=config # directoryserver stop # directoryserver -s lab134 vlvindex -n userRoot -T example.com.getgrent # directoryserver -s lab134 vlvindex -n userRoot -T example.com.gethostent # directoryserver -s lab134 vlvindex -n userRoot -T example.com.getnetent # directoryserver -s lab134 vlvindex -n userRoot -T example.com.getpwent # directoryserver -s lab134 vlvindex -n userRoot -T example.com.getrpcent # directoryserver -s lab134 vlvindex -n userRoot -T example.com.getspent 4. change Passwords into cleartext NOTE: the DS was stopped in step 3. and still needs to be down when performing this step. For "cn=Directory Manager": # cd /var/ds5/slapd-lab134/config # vi dse.ldif change the line ... nsslapd-rootpw: {SSHA}Sata+Zn2dBOkMVbiB7XqQDDszDhfcnJ4FT8MBg== to ... nsslapd-rootpw: {clear}nssecret ... Restart the DS # directoryserver start For the proxyagent: It is SSHA before: # ldapsearch -h lab134 -D "cn=directory manager" -w nssecret \ -b ou=profile,dc=example,dc=com cn=proxyagent \ cn=proxyagent,ou=profile,dc=example,dc=com \ cn=proxyagent sn=proxyagent objectClass=top objectClass=person \ userPassword={SSHA}uNU3CmDQ01hbFj9gxzyB5At1+I2OSacPwwGVEQ== # cat > /tmp/proxy.ldif dn: cn=proxyagent,ou=profile,dc=example,dc=com changetype: modify replace: userPassword userpassword: {clear}nssecret # ldapmodify -h lab134 -D "cn=directory manager" -w nssecret \ -f /tmp/proxy.ldif modifying entry cn=proxyagent,ou=profile,dc=example,dc=com ... and is CLEAR now: # ldapsearch -h lab134 -D "cn=directory manager" -w nssecret -b ou=profile,dc=example,dc=com cn=proxyagent \ cn=proxyagent,ou=profile,dc=example,dc=com cn=proxyagent \ sn=proxyagent objectClass=top objectClass=person \ userPassword={clear}nssecret Test, if proxyagent can BIND (and read it's own entry) # ldapsearch -h lab134 -D "cn=proxyagent,ou=profile,dc=example,dc=com" -w nssecret -b ou=profile,dc=example,dc=com cn=proxyagent \ cn=proxyagent,ou=profile,dc=example,dc=com cn=proxyagent sn=proxyagent \ objectClass=top objectClass=person userPassword={clear}nssecret 5. configure a S9 client: # ldapclient -v init -a proxydn=cn=proxyagent,ou=profile,dc=example,dc=com -a proxypassword=nssecret -a domainname=example.com 10.16.50.134 ... System successfully configured Verify it works: # ldaplist | more dn: cn=Directory Administrators, dc=example,dc=com dn: ou=People, dc=example,dc=com dn: ou=Special Users,dc=example,dc=com dn: ou=Groups, dc=example,dc=com dn: ou=group,dc=example,dc=com dn: ou=rpc,dc=example,dc=com Verify it has configured SASL/Digest-MD5: # cat /var/ldap/ldap_client_file NS_LDAP_FILE_VERSION= 2.0 NS_LDAP_SERVERS= 10.16.50.134 NS_LDAP_SEARCH_BASEDN= dc=example,dc=com NS_LDAP_AUTH= sasl/DIGEST-MD5 NS_LDAP_SEARCH_REF= FALSE NS_LDAP_SEARCH_SCOPE= one NS_LDAP_SEARCH_TIME= 30 NS_LDAP_CACHETTL= 43200 NS_LDAP_PROFILE= default NS_LDAP_CREDENTIAL_LEVEL= proxy NS_LDAP_BIND_TIME= 10 Verify it does really use SASL: On the server # tail -f /var/ds5/slapd-lab134/logs/access and check the logs related to the ldap client. On the client # ldaplist and check that it lists the naming information stored on directory server. 3.2. How to Setup and Implement TLSv1/SSL? There is already an existing Info doc on the subject: Info Doc 73098 Solaris[TM]:Configuration example to secure a LDAP Naming Service environment with TLSv1/SSL http://sunsolve.sun.com/search/document.do?assetkey=1-9-73098-1 4.0 Debugging LDAP 4.1. How to start debugging, if something fails? What to do first? Basic troubleshooting techniques are covered in http://docs.sun.com/app/docs/doc/806-4077/6jd6blbfq?a=view Few basic things to check: * is ldap_cachemgr(1M) running ? * is nsswitch.conf(4) configured for ldap? * can you ping(1M) the LDAP server ? using hostname? * is ldaplist(1M) working ? * messages on the console or on /var/adm/messages * get the current profile by running /usr/sbin/ldapclient list (or on Solaris 8, /usr/sbin/ldapclient -l) Some less trivial things to check: * if connection to the LDAP server is failing, you might want to check that: o proxy agent password is correct o proxy agent account is not locked, and proxy agent password has not expired (this is to be checked on the LDAP server) * if Client Credential Level is anonymous, are all the useful data in the LDAP server readable by anyone(check ACL's) ? * the LDAP server errors and access files. See 4.7 below. * the LDAP data sent and received. See 4.3 to 4.4 below. 4.2. How to use snoop to debug the LDAP Client? You would use the ldap keyword of snoop(1M). E.g.: # snoop -i /tmp/snoop.out ldap 1 0.00000 ldapclnt -> ldapsrv LDAP C port=33823 2 0.00002 ldapsrv -> ldapclnt LDAP R port=33823 3 0.00022 ldapclnt -> ldapsrv LDAP C port=33823 4 0.00024 ldapclnt -> ldapsrv LDAP C port=33823 Bind Request 5 0.00026 ldapsrv -> ldapclnt LDAP R port=33823 6 0.00423 ldapsrv -> ldapclnt LDAP R port=33823 Bind Response Success 7 0.00013 ldapclnt -> ldapsrv LDAP C port=33823 8 0.00065 ldapclnt -> ldapsrv LDAP C port=33823 Search Request derefAlways ... The -v option gives even more data: # snoop -i /tmp/snoop.out -v ldap .... TCP: LDAP: ----- Lightweight Directory Access Protocol Header ----- LDAP: *[LDAPMessage] LDAP: [Message ID] LDAP: Operation *[APPL 0: Bind Request] LDAP: [Version] LDAP: [Object Name] LDAP: cn=proxyagent,ou=profile,dc=exam LDAP: ple,dc=com LDAP: Authentication: Simple [0] LDAP: proxy12 LDAP: ETHER: ----- Ether Header ----- ETHER: ..... Please note that this is useful only if the LDAP port that used is 389, the default LDAP port. snoop(1M) is not able to manage LDAP traffic on other ports. Ethereal, available from http://www.sunfreeware.com/, can also be used to capture network traffic along with snoop. 4.3. How to debug PAM? PAM trace and debug mechanism uses syslogd(1M) and you would hence need to customize syslog.conf(4). PAM debugging is explained in detail in Infodoc 72189 How to Enable Debugging in PAM (Pluggable Authentication Module) http://sunsolve.sun.com/search/document.do?assetkey=1-9-72189-1&searchclause=72189 4.4. How to understand the Sun Java Directory Server v5.2 access log? It is all well documented in Sun Java Directory Server 5.2 Reference Manual, chapter 8: Access Logs and Connection Codes. Check it out from http://docs.sun.com: 816-6699-10 4.5. How to use ssltap to debug TLSv1/SSL communication? Info Doc 73098 covers a little bit in this regard. The Mozilla web site provides the so called "NSS Security Tools", which is a very helpful collection of tools, that can be used when debugging or troubleshooting problems in a SSL/TLSv1 configuration. The source code of these tools may be obtained from http://www.mozilla.org/projects/security/pki/nss/tools The tool ssltap can be used to capture the SSL communication between two systems. This tool is, of course, unable to decrypt data, t (2005-06-20 12:59:14.0) Permalink Comments [3]
Trackback URL: http://blogs.sun.com/raja/entry/native_ldap_psd
Post a Comment: |
Calendar
RSS Feeds
All /General /Music /Solaris /Sports SearchLinks
NavigationReferersToday's Page Hits: 48 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Posted by rajeev on August 01, 2005 at 09:14 PM PDT #
Posted by Mike on October 14, 2005 at 04:04 PM PDT #
Time to update your blog, stop working from home!
Posted by Wellwisher on February 28, 2009 at 04:22 PM PST #