Thanks to http://allisterx.blogspot.com/2008/05/additions-and-ssh-access-to-virtualbox.html and http://mydebian.blogdns.org/?p=148 , I can use ssh to access my VirtualBox guest now.
The steps are actually quite simple. Run the following 3 commands in your host terminal: (Replace MyLinux with your guest name. If there are spaces in the name, just use " to enclose whole name)
VBoxManage setextradata MyLinux "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
VBoxManage setextradata MyLinux "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
VBoxManage setextradata MyLinux "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP
Then use
VBoxManage getextradata MyLinux enumerate
to verify. At last, remember to shutdown the guest OS, restart VirtualBox, then run the guest OS.
Now you can use $ssh username@hostname -p 2222 to access the guest :)