If someone has your IP address and your PC is vulnerable (not patched, not configured correctly, not protected with security software, not behind a firewall) they simply need to browse your files and look for personal information about you. If every possible avenue of intrusion is taken care of, you are 80% protected. Why only 80%? Because you can have all the techical gizmos in the world, it only takes one human mistake (like opening a trojan attachement in an email) to unleash the hounds.

It's like asking, "If someone knows where I live can they discover my name?" Are your doors and windows locked? Yes? Good. Is there any personal information in your garbage can on the curb? Yes? Well..... so much for the locks.....