I'll add a bit to this,
Actually enforced delivery to the cardholder address is completely optional and entirely down to the retailer.
The reason most do it is because of something called Chargeback
Basically in recent years the responsibility for card fraud protection has fallen in the hands of the retailer, this is particularly true for "Card Holder not present" transactions such as those made online or over the telephone.
Within 60 days of a transaction on your card you have the ability to dispute it. If you do this a chargeback notification is sent to the retailer.
The retailer then has a limited window of time (14 days I think) to prove that they actually supplied the goods or services to the card holder, if they are unable to do this then the funds are automatically withdrawn from the retailers account (you have to sign into allowing the merchant to have this ability) and refunded to your card account.
This gets to be a lot of fun for the retailer, imagine trying to get proof of delivery from a Courier 59 days after they made the drop....it's not easy...I know.
For this reason retailers are encouraged to use all of the anti-fraud measures available, including verifying both the expiry and the security digits, only delivering to the card holder address, verifying the card holder address against the card, consignee signatures only and getting proof of delivery.
Some merchants even retain the funds until the 60 day grace period has expired, this is common in high risk categories.
As a retailer you can get extra protection if you operate in a very precise way. Then the chargeback's are underwritten by an insurance company...but this extra expense is passed onto the retailer, usually in the form of higher payment surcharges. (notice how the merchant and card company have taken no responsibility for any of this)
All in all it's a pretty unfair system at the moment, as an online retailer you either take a bit of a risk or pay half your margins to the merchant...these things have stopped me setting up an e-commerce business for high value IT equipment in the past. The consumer SHOULD be protected, I'll never dispute that...but the Merchants and Card companies should take responsibility (or at least some of it) at the moment, as a retailer even if you do all the right things you can still get stung....as a consumer you are pretty much bullet proof.
Funnily enough as soon as you move into Card Holder present transactions the rules are completely different, for the sake of a simple signature (or nowadays a Pin number) the retailer gets a lot of protection.
Further more, just to put your mind at rest. Most small scale E-Commerce setups are arranged through a payment gateway and by the time you are entering card details you are doing it on the merchants web site. This way it is difficult for the E-Commerce site owner to ever see the critical details, they just get a status from the merchant (pass, fail, pending, declined) This is particularly convenient for smaller scale operations because it means that the site owner doesn't have to take responsibility for any information collected (because they are not collecting it)
Hope that helps
Wayne