The best way to handle nervousness is to march right in during daylight hours, bringing a friend or two and ignoring everyone else. But believe me, after years of going into every kind of porn store imaginable to do research, I'll bet cash money that if you're a woman and you go into one of the seedier shops, you'll be the only one in the store within two minutes. Our culture is so packed with sexual shame that whenever I walk into a jack shack to find a DVD, the men run for cover, leaving me free to take my sweet time shopping for my next rental.
Your local mom-and-pop video stores will have a closed back room where they keep the adult DVDs, away from the tender eyes of those under eighteen. Some stores opt to save on space and will have their video selection in large books that are kept behind the counter, containing box covers in a sort of big flipbook. If you don't see an annex for the porn, or a room with western-style swinging doors, ask to see the porn books. Expect that the adult titles will be more expensive than what you usually pay for a rental???they are much less likely to be returned than other videos, and retailers like to have insurance against losing their stock (hence the higher prices).