He blinks in honest surprise at how thrown Sparrow seems by his question, and, seeing nothing else for it, repeats himself.

'Why do you care if they know?'

It's not as if you have a reputation to worry about, it's not like you have to worry about breaking the law, surely they know every time you bring someone aboard the ship to dally with, you are their captain, after all...

There are a multiplicity of possible things to say to explain himself, but he remains silent. He's genuinely curious, and to accidentally offend Sparrow is not at all the way to go about getting an honest answer, if indeed there's any way at all.