Should I let these kids come over to swim?

I just got a new pool. One of those above ground pools for my 5 year old daughter and myself. It's pretty large and we like all the space. I put it in our backyard which is enclosed by a privacy fence. These two kids riding bikes saw me watering my grass and stopped to talk to me. I had seen one of them only once before when he asked me if he could walk my dog. Which I said no because my dog doesn't like strangers, but anyway they asked me if they could swim in my pool. One looks like he is maybe 7 and the other probably 11 or so. I asked them how they know I had a pool? They said, "We just saw it. So can we come over and swim?" I didn't know what to say. I don't know these kids at all not even by their first names, and they certainly don't know me. If I was there mom I wouldn't like them going over to some strangers house to swim. I asked them where they lived and they pointed to a house like a block away on my street. I said if their mother says it's okay then yes, but I said not today, maybe later. I just don't know if I should let them. I don't want to be responsible for these kids. But I don't want to be mean either. Is it inappropriate to let them? What do you think?