Do not dismiss anyone as irrelevant, maybe they do not come across as an immediate benefit or a good contact because you work in one industry and they in a totally different environment, or there is no correlation or that you are not looking for the same sort of customer. People do need people to properly network and they may surprise you with their abilities to remember you, what you do AND who you are targeting as a customer.
In addition, when they ask about you, train them well. Simple key words or phrases that they can listen for.
However, time is a precious commodity and you must be diligent in how much time and energy you put forth. I give everyone respect and do my best to learn about them and remember their needs, and enjoy introducing them to possible contacts. Put together a simple procedure on how to categorize people to better remember them and keep track of them even if you do not feel an immediate fit.