People typically do distinguish between arguments that are valid and not valid. A valid argument is one in which the truth of the conclusion is a logical consequence of the truth of the premises. Since fallacies fail to meet this criterion, they're thought of as arguments that are not valid.