I haven’t done this before… but the only way I can think is to create a custom function and “hook” it to the filter for showing comments.
Something (in logical terms) like this…
IF comments = 0, THEN hide the display of “leave a comment”.
ELSE IF comments != 0, THEN show comment count.
Note this isn’t a real function. Rather it’s just to point you in the right direction.
You would still need to find the filter to use, and setup your function accordingly. Also, you would want to use a child theme so that your custom function is preserved during theme updates.
http://codex.wordpress.org/Plugin_API/Filter_Reference#Comment.2C_Trackback.2C_and_Ping_Filters
This is tricky… and will require a fair amount of php knowledge.