I rechecked the issue, and found out that the link is only in the list of posts, and NOT in the post when I open it.
If you want to look at the site, goto "drlaporte.fr/WordPress3/"
You have an example under the tab "Calendrier" (this is a category, and the pane simply lists all posts with this category set). There are 2 simple posts, with only text inside.
The first one you have "Commentaires fermés" (Closed comments), because in the article I both de-activated the comments and the retrolinks.
The second one, you have "Pas de commentaires" (No comments), because in the article I only de-activated the comments.
If you open any of the two posts, you have no longer the link. But it is still in the category post list, and I have not found a setting to disable that.