My understanding is that the 6-year "look back" period begins on the date of patent expiration. This makes sense, otherwise (if as you suspect) the Notice would be a self-imposed time limitation.
The significance of the Date of Legal Notice (of infringement) is that this is the date from which the "infringer" is liable for damages for claimed infringement.
But I KNOW nuttin'!
SGE