In a recent talk at Carnegie Mellon University, American statistician and professor emeritus Edward Tufte said "Respect your audience." Pithy advice, but what does it mean? Tufte goes around the U.S. conducting seminars on information design, so presumably he directed these words at information designers, people whose primary goal is the effective communication of information. He went on to say that, in many cases, the audience knows more about the content than the designer does. For instance, designers tasked with communicating information about the federal budget to lawmakers and economists are really designing for people who (we hope) know a great deal more about the deeper meaning of all those numbers.
Respecting your audience, then, means making an effort to understand what your audience will find obvious, and what it won't. It means using a level of explicitness that is appropriate based on that understanding, rather than assuming your audience a) can read your mind or b) has no specialized knowledge of the domain. It means using language that your audience finds familiar and mental models that it recognizes.
Don't worry that your work will become less accessible to a general audience. There is no general audience, just as there is no such thing as a family with 2.5 children. Faced with the choice of being unhelpful to a lot of people or being helpful to a few, opt for the latter. If more than one specific audience must be addressed, figure out what each audience wants to get out of the information and create a design tailored to each.
Below are some examples of information design I found related to the federal budget. Just for fun(?), guess who their audiences are.