There are times when hyphens do not belong in modifiers. Per diem is one of those phrases that does not need a hyphen, even when used as a modifier. So the phrase "A per diem charge" does not require a hyphen.
I would allow that error per diem, but not more.
Of course, there are other schools of thought on hyphens, and that means that there are other compound modifiers that some would hyphenate and others would not. I am sure that there are more phrases that the dictionary lists as adjectives, which would not need to be hyphenated.