Do language models lexicalize phrases?

Do langauge models “lexicalize” certain multi-token words or phrases (i.e., treat them as atomic units)? How would we measure lexicalization in LMs?