Yeah, it’s a data point it’d be useful not to have to count on. But, then again, it’s only the one. And it mostly serves as a data point only, not as an entry point. It’s not possible to (normally) access anywhjere near all that data — in particular, the chats — primarily via the phone number, so in as much as it’s about privacy, privacy is preserved (note however: not increased). Signal’s intended use is for privacy, not anonymity.