Cleared the library 6 times and went to bed. Woke up, checked in and there he was. He was the roamer spawning on the 2nd floor close to the tome keeper specter guy.
Don't know how long he's been there but it doesn't seem like he depops. Some people claim he depoped after dropping aggro, which is possible but I've seen a lot of funny business with the spawns in that area, like regular mobs just straight up vanishing out of thin air for no reason or the tortured librarian depopping into the spectral librarian, depoping into a spectral guardian so if something is up, I would advise against messing around and getting straight to business.