I remember years ago having something like that in relation to housing benefit. The council was suspicious that my rent seemed too low for a whole house! (It was cheap because I'd struck a deal to rent a dump on the cheap in exchange for tarting it up.)
I've also had an out of the blue letter relating to ESA in the past that turned out to be totally innocuous.
To give you a smile, Concentrix (who no longer have a contract to deal with such matters) were in the midst of a scandal where they were accusing single mums claiming child tax credits to be cohabiting with all sorts of people, including former tenants and dead people. The most absurd was a number of investigations where staff based in England flagged up lots of single mums in Scotland living in flats above shops as cohabiting. I don't remember the name of the shops but it was a chain shop that English people like me wouldn't recognise, whereas if we found WH Smith at the same address, we'd know that it wasn't a surrepticious cohabitation problem.
The reason why I've mentioned the Concentrix thing and my housing thing is that if someone actually turns up on your doorstep, they can see that any allegations that you don't live at that address, that you're someone else, that you're cohabiting with someone you're not etc. aren't true.
A home visit also means that any documents you've got are to hand and they can look at them without them getting lost in the post.
So it's stressful, but I reckon a home visit has a better potential to sort out any problems than just an exchange of letters or phone calls.
Have you got someone who could be with you, or alternatively could be on standby near a phone in case you're distressed or struggling?