I was trying to trace back further from my four-times-great-grandfather Jobe Henry West who was a brewer. He was born in 1802 in Chatham and died in 1880 in Croydon. His parents were John and Eleanor West. Apparently a drawing of Eleanor exists, on the back of which reads 'daughter of a former Duke of Norfolk, her mother was Lady Barrymore'. I put the query about Eleanor here because I think she was born in Finsbury in the late 1760s. Her maiden name might have been Farrel, and there's evidence that an Eleanor Farrel spent her childhood at the Duke's estate - note that the spelling may have been different. Some say that the estate in question was Arundel Castle, but I think considering the time and place of her birth, it would've been Norfolk House in the City of Westminster borough. I think it's likely that Charles Howard (the 11th Duke) was Eleanor's alleged father - it would've made sense, as Charles definitely had a number of illegitimate children. There are many shreds of stories to support this theory, some of them hard evidence, some mere speculation.
He didn't succeed to the dukedom until 1786, but he did restore Arundel after it had fallen into disrepair over the preceding couple of centuries. This would explain how Arundel came into the story. Another piece of speculation that seems to be solid is that the figure in the drawing has a very prominent and bulbous chin, something she shares with the portrait of Charles Howard. There's apparently a story about a Duke of Norfolk and a Lady Barrymore going to Gretna Green.
An interesting side-note: According to the family of Jobe's sister Sarah, their father John West was the son of a peer, having large estates in Middlesex, notably in Camden Town, around and including Red Cap Tavern. It is unknown who that peer was if this particular story is true. John and Eleanor supposedly met 'in the Norfolk household'.