Just a note about being careful to look at the actual census records rather than transcripts. The actual 1851 census records list Robert Noble and his first wife Margaret as "Americans" which someone transcribed as "African-American". A later census, 1871, lists them as "English". Family history does maintain that Robert was a son of John Noble of the Maryland Loyalists rather than one of the all-Black companies of loyalists.
Best wishes with the research!