Coming from a French aristocratic background, Silvy established himself as one of the leading portrait photographers in London. He moved to London in 1859 and bought Caldesi and Montecchi's studio on Porchester Terrace, which became renowned for its tasteful furnishings. Except for Queen Victoria, he photographed all of the Royal Family and most of the British aristocracy. By 1864, aged twenty-six, he had forty employees. He kept daybooks with the details of each sitter, the date, and a file print, and these are now in the National Portrait Gallery's archives. Silvy published his portraits as The Beauties of England. In 1868 he retired due to ill health, and returned to France.