John Pitt, 2nd Earl of Chatham, by Charles Turner after John Hoppner 
John Pitt, the 2nd Earl of Chatham, and William Pitt the younger's older brother, lived at No. 6 (now No. 47) Berkeley Square in the 1780s. William Pitt often stayed with his brother at this residence before he moved in to Downing Street in the spring of 1784. In fact, when Pitt became First Lord of the Treasury at the age of 24 in December 1783 he was residing there.
A Google Street View image of No. 6 (now No. 47)
It was from this address that William Pitt, accompanied by his brother and Lord Mahon, went to Grocers Hall in February 1784 to receive the Freedom of the City. Unfortunately, their carriage was attacked on the way back when they went up St. James's Street, and they were forced to flee into White's gentleman's club in order to escape their attackers.
A photo I took of No. 6 (47) from the vantage point of the square
Of interest, just three doors away from this residence is 50 Berkeley Square - well-known for the reputation of being London's most haunted house! Pitt's friend and political adherent George Canning once lived at No. 50 (formerly No. 3). Apparently, from the history of the house, it is very difficult to let it out! Whether or not No. 50 is actually haunted is a different matter, but it certainly makes for a compelling reason to visit Berkeley Square!
No. 50 Berkeley Square
1. John Pitt, 2nd Earl of Chatham, by Charles Turner after John Hoppner, National Portrait Gallery, NPG D20092.