The Rose Bay house was built in 1859 by Robert Tooth (1821–1893), one of three brothers of the well known Sydney Tooth's Brewery family. It was named after a village in Kent, England from where the Tooth Family originated. His brother Frederick, also inherited adjoining land and on this he built his house called "Buckhurst".
In 1864 Tooth sold Cranbrook to Robert Towns who was a successful Sydney merchant, shipping company owner, the founder of the city of Townsville, Queensland and a Member of the New South Wales Legislative Council. He was married to Sophia Wentworth (sister to William Wentworth) and had two sons and three daughters. When Towns died in 1873, Cranbrook was sold to James White.
James White was a pastoralist and racehorse owner. He was born in 1828 in New South Wales to relatively wealthy parents. In 1853 in partnership with his two brothers he bought Belltrees near Scone and expanded it. In about 1860 James bought Matindale near Muswellbrook where he lived with his wife Emily (née Arndell) until 1873. During this time he was a Member of the New South Wales Legislative Assembly representing the Upper Hunter. He resigned from parliament in 1868.
In 1873 he bought Cranbrook and employed John Horbury Hunt,a prominent architect, to carry out large extensions on this property. He then moved in with his wife Emily and brought into his house expensive art treasures including pictures by Italian German and English painters. In 1874 he was appointed to the New South Wales Legislative Council and over the next decade held many important posts in the Societies of that day.
White's art treasures were described in detail in a publication of that time as follows;
"The interior posses<1-- sic? --> many costly art treasures - rare china of Dresden, Meissen, Sevres, and Vienna manufacture, also from Worcester and the potteries of England. In the drawing room there may be seen a magnificent specimen of ivory carving - a female figure carved from a block of ivory fully two feet long. Pictures by Italian, German and English artists decorate the walls."
These art treasures are shown below in photos of the hallway and drawing room.
One of White's main interests was horseracing and shortly after he bought Cranbrook he also purchased Kirkham near Camden where he developed his horse stud. John Horbury Hunt was commissioned to build two houses for him on this property.
One of the buildings constructed by Horbury Hunt while he was employed at Cranbrook was a stone house called "Cranbrook Cottage". It appears that this was originally on White's land and was owned by him because when Cranbrook was put on the market in 1898 the advertisement stated that there were two lots to be sold. Lot 2 was described as follows:
"At the intersection of New South Head Road and Victoria Road is a stone built cottage residence at present occupied by Horbury Hunt Esq. containing hall dining and drawing rooms, 2 bedrooms, servant's room, kitchen, bathroom, boy's room and library."
Some time after this Horbury Hunt must have purchased the property as he sold it in his own name in 1902.
While James and Emily White owned Cranbrook the gardens and grounds were also developed. Horbury Hunt built a tennis pavilion (see photo below left and also below in section on garden fete) which still exists today. It is the Rotunda at Cranbrook School.
A publication made the following comments about the grounds.
"The terrace in front which shuts out the high road is a very happy arrangement, and thanks to the well-stocked glass and bush houses a continuous supply of flowering plants is maintained. The Gardens and grounds are in fine order, and the view from the water side is charming."
Some of these features are shown in the photos below.
James White died of heart disease at Cranbrook in 1890. They had no children so Emily inherited both Cranbrook and Kirkham. Six years later in 1896 Emily who was then 62 married Captain William Scott who was aged 37. He was a veterinary surgeon in the NSW Defence Forces. In the following year Emily and Captain Scott sailed to Scotland but unfortunately Emily died shortly after arriving and was buried at Melrose, Roxburghshire in Scotland. Scott inherited all her property and the following year 1898 put both Cranbrook and Kirkham on the market. He sold Kirkham in the same year but did not sell Cranbrook.
William Harvey Chapman Lovely, an attorney of the South Kalgurli Gold Mines Ltd and his wife Florence Emily May nee Anderson (daughter of George Anderson and Emma Jane nee Bell) lived in Cranbrook, Rose Bay from c1899 to 1900 (likely rented). On the Monday, July 10, 1899, "Mrs Harvey Lovely gave an "at home" at her residence, Cranbrook, Rose Bay and despite the inclemency of the weather, a large number of guests were present... entertainment provided by Miss Kate Rooney, Miss Esmond (Mrs Skrine), the Misses Paterson, Miss Lilian Frost, Mr Burns Walker and Mr Grist. Those invited included Sir Matthew and Lady Harris, Miss Harris, Major-General and Mrs French, Sir William P. Manning and Lady Manning, Mr and Mrs A. Allen, Colonel and Mrs Lassetter, Mr and Mrs Teece, Dr and Mrs Newmarch, Cr and Mrs Hamilton Marshall, Mr and Mrs Macarthur, Mr and Mrs Macdonald, Mr and Mrs H. Levy, Mr and Mrs Mongomerie Servern, Mrs Acheson, Miss Plunkett, Major and Mrs Lee, Mr and Mrs J. L. Mullins, Mr and Mrs G. Cruikshank, Mr and Mrs Noyes, Mr and Mrs Walker, the captain and officers of H.M.S. Katoomba, H.M.S. Goldfinch and Pylades.."
Their stay in Cranbrook was short as by 10 March 1900, William sold up all household goods in "consequence to his departure to Europe". Whilst we find that Florence departed on 11 April 1900 for England on the H.M.S. Omrah, she returned to Australia and set up home in Randwick where their daughter Gwendolyne Harvey Lovely was born on 22 November 1900 and later died 5 months on 16 April 1901.
William and Florence separated and later divorced in South Africa according to the South African Divorce Archives. William married Trixie Linnelle Whiteman and raised their family in Queensland. According to newspaper reports William went out on his own rather than accept any position under a South African company deciding not to remain in South Africa but set up home in Winton, Queensland. More can be found on William Harvey Chapman Lovely through a book "History of West Australia" by Warren Bert Kimberly pg 118. 
In the lead-up to the federation of Australia, there was intense rivalry between Sydney and Melbourne to be the national capital. Eventually a compromise was reached that the federal capital would be established somewhere in rural New South Wales (which would eventually become Canberra) and that, in the interim, federal parliament would meet in Melbourne and the Governor-General would have his residence in Melbourne. Many in Sydney were not convinced that the federal capital would ever be established and that Melbourne would become the national capital by default to the social and economic disadvantage of Sydney. It was seen as vital to give Sydney a greater role in the new federation by proposing that the Governor-General should have a residence in Sydney and Government House was seen as the most fitting residence to be offered to the Governor-General (noting that the Victorian Government was also offering their Government House in Melbourne). Eventually the compromise was reached that the Governor-General would have official residences in both Sydney and Melbourne with the intention that the Melbourne Government House would be used when parliament was sitting and that the Sydney Government House would be used when the parliament was not sitting.
With the matter of the Governor-General's residence resolved, the New South Wales government leased Cranbrook from Captain Scott for five years to be used as the residence of the Governor of New South Wales. In October 1906 the Government attempted to extend the lease but Captain Scott was unwilling, preferring to sell or subdivide the property. Eventually Captain Scott agreed to extend the lease for one further year but at much higher rent, after which the Government purchased the property for £22,500.
Cranbrook served as Government House for New South Wales from 1901 to 1917. Three Governors resided there during this time with their families. Sir Harry Rawson was Governor from 1902 to 1909. His wife, Lady Florence Rawson, (pictured below) died during this time in 1905 and his daughter Alice (pictured below) resumed her mother's duties as hostess at official functions. In 1909 Frederic Thesiger, 1st Viscount Chelmsford (Frederic John Napier) became Governor and he and his wife Lady Frances Chelmsford (pictured below) resided at Cranbrook until 1913. Sir Gerald Strickland was Governor from 1913 until 1917. His wife was Lady Edeline Strickland (pictured below).
While Cranbrook was the Governor's residence many notable functions were held there. One that received press coverage was the garden fete held in 1903 in aid of the District Nursing Association. One publication described it in detail as follows.
"Friday last was the most perfect day we have had this season and the world and his wife took advantage of the sunshine to visit Cranbrook, Rose Bay, to assist in making a success of the fete in aid of the District Nursing Association. Never had the grounds of the State Government House appeared to greater advantage. The view from the top paths and terraces was truly magnificent, for the harbour was almost as blue as the sky. There were flowers everywhere, the garden beds were gay, and the bright gowns of the many visitors added spots of colour to a delightful scene.
Upon the asphalt tennis court small tables for afternoon tea were set, each table enhanced by the addition of its vase of flowers and the whole was presided over by a bevy of pretty girls in white frocks and black picture hats their distinguishing colours being huge bows of pale blue satin ribbons placed on the corsage. Down the hill towards the Rose Bay Road some dozen marquees were erected and these were furnished with gaily-draped stalls laden with the most attractive articles tempting to the would-be purchaser.
Miss Rawson with the assistance of Miss Fanshawe and Miss Suttor, did a great trade in lamps and candle shades; while in the flowerstall Miss Fairfax, Mrs Geoffrey Fairfax and Mrs Ewan Frazer sent away their customers laden with sweet-smelling boronia, violets, jonquils, and attractive looking pot plants."
Some of the photos of the occasion are shown below.
One of the largest official functions held at Government House was the garden party of over 3500 guests for the American Fleet in 1908. This function was widely reported in the press and the Sydney Morning Herald gave it a great deal of coverage. Some of it is outlined as follows.
"The late rain had made everything look so beautifully fresh and green that seldom if ever have the picturesque grounds of Cranbrook been seen to greater advantage. The harbour made a most fascinating foreground. The water was so calm that it looked like a huge sheet of glass with the stately American ships lying quietly at anchor, most of them being within view from the lawn. His Excellency and Miss Rawson received their guests on the verandah where each visitor was announced by Captain Wilson A.D.C, Mr Miller being also in attendance.
A picturesque red and white marquee was erected at the spot chosen for the guests to enter from whence they passed up to the verandah and then on to the lawn strolling later down to the lower tennis lawn. Here a band was stationed and small tables were placed about and refreshment served. On the upper lawn refreshment were served in a large marquee, the top of which was draped inside with pale blue and white (his Excellency's colours), and a band stationed on the verandah played a bright selection of music including many familiar American airs."
In June 1911, the Government of New South Wales rather abruptly announced that it intended to consolidate its use of buildings and, as part of that, wished to use the Government House in Macquarie Street as a library or museum, rather than extend the lease (which was terminating in August 1911) with the Commonwealth Government for use as the residence of the Governor-General. It was noted that the Commonwealth did not pay any rent for the property. The New South Wales Government agreed to extend the lease for a few months owing to the imminent arrival of the new Governor-General Thomas Denman, 3rd Baron Denman to provide initial accommodation for him; however the Commonwealth Government did not see any point in installing the new Governor-General into a very short-term residence and sought a longer commitment. The New South Wales Government rejected any longer lease, despite not appearing to have any immediate intentions to proceed with the development of the library or museum.
In 1913, the newly established Royal Australian Navy took over responsibility for Australia's naval defence (previously provided by the British Navy). As a consequence of this, a dispute developed between the New South Wales Government and the Commonwealth Government over the ownership of various naval facilities including Admiralty House, Kirribilli, the residence of head of the navy. The New South Wales Government had previously given them to the British Government for naval purposes and with the withdrawal of the British Navy believed the ownership reverted to them, whereas the Commonwealth Government believed they belonged to the Commonwealth from the taking over of naval responsibilities from the British Navy.
In 1917 the NSW Government put Cranbrook up for auction. It was purchased by Mr Samuel Hordern on behalf of a group of men from the Church of England who had decided to utilise it as a school. In July 1918 Cranbrook School was officially opened and is still operating from that site at the present time (2016).