Waldheim was the Scarborough, New York residence of James Speyer, a New York banker and public and financial figure. The estate was located on a hill between the Ossining and Scarborough train stations. The estate's name means "Forest Home" and its lands occupied a large tract of the property between Scarborough Road and Albany Post Road. Today, the estate's red-brick wall is still visible on the borders of Scarborough and Holbrook roads going as far south as where the entrance is to Philips Laboratories.
By 1901, the Waldheim estate covered approximately 100 acres that held large meadows, a small lake, orchards, gardens of vegetables and flowers, and portions of woods that were left undisturbed. The decorative elements of the gardens of the house were not very elaborate by 1901, consisting only of a hedge that encircled the garden on the west side of the house that faced the Hudson, which had as a centerpiece a beautiful fountain of bronze which was in a basin cut from a single piece of red marble.
The structures on the estate by 1901 consisted of a lodge made of white stone near where the red-brick wall was broken by the main gateway. The only other outbuildings of the estate as of 1901 was the stable to the north of the manor house, and a nearby tall windmill. Waldheim itself was large with a wide veranda on three sides, high windows and several gables, all of which looked out over the meadows and gardens of the estate. The house was built in wood and stucco, and it looked as if it was made more for comfort than for elaborate display. Everything about this house and grounds, from its fences, stables, winding paths, roads, and grounds showed perfect maintenance.
When compared to other local wealthy landowners in the Briarcliff Manor-Scarborough area, James Speyer conducted Waldheim in a more "aristocratic European manner". One example of this is that he had a footman positioned behind every chair of each of his guests at his dinner table.
Despite the absence of both display and decoration, which suggested a house of "severe simplicity," the house was still apparently "luxuriously furnished, and every apartment is decorated in perfect taste." As far as the rooms of the house were concerned, the entrance hall of the house was the main apartment, which was completed with lounges, fireplace, easy chairs, and comfortable corners, which implied perhaps that James Speyer combined the functions of the old-fashioned sitting room and drawing room. Located to the right of the front hallway was the library, which had a view over the lawn, and was furnished in olive green; and its books on its shelves appeared worn as of 1901, and this room ultimate lacked the "severe order" which was apparent in most libraries for show of this time. Adjoining the library was the dining room, which was furnished in the Colonial style, complete with a frieze of tapestry, which formed a relief to the overall light tone in the room: the whole effect of this room was noted as "artistic and simple."