Originally 'The uppermost rom or cabin of any note upon the stern of a ship. and it is the proper place for the Master; and the deck or cover over it is named the Poop of the ship' (Boteler's Dialogues, 1630)
Roundhouse is defined in Blanckey's dictionary (1740s) as spaces under the poop. Smaller ships at that time had small cabins right aft, with the poop serving as the deckhead. Larger East Indiamen and other merchantmen had a roundhouse ahead of those cabins (Smyth, Sailor's Word Book, 1867); frequently named the coach in ships of war ('place set apart for the convenience of the officers' Hamersley, Naval Encyclodaedia, 1884).
For a short period in the early 18th century, the very largest warships had a 'poop royal' above the poop with the space below it being, by extension, the 'topgallant roundhouse'. It was used for the accommodation of the Master and some of the Lieutenants - the rest of whom would have had cabins off the wardroom, at the after end of the middle gundeck (Lavery, Ship of the Line, vol 1, 1983).
The roundhouse, always square or rectangular, was so named because the bulkhead at its forward end was rounded (McEwen and Lewis, Encyclopedia of Nautical Knowledge, 1953) or because it was possible to walk 'round' the cabin (Smyth, op cit; Kemp, Ships and the Sea, 1976)
More recently, it was apparently applied to a cabin on deck, sometimes just abaft the mainmast, e.g. accommodation for apprentices (the 'halfdeck' in English ships - McEwen and Lewis, op cit).
A "texas" roundhouse is mentioned on some early 20th century Great Lake steamers (example: the FAIRFAX ex-IONIA). This would appear to be a cabin structure aft of the pilot house. The "texas" on western river steamboats was the accomodation built on the hurricane deck for the officers, and eventually, when by the 1850s the texas extended for about one third the length of the vessel aft of the chimneys, the cabin passengers. [Hunter, Louis C, Steamboats of the western rivers, Harvard UP, 1949]. The origin of the term would be from the naming of staterooms for the states, and at the time that Texas was 'annexed', this was the only state name available for added accommodation [Cincinnati Gazette, 11 Oct 1846]
N.B. the best known roundhouse may be that of the brig "Covenant" of Dysart, Captain Hoseason, described by Robert Louis Stevenson for Alan Breck's fight in "Kidnapped":
The round-house was built very strong, to support the breaching of the seas. Of its five apertures, only the skylight and the two doors were large enough for the passage of a man. The doors, besides, could be drawn close: they were of stout oak, and ran in grooves, and were fitted with hooks to keep them either shut or open, as the need arose.