Plots having triangular, round shape, hexagon, octagon or polygon shapes should be avoided because these are not good for construction. Instead plot should be of square or rectangular shape. Ratio of width and length should not be more than 1:2 in rectangular plot. The square plot is the best
For more detail read artilce at link below:
Vastu tips for plot
Octagonal Shape: The room with eight corners in a circular shape symbolizes the closed circuit. It helps an individual to trace the path of success without obstacles. The octagonal shape ensures good health and calm life.
But I assume you want Octagon shape of entire house......?