In seismically active regions, buildings are inevitably exposed to extreme ground motions. Traditionally, the main structural system is designed iteratively to resist these loads, which provides safe systems, but is usually suboptimal. Topology optimization provides an approach to obtain optimal material layout; however, most approaches only accommodate deterministic loads. Moreover, typical structural design goals require minimization of the maximum of some set of responses; such a goal is typically non-smooth, which impairs the use of efficient gradient-based optimizers. This study models the stochastic ground excitation as a zero-mean filtered white noise and combined with the model of the structure to form an augmented system. The structural response stationary covariances are obtained by solving a corresponding Lyapunov equation. The optimization problem is formulated to minimize the maximum structural response covariances, employing equivalent smooth formulations. Dynamic condensation is also employed to increase the efficiency. Sensitivities are computed by solving an adjoint Lyapunov equation, allowing for a gradient-based solver to be used. This study implements the following building features: additional discrete floor masses, boundary elements, and floor diaphragms. The proposed strategy is illustrated for seismically excited buildings with different properties. The results presented herein demonstrate the efficacy of this approach for efficient topology optimization of buildings subjected to stochastic ground motion.