by Rich PangIrregularly timed action potentials, or spikes, are pervasively observed in the brain activity of awake mammals. However, the role of this temporal irregularity in neural computation is still not well understood. In canonical network models irregular spiking emerges via balanced, fluctuating input currents, leading to collective responses that track inputs linearly. How networks characterized by irregular spiking could support flexible nonlinear dynamics needed for general-purpose computation remains under ongoing debate. Here we characterize the dynamics of networks whose elementary unit is not a single neuron but a small group of neurons, with distinct tunings, that compete at each timestep via a winner-take-all (WTA) interaction. While WTA has long been proposed as an elementary functional motif in the brain and represents a powerful computational primitive, how large networks of such units behave has received less investigation. We show that these networks, like classic excitatory-inhibitory balanced networks, exhibit a chaotic fluctuation-driven regime characterized by sustained irregular activity resembling realistic cortical spiking, which we interpret as a multidimensional balance spread over several competing neural populations with different tunings. We develop a mean-field theory for the network, which shows how irregular spiking sustained by time-varying input fluctuations can support flexible nonlinear collective dynamics. Using the theory we predict and verify network regimes in which input fluctuations alone yield multistability, stable sequence generation, or complex heterogeneous firing rate dynamics—three core dynamical primitives thought to underlie memory-dependent neural computation—via consistent Poisson-like spiking produced through chaos. Thus, networks of WTA units support a chaotic fluctuation-driven regime characterized by irregular spiking that can power complex nonlinear collective dynamics. This represents a new model of brain activity capable of simultaneously reproducing realistic spike trains and diverse nonlinear firing rate patterns well posed for flexible computation, and which can be trained or fit to data.