The mechanisms of the formation and evolution of dislocation cellular structures in single crystals and polycrystals are investigated using correlation instability in a dislocation ensemble. It is shown that at the nonlinear stage of the system evolution, a cellular structure develops, characterized by both total and excess dislocation densities. At a certain stage in the dislocation ensemble evolution, equilibrium configurations form: configurations with excess density represent sub-boundaries, and those with total density represent dislocation dipole configurations that annihilate via the dislocation climb mechanism. It is shown that the occurrence of sub-boundaries is associated with rotational instability caused by the formation of walls in the regions of maximum values of excessive dislocation density in a cellular dissipative structure. Upon unloading, the dissipative structure is destroyed, and the remaining dislocations migrate into sub-boundaries, leaving behind an equilibrium "framework" of the subgrain structure.