Abstract: Block-based background modeling couldn’t obtain the exact shape of foreground, while pixel-based approaches couldn’t handle non-stationary backgrounds effectively. To solve the problem, a hierarchical scheme for background modeling was presented. The hierarchical model used block-based method proposed to obtain coarse background and foreground regions firstly, and then the operations of pixel-level foreground refining and model updating based on Gaussian mixture model were performed on special regions of the input image. These two algorithms in different levels were combined by adopting an asymmetric feed-forward strategy. Experimental results show that the hierarchical method proposed can obtain the exact shape of foreground and process non-stationary scenes well, in addition, it is insensitive to illumination change and can provide better results than any single approach in it, meanwhile, the integrated computation time is shorter than the sum of those of running the block and pixel-level methods, and satisfies real-time processing.