A submatrix is a matrix formed by taking certain rows and columns from a bigger matrix.
For example
There is no standard way to denote a submatrix - although the above two methods are the most common - so one should be careful while reading an article on matrix theory.
The corresponding concept in determinant theory is of minor determinant, that is, determinant of a square submatrix.