A nested transaction is a database transaction that is started by an instruction within the scope of an already started transaction.
Nested transactions are implemented differently in different databases. However, they have in common that the changes are not made visible to any unrelated transactions until the outermost transaction has committed. This means that a commit in an inner transaction does not necessarily persist updates to the database.
In some databases, changes made by the nested transaction are not seen by the 'host' transaction until the nested transaction is committed. According to some, this follows from the isolation property of transactions.
The capability to handle nested transactions properly is a prerequisite for true component-based application architectures. In a component-based encapsulated architecture, nested transactions can occur without the programmer knowing it. A component function may or may not contain a database transaction (this is the encapsulated secret of the component. See Information hiding). If a call to such a component function is made inside a BEGIN - COMMIT bracket, nested transactions occur. Since popular databases like MySQL do not allow nesting BEGIN - COMMIT brackets, a framework or a transaction monitor is needed to handle this. When we speak about nested transactions, it should be made clear that this feature is DBMS dependent and is not available for all databases.
Theory for nested transactions is similar to the theory for flat transactions.
The banking industry usually processes financial transactions using open nested transactions, which is a looser variant of the nested transaction model that provides higher performance while accepting the accompanying trade-offs of inconsistency.