Literary fiction is a somewhat uneasy term that has come into common usage since around 1970, principally to distinguish 'serious' fiction from the many types of genre fiction and popular fiction. For example, a traditional first novel is supposed not to be science fiction, nor a detective story, but with literary content usually partly autobiographical.
Literary fiction includes works written as short story, novella, novel and novel sequence. Of these, the novella is relatively uncommon in English literature, and more important in German literature or Russian literature. There is no particular reason that forms should be so limited; other categories could include the novelette, and the graphic novel as represented by a work such as Jimmy Corrigan, the Smartest Kid on Earth.
The distinction has its artificial side, in the sense that magical realism counts as literary, while fantasy writing is excluded; the dividing line cannot be accurately drawn on the basis of content alone, and has to include style as a consideration. Literary prizes usually concern themselves with literary fiction, and their shortlists can give a working definition.
It has become a commonplace that 'literary fiction' is in itself just another genre. This accords with the marketing practices now general in the book trade. It may also be taken to be the latest version of the death of the novel debate that has run from 1950, and reflects at one remove the importance accorded the novel as it replaced poetry as the central literary form in Western Europe and North America from the 1930s.