The irrelevant speech effect refers to the degradation of serial recall when speech sounds are presented, even if the list items are presented visually. The sounds need not be a language the participant understands, nor even a real language; human speech sounds are sufficient to produce this effect.
Interference hypothesis
It is possible that the visual stimuli (the list of items) is held in working memory as a phonological code. The phonological loop is composed of the articulatory rehearsal loop and the phonological store. If that is the case, the irrelevant speech could interfere with the articulatory rehearsal process, degrading the information in the phonological store. This would result in degraded performance on trials where irrelevant speech is presented. However, the effect should be greater for words rehearsed longer (i.e. presented earlier in the series) since they are rehearsed more often, hence having more opportunity for degradation.