Compliments to the Complementizer: Indirect Routing of Syntactic Structure in a Transformer Language Model

Unlike recurrent neural networks, which have a fixed path for accumulating information from the input, transformer language models' use of attention accommodates more flexible regimens for how information is routed through the model. In this paper we investigate how syntactic information about long-distance filler-gap dependencies is routed in a transformer language model. We focus specifically on German sentences involving "verb-second" structures. In such sentences, a constituent (often a noun phrase or prepositional phrase) is displaced from its base position to the left edge of the clause, and immediately followed by the finite verb, which occupies the syntactic position of a complementizer. This establishes a long-distance filler-gap dependency between the fronted element and the verb on which it depends. A language model will need to encode information about the fronted element and move it to downstream positions in order to modulate expectations about possible continuations. Through interchange interventions on contrastive minimal sentence pairs, we identify that transformers systematically make use of an indirect path that routes relevant syntactic information first from the fronted element to the complementizer position and from there to the downstream position at which predictions are made. This non-trivial detour of information through the complementizer aligns with linguistic theories of the structure of long-distance filler-gap dependencies, and is echoed in the treatment of abstractly similar dependencies in English and Italian. We then explore the content of the model's internal representations by training a hierarchical Tensor Product Decomposition Network (TPDN) that identifies the model's representation of grammatical features. We can then reconstruct model activations at the complementizer position under the assumption that this neurosymbolic representation captures the essential aspects of the model's encoding of grammatical structure. Causal interventions using these reconstructions confirm that this manipulation of the grammatical features produces the predicted changes in downstream continuation predictions.