Autoregressive Generative Models

We have already encountered autoregressive models, when we were dealing with decoder-based language models, like GPT. An autoregressive model relies on its previously generated values. If for example you want to produce the fifth word in a sentence, you provide the model with the previous four words.

Autogenerative models can also be used to generate images. We simply generate one pixel at a time, while conditioning the model on previously generated pixels.

The interactive example above shows how pixel generation looks like. To generate the next (red) pixel, the model looks at all previous (blue) pixels. As you can imagine this recursive process is quite slow and scales quite badly with increased image size. While autoregressive procedures are state of the art for text generation, for image generation there are more efficient techniques, that we will introduce in future chapters. This chapter is merely an intorduction into the world of generative models.

The goal of a generative model is to estimate the probability distribution p(\mathbf{x}) undefined . \mathbf{x} undefined could by a piece of text, a song or in our case a full image. Once we have learned this distrubution, we can draw random images from it.

For autoregressive generative models we rewrite this task by using the chain rule of probabilities and express the probability distribution as the product of conditional probabilities.

p(\mathbf{x}) = p(x_1) * p(x_2 | x_1) * p(x_2 | x_1, x_2) * \cdots * p(x_n | x_1, \cdots, x_{n-1}) undefined

Essentially we are answering the following question: 'What is the probability of the next pixel, given that I have observed all those previous pixels?'. Learning the conditional distribution is often much easier, than learning the joint distribution. Imagine you have been provided with a half finished image, filling in the blanks should be relatively straightforward. Even though you do not know how the image looks like exactly, you can manage this task much better than creating an image from scratch. Even if you are not an artist, if we give you an image, where only the last pixel is missing, you will be able to fill in the blanks.