How Do AI Image Generators Really Work? The Hidden Secret
By Avani Agarwal · · 862 words
Topics: AI, AI Algorithms, AI Applications, AI Art, AI Art Generators, AI Content Creation, AI Creativity, AI Datasets
AI image generation is a type of generative AI used for producing visual content. Usually, based on the user’s textual input, these models produce highly efficient and relevant images.
Using mathematical analysis, it identifies and duplicates patterns within the images and illustrations. Its created content is based on preexisting content.
It involves using an ANN (Artificial Neural Network). A neural network aims to mimic the human mind and the system of neurons to think and visualise.
How does AI correlate?
The AI image generators we use are heavily trained on large datasets to recognise patterns, associate keywords, understand complex characteristics and analyse various aspects of data. During this period of training, the AI model covers a major chunk of its learning.
What all techniques are involved?
To understand the text prompts provided by users, Image Generators use NLP or Natural Language Processing. The NLP model converts the textual input of the user into a numerical format for the model to understand. These are called as numerical representations or embeddings. The NLP model also breaks down the elements mentioned in the prompt and the relationship among them. From here, GANs, or Generative Adversarial Networks,s come into play.
1. GAN Or Generative Adversarial Networks
GAN is a type of machine learning algorithm used for unsupervised learning. They are basically two neural networks pitted against each other to improve over time. It has two parts-
The generator
The discriminator
Let’s have a look at both of them.
- The generator: It creates new data by learning patterns and trends from the training dataset.
- The discriminator: It acts as a cross-checker for the generated output. It can verify if the image produced is real or fake and whether it meets the required criteria and specifications. This helps the generator improve over time
In the initial phase, the generator gives absolutely fake images that can be easily distinguished by the discriminator. Over time with training, the discriminator can no longer distinguish between fake and real images generated.
2. Neural Style Transfer (NST)
NST is an optimisation technique used to take a content image and a style reference image (texture, fusion, artistic style, etc) and blend the two together to create a unique output. Here,e the base content image is reimagined in the reference style. For example, if we take an image of an apple and reimagine it in the style of paintbrush strokes,s we will get the following output.
At higher levels, NST is based on CNNs or Convolutional Neural Networks. It employs techniques to extract the style of one image and apply it to the other. It uses deep layering. The outer layers extract low-level specifications, while the deeper layers are more inclined towards the extraction of abstract features to maximise the quality of the output. It uses this method to preserve content-specific details as well.
After all style adjustments, the content image is also adjusted pixel-wise to ensure a smooth blend of style and content-specific details. NST has three major components for this
Content loss (measures the difference in content between the original and generated image).
Style loss (measures the difference in stylebetween thef second and output image)
Total Variation loss (associated with smooth blending of images)
3. Diffusion Models
They are essentially based on generative AI. These generative models work in two stages, a forward diffusion stage and a reverse diffusion stage: first, they slightly change the input data by adding some noise, and then they try to undo these changes to get back to the original data. This process of making changes and then undoing them helps generate realistic images.
It uses an iterative approach to adding noise to the image. In these models,s Gaussian noise is added step by step to images to convert them into completely distorted images. The noise is then removed step by step to give a high-resolution output. This process of adding and removing noise is done for sharpness, high quality, and finer details.
Famous image-generation AI tools like DALL-E2, Midjourney & Stable Diffusion are based on this algorithm.
4. Variational Autoencoders (VAEs)
Variational Autoencoders (VAEs) represent generative models integrating neural networks with probabilistic approaches to learn efficient data representations.
VAEs are essentially two-part networks:
Encoder: The encoder of a VAE takes input data and transforms it into a latent space by generating parameters that represent the mean and variance of a probability distribution across latent variables. (also based on CNN)
Decoder: The decoder samples from the latent distribution to reconstruct the original data. This allows the generation of new data similar to the original ones. (also based on CNN)
Here, the latent space refers to the selection of a random point in the range rather than a fixed point; this randomness allows the generation of slightly different versions of the data, a useful for generating new, realistic examples.
The sample point is taken using the reparameterization technique.VAE’s loss function consists of two parts:
Reconstruction Loss (It measures the difference between the input image provided to the encoder and the reconstructed image generated by the decoder)
and KL Divergence Loss (It helps in preventing overfitting by ensuring that the learned latent space is close to a unit Gaussian Distribution).