Facebook AI’s typography deepfake can scan real-life photos and emulate the text style

TextBrushStyle is an AI research project that has the potential to translate street signs and text to different languages, as well as create personalised messages and captions.

Date
17 June 2021

Share

Facebook AI has released TextStyleBrush, an AI research project that copies the style of text in a photograph, based on just a single word. This means that the user can edit and replace text in imagery, and the tool can replicate both handwritten and typographic compositions and bring them into real-world scenes.

Currently a research project, the tool could one day have the potential for personalised messages and captions, as well as translating text in images to different languages and even real-time translation of street signs using AR. It also hopes to open the dialogue around detecting misuse of this sort of technology, “such as deepfake text attacks – a critical, emerging challenge in the AI field,” as stated in the research paper by Facebook AI, written by postdoctoral researcher Praveen Krishnan and research scientist Tal Hassner.

TextStyleBrush works in a similar fashion to style brush tools in word processors, but focusing more on the text aesthetic found in images. It also uses a generator architecture based on the StyleGAN2 model, and because TextStyleBrush is generating photo-realistic text images, there are a couple of limitations. “First, StyleGAN2 is an unconditional model, meaning it generates images by sampling a random latent vector. We, however, need to control the output based on two separate sources: our desired text content and style,” states the research. “Second, we have an added challenge of the unique nature of stylised text images. Representing text styles involves a combination of global information (e.g. colour palette and spatial transformation), along with detailed, fine-scale information, like the minute variations of individual penmanship.” To combat this, the generator is conditioned to the TextStyleBrush content and style.

Another challenge is that, due to the nature of real-life photography, there are limitless variables of text styles available. To address this, TextStyleBrush uses a typeface classifier, text recogniser and “adversarial discriminator”, explains the research. “We first measure how well our generator captures the sale of input text by using a pertained typeface classification network. Separately, we use a pertained text recognition network to evaluate the content of a generated image to reflect how well the generator captured the target content. Taken together, this allows for effective self-supervision of our training.”

Head here to read the full paper, TextStyleBrush: Transfer of text aesthetics form a single example. The handwriting data set can be downloaded here, and contains real-world handwritten samples from different writers.

GalleryFacebook AI: TextStyleBrush (Copyright © Facebook AI, 2021)

Hero Header

Facebook AI: TextStyleBrush (Copyright © Facebook AI, 2021)

Share Article

Further Info

About the Author

Ayla Angelos

Ayla is a London-based freelance writer, editor and consultant specialising in art, photography, design and culture. After joining It’s Nice That in 2017 as editorial assistant, she was interim online editor in 2022/2023 and continues to work with us on a freelance basis. She has written for i-D, Dazed, AnOther, WePresent, Port, Elephant and more, and she is also the managing editor of design magazine Anima. 

It's Nice That Newsletters

Fancy a bit of It's Nice That in your inbox? Sign up to our newsletters and we'll keep you in the loop with everything good going on in the creative world.