StyleCLIPDraw
The StyleCLIPDraw algorithm was proposed in StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation.
The abstract from the paper is:
Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP imagetext encoder model; however, current methods lack artistic control of the style of image to be generated. We present an approach for generating styled drawings for a given text description where a user can specify a desired drawing style using a sample image. Inspired by a theory in art that style and content are generally inseparable during the creative process, we propose a coupled approach, known here as StyleCLIPDraw, whereby the drawing is generated by optimizing for style and content simultaneously throughout the process as opposed to applying style transfer after creating content in a sequence. Based on human evaluation, the styles of images generated by StyleCLIPDraw are strongly preferred to those by the sequential approach. Although the quality of content generation degrades for certain styles, overall considering both content and style, StyleCLIPDraw is found far more preferred, indicating the importance of style, look, and feel of machine generated images to people as well as indicating that style is coupled in the drawing process itself. Our code, a demonstration, and style evaluation data are publicly available.
Example 1: 3D temple, Starry style
StyleCLIPDraw synthesizes SVG based on a text prompt and a reference image.
Synthesize A 3D rendering of a temple in the style of Starry Night by Vincent van Gogh:
$ python svg_render.py x=styleclipdraw prompt='A 3D rendering of a temple' target='./data/starry.png'
Result:
Style image |
SVG, Number of strokes: 512, style_weight: 2.0 |
Example 2: Self, Starry style
$ python svg_render.py x=styleclipdraw prompt='self' target='./data/starry.png'
Result:
|
Style image |
SVG, Number of strokes: 512, style_weight: 2.0 |