LIN Jinmin ¹, ZHANG Yilai ², HU Kaihua ²

(1. School of Information Engineering, Jingdezhen Ceramic University, Jingdezhen 333403, Jiangxi, China; 2. Jiangxi Province Engineering and Technology Research Center of Ceramic Enterprise Informatization, Jingdezhen

Ceramic University, Jingdezhen 333403, Jiangxi, China)

Extended Abstract:[Background and purpose] As the essence of traditional Chinese art, traditional Chinese painting has important value in the art design of ceramic products and other daily necessities. However, the creation of traditional Chinese painting decorative patterns is highly dependent on the manual drawing of professional artists, which has the problems of high cost, limited production and insufficient composition adaptability. Traditional Chinese painting materials obtained from the Internet are often difficult to be directly applied, due to their low resolution, unclear copyright or composition that does not meet the design requirements, which restricts the diversification and personalized development of ceramic product design. At present, although image generation technology has made progress in oil painting, animation style and other fields, the generation of traditional Chinese painting style is still facing challenges. From the existing research, the most promising solution is to realize the diversity generation of ceramic decorative patterns through the Stable Diffusion model. However, most of the current generation models are trained based on Western image datasets, which are lack of adaptation to the characteristics of traditional Chinese art and have the problem of uncontrollable pattern structure. In this study, by introducing the dual-layer U-net structure and combining with the convolution enhanced low rank adaptive algorithm, the Stable Diffusion model is fine-tuned to optimize the expression of traditional Chinese painting features. At the same time, the Controlnet is used to accurately control the image frame and layout structure, so as to solve the problem of insufficient adaptability of traditional composition methods. Finally, the stylized semantic guidance strategy is constructed through the prompt engineering, while the design effect is displayed combined with the three-dimensional ceramic model. This research realizes the efficient generation and design application of traditional Chinese painting patterns, thus providing technical support for the diversification and personalized customization of ceramic products.[Methods] In this study, the dual-layer U-net structure is introduced into the Stable Diffusion model to realize the learning of traditional Chinese painting style features and image structure control. Firstly, a data set containing 200 images and corresponding text is constructed to train Lora model, which contains the sample types of mountains and waters, flowers and plants, animals and people, while each type has 50 images and corresponding text. The training set image is of 512×512 resolution. The number of batchsize is 1, each image is learned 10 times, and the max train epoches is 10. The network rank is 16, the convolution rank is 8, and the initial learning rate is 0.0001. The training adopts the prodigy optimizer, which adopts the dynamic adaptive learning rate strategy, which can quickly self-adjust and adapt to the network requirements in the training process. At the same time, this study is based on CycleGan network, using 2532 landscape photos and 2532 traditional Chinese painting pictures to train the style transfer model, which is compared with the traditional Chinese painting generation method based on fine-tuning Stable Diffusion model. Then, the control network technology is added in the process of model fine-tuning to verify the effect of image structure control. Finally, in order to explore the effect of prompt engineering on Chinese painting image generation, different prompts are used for image generation, while the generation effects are compared.[Results] From the results of text-to-picture result graph and picture-to-picture result graph, it is found that the images generated by using the method of fine-tuning Stable Diffusion model are clearer, without obvious artifacts and messy color blocks, more suitable for the style characteristics of "freehand brushwork" of traditional Chinese painting and more diverse. It can be seen from the results of dual-layer U-net generation that this method can extract the edge information of objects in the photos of ceramic products with different types, which is taken as the condition to guide image generation and realize image structure control. By comparing the generation effects of different prompts, it can be seen that adding style prompts, positive and negative prompts, the generated pattern has a more delicate and beautiful picture.[Conclusions] The method of fine-tuning the traditional Chinese painting pattern based on the Stable Diffusion model proposed in this paper can combine the traditional Chinese painting style characteristics learned from Lora model and the rich semantic information of the Stable Diffusion model to generate the traditional Chinese painting style pattern with Chinese characteristics. At the same time, Controlnet is used to control the frame and layout of the generated image, which proves the effectiveness of the dual-layer U-net structure. In addition, the quality of image generation is effectively optimized by using prompt engineering. Finally, the effect of automatically outputting traditional Chinese painting style patterns can be achieved by inputting appropriate prompt words or pictures. Users can display the generated patterns in the ceramic 3D model system to realize the diversified and personalized customization of ceramic products.

Key words: traditional Chinese painting; image generation; stable diffusion; low rank adaptation algorithm; U-net; prompt engineering