Background of Open Source
rinna is an AI development company founded in June 2020 as a spin-out of the team that worked on AI chatbots in Microsoft's AI & Research divisionIt is.
We have presented technologies such as the "Empathetic Chat Model," which uses deep learning technology to enable AI to automatically generate context-sensitive conversational text for natural conversations with humans, and the "Speech-to-Speech System," which enables AI to express rich emotions through spoken and sung voices.
These state-of-the-art technologies have been applied to the development of our AI chatbot "Rinna" and "AI Character," an AI chatbot with customized conversational content and voice expressions that give it character, for use in corporate marketing and other applications.
In the process of experimenting with natural language processing (NLP) for product development, we have now built a large-scale language model of GPT-2 specifically for Japanese.To contribute to the Japanese NLP research community, the language models developed and the training code for researchers to reproduce the experimental results on their own machines are available as open source on GitHub and HuggingFace, an NLP model library.I will do so.
- HuggingFace:. https://huggingface.co/rinna/japanese-gpt2-medium
- GitHub:. https://github.com/rinnakk/japanese-gpt2
What are the features of the Japanese GPT-2 model?
Language models are based on machine learning of linguistic data, and are used to model conversational and writtenModel "human-like language" as a probability.The model is the one that is used to estimate the probability of the language. A good language model is one that can accurately estimate probabilities. For example,"Probability (I am a cat) > Probability (I am a cat)."The ability of the language model to estimate the
GPT-2 is a language model (autoregressive language model) that computes sentence probabilities from word-level probability combinations. For example,"Probability (I am a cat) = Probability (I am) x Probability (is|I am) x Probability (cat|I am, is) x Probability (in|I am, is, cat) x Probability (is|I am, is, cat, in)"The estimation is performed in a manner similar to that of the GPT-2. Using this capability, GPT-2 can automatically generate a sentence given the prefix "I am a cat" by selecting "is" as the next word from the probability estimation.
The Japanese GPT-2 model released by rinna, Inc,Automatic generation of advanced Japanese text with common Japanese text featuresThe user/researcher can Users/researchers can also create their own models from this model by fine-tuning specific textual data.
For example, when the sentence "Together, to the heights of artificial intelligence that no one has reached yet" is given as a Prefix,Generate response sentences in specific contexts (Demo 1: Impressions of a lecture, Demo 2: Introduction to a book)(*1).
Demonstration 1] Sentence generation in the context of lecture impressions

Demo 2] Sentence generation in the context of a book introduction

(*1) The above demo sets the maximum number of characters for the text to be generated, not the actual full text to be generated.
Features of rinna's Japanese GPT-2 model
rinna's Japanese GPT-2 model has the following features
- As training data, the CC-100 (http://data.statmt.org/cc-100/) of open source data.
- Using a Tesla V100 GPU, 70 gigabytes of Japanese text was trained over an extended period of approximately one month. As a result, the model achieved a performance of approximately 18 perplexity (*2).Models are well trained and versatileIt can be said that
- For easy access by users/researchers,Publish developed models on HuggingFaceWe do this. In addition, to enable researchers to reproduce our results on their own machines, we have developed a toPublished the training code on GitHub.The company is doing so.
- The model we have released is a medium size model defined as GPT2-medium. In the future, so that users/researchers can make the best choice based on the trade-off between performance and cost,Models of different sizes will also be availableIt is. We also plan to release new models trained on different data.
(Â 18 perplexity refers to the performance of the GPT-2 model in predicting the next word from a previously given word, leaving only 18 options that contain the correct one.
Future Development
The large-scale language models being developed by rinna's research team are already widely used in our products. We will continue to research and develop more accurate and larger-scale language models that include different text styles and different amounts of data,Increase the capabilities of AI chatbots.We will continue to work on this project. We will also open source these models for the Japanese language research community.
Company Profile
Company name: rinna Corporation
LocationWeWork: 39F WeWork, Shibuya Scramble Square, 2-24-12 Shibuya, Shibuya-ku, Tokyo
Date of establishment:June 17, 2020
Representative Director:Jean "Cliff" Chen
Web site:https://corp.rinna.co.jp/
Business Description:Research, planning, development, operation and sales of AI services









