First place in the AppStore: how Ukrainians created an application that Ilon Musk uses

Reface can already be called one of the most successful mobile applications from Ukrainian developers. The technology based in Kiev makes it possible in a matter of seconds to synthesize video using artificial neural networks: for example, combining user faces with memes in video and GIF formats, writes The Village Ukraine.

Collage: ForumDaily, photo: screenshots reface.app, YouTube / Dave Business

Thanks to Reface, Tesla and SpaceX co-founder Elon Musk merged his face with the body of actor Dwayne Johnson and shared the results with more than 38 million followers on Twitter. After that, Reface only grew: if in January, when the application was launched (then called Doublicat), it attracted a million users, by August more than 20 million people were armed with it. Over the last summer month, Reface has almost doubled - about 15 million more attracted users and 35 million downloads, the founders said.

The Reface project became the first Ukrainian-made application, which took first place in the top of the American AppStore, ahead of TikTok, Netflix, Hulu and Amazon applications. At the beginning of September, it is in the top 20 in the general ratings of free applications in the AppStore and Google Play Market in the United States. Forbes, The Verge and Mashable have already written about him.

Reface continues to gain an audience: according to developers, the total number of users should grow to hundreds of millions in the coming year. At the same time, Reface says that at the moment they have provided users with no more than 10% of the total potential of the technology. In the fall they plan to launch Reface Studio, a full-fledged video editor based on technology for fast processing of synthesized videos. Next year, the Full Body Swap function is expected to be launched - the ability to replace not only the face, but the user’s entire body. Reface also has more global plans: they want to use the technology not only for memes, but also to change the idea of ​​personalization on the Internet and rethink all content created by humanity.

Reface's Kiev office has over 50 employees and continues to recruit the technical team. The co-founders of the company are Roman Mogilny, Oles Petriv, Yaroslav Boyko, Dmitry Shvets, Ivan Altsibeev, Denis Dmitrenko and Kirill Sigida. The Village Ukraine spoke about technology, application and responsibility with Reface CEO Roman Mogilny and CTO Oles Petriv.

— We can say that the history of Reface was divided into two periods - before and after Elon Musk’s tweet. What did you think the moment he shared your app with nearly 40 million followers?

Roman Mogilny: It was something special. We, of course, understood that we were doing something more than just a fan. They expected that people would appreciate it, that the stars would pick it up. But to immediately “get into Elon Musk” - we did not expect this. For us and the technology business in general, this is a super cool visionary. This is a person who does everything we dream of as children: cars, spaceships...

Therefore, of course, we were inspired. But I can say that this state did not last long: since our watermarks were on the images, new users began to massively visit us.

— How big was the jump in users?

Mogilny: Hundreds of times. I had to expand the structure of cloud servers, and do it quickly and efficiently.

- That was the moment when you realized - we did it?

Mogilny: We have always believed in what we do. I think this is what has always distinguished us. Even in previous projects, when things were going tight. Even when they failed [laughing]. And here we believed.

And [Musk's tweet] was the first validation - yes, we are on the right track. There were different validations at different stages of the company. But when you work almost around the clock and try to give your all, moments like these give you the strength to move on.

How Reface works and how the technology was invented

— You mentioned failures in previous projects. Can you talk about the low points? Previously, you worked with both website creation and post-production for Hollywood studios. How did you use negative experiences when building Reface?

Mogilny: Of course, there were many failures. We had an interesting model: we both worked for outsourcing and prepared our own products and services. But, as you can see, we now have only one working service. Therefore, yes, at different stages we closed various projects. We didn't find what is called product / market fit, realized it, switched and moved on.

By the time we started Reface, we only had one application running. We decided to leave it and focus on a new product. In things like this, focus is extremely important - choosing a priority and devoting time to it. Even within the same project, it is very difficult to simultaneously focus on users and business clients. Therefore, for us now, B2C is the main story, and B2B is an additional one.

Oles Petriv: We have tried an incredible number of different directions in machine learning. We tried to turn sentiment analysis technologies and automatic video conversion into 3D into products. We had products, we had clients. But more importantly, thanks to this experience, we received feedback from film studios: “Translating a XNUMXD image or video into XNUMXD is, of course, cool, but could you, for example, turn people of one ethnic group into another?” That is, to find solutions that could seriously cut studio costs.

Therefore, we started the experiment moving away from the main pipeline. As a result of this experiment, we realized that our development can not only solve the problem of one film studio or one film, but also simplify the process of modifying facial features; this process can be accelerated from a few weeks to a few seconds.

Mogilny: The history of our company stands out because we have always been able to enter new areas early. We started working with machine learning in 2011, long before anyone believed in this story - except perhaps scientists who worked with it at a theoretical level. Therefore, during this time, the team and Oles in particular have accumulated enough experience to enter this story.

Now many companies are trying to enter the sphere. But we even started developing the face replacement technology in 2018: we saw the first deepfake videos and realized that this is the future of personalization. Thanks to this, we have a large enough user base, which allows us to break away from the competition. We believe that in terms of technology, we have several years of lead over the competition. In the final version of our technology, we get the ability to replace a face in high quality almost instantly.

How to use Reface?

1. Download Reface: face swap videos app for iOS or Android platforms.
2. Install app.
3. Give the app access to your phone camera and take a selfie. “Every photo you take will be uploaded to our servers for processing and facial transformation. By continuing, you agree to the “Terms and Conditions of Use” and “Privacy Policy,” they specify before uploading the photo.
4. Immediately after uploading the selfie, Reface will offer to subscribe to the paid version. For $ 2,49 per week, 3,99 per month, or 24,99 per year, they offer a premium version that offers unlimited photo enhancements, no logo ads, and the ability to upload your own GIFs. You can unsubscribe: in this case, the user will receive a free version, in which you need to watch an advertisement before watching the received video.
5. Processed videos and GIFs can be stored on the phone or transmitted via links.

— How can I explain the operating principle of Reface as simply as possible? When, say, your mother or aunt asks you about this, how do you answer them?

Petriv: We have created a system that allows you to personalize any video in one click and in a few seconds.

Mogilny: And so you can become the hero of any video.

Petriv: An element of any meme, clip or any visual content where there are people.

— In your first interviews about Reface, you talked about hypothetical scenarios when studios or other customers might need tools to replace faces in the frame. We've been living in quarantine for six months now, and these scenarios have already become part of life: like, for example, replacing basketball player Damian Lillard in a Hulu commercial...

Petriv: We don't want to replace deepfake technologies. We have a fundamentally different advantage: we can not only replace one face with another, we can do it almost in real time, in a matter of seconds. In addition, in our new developments, this can be done directly on the user's mobile phone.

Mogilny: It was possible to replace one face with another a year or ten years ago.

Petriv: It just requires a lot of money, as well as large teams, years or months of time. With the advent of open deepfake repositories, these years and months turned into weeks. But still, this is weeks of work by a professional - for the sake of a few seconds of video. We can do this much faster - in one click and in a second.

Moreover, we have provided access to the use of this technology to everyone. You don't need to be an expert in machine learning, you don't need to own a super-powerful computer for several thousand dollars. And this is precisely where our competitive advantage lies.

Mogilny: We process more videos per day than during the entire history of deepfake videos.

Petriv: Many times more, tens and hundreds of times. As of June, approximately 200 deepfake videos had been created. Of these, 000 are videos specially generated by Google and Facebook, created for training in recognizing fakes. Today our users create about 120 videos per day.

Personalization

— You say the main purpose of Reface is personalization. What exactly do you mean?

Mogilny: We believe this is the future of personalization - the way people show themselves online.

Ivan Altsibeev, product director of Reface: “Thanks to personalization, we can change not social networks themselves, but influencer marketing as such. It is now gaining great momentum: influencers are no longer just people who post content, they unite micro-communities around themselves. These include people who do not want to read the news of the whole world; they are only interested in what truly resonates. Media is fading into the background, people are becoming a new form of media.

Reface encourages influencers to create content that others will want to be a part of. That is, it will change the level of user engagement. People love to look at themselves in the mirror because our brains are so trained. Therefore, we are not competing with other applications, we are competing with human imagination. When the user looks at the influencer, he imagines how he would look in this car or in this environment. In this sense, personalization helps us quickly move to one or another context. It creates a strong emotional connection.

We came up with the name Reface as a verb that simultaneously describes the essence of the technology. We wanted to become a company, a technology that changes a niche so much that at some point we begin to be associated with it, just as the words “Google” (Google) or “photocopy” (Xerox) once entered people’s dictionaries. Previously, this seemed like something fantastic, but now that the application has become viral, we are increasingly seeing a new verb in users’ social networks recover«.

— Is the function of complete body replacement one of the planned ones? When are you going to introduce it?

Mogilny: Next year.

On the subject: Two Ukrainian startups entered the ranking of the best US employers: the history of their creation

— How difficult is it from a technology point of view? How many more variables are there to interact with?

Petriv: It is much more complicated, there are many times more variables. Facial features have their own characteristics, but it is still a certain limited number of possible options.

Mogilny: If we are talking about the body, then these are different physiques, incredible variability of clothes, behavior and the like.

Petriv: I see three big phases in content development. The first stage is from the beginning of book printing to the end of the television era. At this stage there was a clear division: the authors and editors of the content, as well as the audience consuming it; the only thing that has changed over these few centuries is content formats and audience sizes.

The next fundamental step, in my opinion, was the invention of computer games. This is a new type of content product, in which, in addition to a set of images, plots and visual patterns, there are certain degrees of freedom. An element of interactivity, interaction with this world. The process of content consumption was already fundamentally different from what was created earlier: a thousand people who play the same game can get a thousand different experiences.

We believe the next step will be to combine the game's interactivity pattern with the content that existed before. This is what we call content recontextualization: relatively speaking, you can take any movie and make it interactive; it will develop, for example, depending on which country you are from, what you like, who your friends are, what your mood is and what you ate today. A million people who watch this new kind of content will have a million different experiences. Thus, any experience of perceiving content can be turned into a process of creating a new one.

Fakes and responsibility

— Another possible chronology is before and after the advent of social media, when traditional media lost their monopoly on the dissemination of information. Accordingly, people have become both creators and audiences of content, while limiting the role of media as intermediaries. This opens up many opportunities, but also threats. In these circumstances, the responsibility of developers and publishers is critical.

Mogilny: We believe that everything depends on the final goal of the developers. There are many apps in the market whose purpose is to create a money making factory. We defined the idea and vision for ourselves as “moving towards something more.”

We understand that with so many users we need to be responsible. In addition, as a company we operate in the USA, so we treat the issue even more strictly. It is important for me that I can finally say to myself: we have done everything possible to ensure that users feel comfortable and safe; that we did not give this technology as something negative for humanity. There is a high probability that we will be the ones who will manage to commoditize it (turn the product into a relatively cheap consumer product. - Ed.).

Already now we are actually a media house, content holders write to us with a request for their content in the application.

We now have 35 million users and are actively growing. I think we will have hundreds of millions of users within a few years.

— You announced the latest figures in August, then you reported 20 million users. So within a month you grew by another 15 million?

Mogilny: Do not offend us either, in less than a month. [Is laughing]

In short, we treat this responsibly. And we do this not because we have to, but because we want to. Even now, when some questions arise, such as analyzing certain errors, we do not touch user data. No one will know about this anyway. But we will know about this. Therefore, the data that is sent to us is a closed box that we do not go into.

Petriv: Actually, we arranged the infrastructure in such a way that it was impossible to look under the hood. All data is stored in the cloud in the form of closed storage. Any intervention would be technically difficult.

Mogilny: Plus it is dangerous for the system.

Petriv: We don’t need to get into people's data at all. In addition, in many moments it is simply unprofitable for us to store user data, because we have to pay for storage in the cloud, a database, and the like. When it comes to millions of users every day, even keeping one entry can be overkill.

— You use Google Cloud Platform cloud storage, right?

Mogilny: Yes.

— And the physical servers are located in the USA?

Mogilny: Yes, we do not process everything on our servers.

Petriv: What for? If there is infrastructure from a global company that provides security guarantees.

— What I mean is that interference in user data can come not only from you, but also from external agents.

Mogilny: This is a basic story for us, a security story. The next step is content moderation. We are already introducing moderation.

Now we have given access to about 10% of the potential of the technology: it works either on low-quality GIFs, or on high-quality videos, limited to the set that we give. When we give 100% of the possibilities, we will have a definition of the wrong content: political speeches, porn, etc. will definitely be blocked by the system. In addition, we will limit the ability to load individuals, such as politicians, into the system to limit the harmful effects of technology.

Petriv: In addition, we spent a lot of effort on the digital watermark.

Mogilny: This is a way to label videos - it is not visible to users, but is readable by our system. We believe that if you create something that could mislead people, you should provide a tool with which the video can be verified. I think in the future there will be services that will allow you to validate a video - whether it is real or synthesized. Of course, we will work with them and provide our service. This is a serious challenge, as demonstrated by the results of the Facebook Deefpake Detection Challenge.

“And there are separate statements from Mark Zuckerberg that deepfake is an unprecedented challenge for Facebook.

Petriv: And this is not only a technical problem, but also a conceptual one. Let's say tomorrow some company announces that it has invented a way to detect synthesized videos with 100% accuracy. Let's say it's true. In less than a week, a small modification to one of the deepfake methods will appear on Github or (worse) on the darknet, and the entire authority of this verification system will collapse. It's one thing when we don't know for sure if the video is real or not. And it is completely different if it is not real, but the authoritative system calls it real. And this is the biggest problem in trying to invent a "universal antivirus". As soon as it occurs, a virus appears worse than anything that came before.

“And then the question arises about your video authentication mechanisms.”

Petriv: We offer an alternative method. We do not claim that we can recognize 100% of modified videos. Since we have control over the system that creates these videos, we train it so that it not only changes facial features, but also encodes a pattern into the final image that is very difficult to remove. And in the process we train our system to recognize these encoded patterns. Currently, the recognition system is at its final stage. We can’t vouch for others yet. But we can identify our video with a high degree of confidence. If we said that it is ours, then it is definitely ours.

On the subject: Ukrainka attracted $ 12 million investment in its startup in the USA: what kind of project

— Peter Pomerantsev (one of the leading modern researchers of the phenomenon of disinformation) says that the main threat of fakes is not even misleading, but in the oversaturation of the information background to such a level that a feeling of uncertainty is created. Producers of fakes and bot farms do not want to deceive so much as to create the feeling that it is impossible to separate truth from lies. You just don't know who or what to believe. Russia's information war in Ukraine since 2014 and the election process in the United States in 2016 have shown that we are still not ready for these challenges in the already familiar social media. Synthesized videos take the problem to an even more complex level.

Petriv: As for me, the main value of media is not that they provide new content to the public. Now anyone can give new content. Authoritative media will remain authoritative because readers understand that the information they provide can be trusted. In a world where anything can be changed without problems, such authority is easy to lose. To do this, it is enough to make mistakes several times.

I don't think this is a disaster. This is another reminder to people of the world we have been living in for a long time. The fact that people live with rose-colored glasses and take any yellow headline on Facebook at face value is not a problem with technology and Facebook.

“And this problem showed the current low level of media literacy. But the fact is that Facebook and other social media have made it much easier for misinformation to spread. And the emergence of even more advanced technologies, such as synthesized videos, risks making this problem even worse.

Petriv: And stimulate the immune system.

— We started the interview with Elon Musk’s tweet, but the same Musk calls artificial intelligence the main threat. I understand that “artificial intelligence” is a very broad and vague term. But do you agree with Musk that AI is an existential threat to humanity?

Petriv: I don’t believe that the term “artificial intelligence” currently serves any useful function. Words are useful when we all understand them the same way. Now there is a situation where the term “artificial intelligence” has been deprived of meaning, because every person understands something different by it. Some people imagine "The Terminator", others - a set of stupid algorithms, and others - video cards and burning electricity.

People have been developing tools to make mechanical work easier for thousands of years. A large number of intellectual functions that people perform can also be called mechanical: tasks of the same type that need to be repeated and that can be automated. And many people are scared that certain intellectual work can be automated. Because for some reason they believed that intellectual work is something unique and characteristic only of man. But technology shows that at least some human intelligence can be automated, transferred to external processes outside the skull. And there is nothing wrong with this, thanks to this a person has time to do work at the highest level - instead of doing dull work.

We can talk about machine learning as an industry, we can talk about solving specific engineering problems. If we talk about general artificial intelligence that may arise (and this is still in question), then I will say that we are far from even understanding what natural intelligence is. Because of the media, people get the impression that if not today, tomorrow there will be self-consciousness inside silicon, which will begin to be friends or fight with us. I am convinced that it will be very different.

— I agree that this is a vague term and in popular culture it has received the most bizarre manifestations. But I deliberately asked the question through a quote from Elon Musk, who considers artificial intelligence “the most existential threat to humanity” and gives a time period of up to five years when AI “will be able to surpass humans.”

Petriv: Using Reface as an example, we want to show that access to generative media technologies can be provided with control and minimization of potential negative use. No technology provider, be it a shovel, fork, or kitchen knife, can 100% guarantee that the user is not using it for bad purposes. But it can be done to minimize and complicate misuse.

We try to provide an example of how you can minimize potential negative use. Of the millions of videos that are created daily, 99,999% are used to increase smiles, not threaten. So far, everything looks like we succeed.

Read also on ForumDaily:

Contactless rental housing: Ukrainian startup conquers the American market

The most successful Ukrainian woman in Hollywood: what connects a Kiev woman with Al Pacino, De Niro and Jolie

Different than we imagined: what do Americans think about Ukraine

Left Google nowhere: how a Ukrainian woman created a successful startup and moved to the USA

Do you want more important and interesting news about life in the USA and immigration to America? — support us donate! Also subscribe to our page Facebook. Select the “Priority in display” option and read us first. Also, don't forget to subscribe to our РєР ° РЅР ° Р »РІ Telegram  and Instagram- there is a lot of interesting things there. And join thousands of readers ForumDaily New York — there you will find a lot of interesting and positive information about life in the metropolis.