With the "Deus Ex Mach­ina? — Test­ing AI Tools"-series we want to show you dif­fer­ent tools, that aim to sim­pli­fy writ­ing, design and research by using Arti­fi­cial Intel­li­gence. More on the "Deus Ex Machina?"-series can be found here.

Overview

Last March, New York-based com­pany  Run­way released the second gen­er­a­tion of its online tool Run­way Research. Sim­il­ar to the text-to-image tools Mid­jour­ney and DALL‑E, Run­way Research Gen­er­a­tion 2 gen­er­ates new images and videos from text inputs and image templates.

In addi­tion to the browser-based video edit­or, which is strongly remin­is­cent of Adobe's Premi­er Cut or the free edit­ing pro­gramme DaV­inci, there are also usable tem­plates for audi­ovisu­al clips and present­a­tions. These can be per­son­al­ized with stock mater­i­al or your own record­ings. The edit­or does not offer any func­tions for post-pro­cessing videos in terms of their bright­ness and col­our; it is more suit­able for arran­ging and cut­ting fin­ished clips and bring­ing them togeth­er with text and audio.

Func­tions to fur­ther edit the clips are offered by the vari­ous AI Magic Tools, which can be opened in new tabs.  These func­tions can be used to auto­mat­ic­ally add sub­titles to your own videos, optim­ize the sound or cor­rect the col­or. In addi­tion, objects and people can be removed or sep­ar­ated from the background. 

All the func­tions men­tioned up to this point can be used after stat­ing an e‑mail address. With the free sub­scrip­tion, the num­ber of total seconds that can be used for edit­ing the mov­ing image are severely lim­ited and there are restric­tions in the qual­ity of the export, as well as a lim­ited num­ber of gen­er­a­tion attempts. The tool can be upgraded with paid sub­scrip­tions, which cost either $12 or $28 a month and allow for high­er qual­ity res­ults with no time lim­it on down­load­able media.

Also free to use and in tra­di­tion with the first gen­er­a­tion of the tool is the func­tion to gen­er­ate videos with a trained intel­li­gence. Using an image or text as a prompt, new videos can be com­piled and com­pletely new scenes can be cre­ated and visu­al effects incor­por­ated by using already exist­ing videos. Those who have seen this year's sev­en-time Oscar win­ner Everything Every­where All at Once may remem­ber the scene with the two stones, whose move­ments were anim­ated using this tool (among oth­ers). In an inter­view pub­lished on Runway's web­site, Evan Hadleck, one of the film's visu­al effect artists, talks about how the tool helps him pro­duce music videos and commercials.

The AI behind the application

In addi­tion to this inter­view, Run­way not only pub­lished oth­er reports on film­makers who use their tool, but also refers to vari­ous papers with inform­a­tion on the pro­gram­ming and devel­op­ment of the tool. These are freely access­ible via the Cor­nell Uni­ver­sity web­site. How­ever, the doc­u­ments on Run­way Gen­er­a­tion 1 provide little inform­a­tion on how it works, espe­cially for those with little to no pre­vi­ous pro­gram­ming knowledge.

The paper on Gen­er­a­tion 2 is said to be pub­lished later this year and will com­mu­nic­ate the devel­op­ment and train­ing of the tool. In the six months since pub­lic­a­tion, how­ever, no such inform­a­tion has been pub­lished and no inform­a­tion on the applic­a­tion and its train­ing can be found. It is also not known yet wheth­er the videos and images uploaded by users are used to train Gen‑2 and may appear in oth­er users' work (as of August 2023).

The rhetorical potential of the tool

More explan­a­tion-friendly, on the oth­er hand, are the tutori­als that can be found on You­Tube for the indi­vidu­al func­tions. On a sep­ar­ate chan­nel, sev­er­al videos show how to use the tools and present the "next-gen­er­a­tion con­tent cre­ation with arti­fi­cial intel­li­gence". The motto is often com­bined with the prom­ise of sup­port­ing cre­at­ive work and redu­cing costs for the pro­duc­tion of visu­al media, as Designs.ai can be used to cre­ate entire films without the need for the often expens­ive record­ing of film material.

How­ever, this requires some prac­tice and patience, as authen­t­ic-look­ing res­ults are not always guar­an­teed and depend on source mater­i­al and prompt. Rep­res­ent­a­tions of non-exist­ent people in a video work less con­vin­cingly, as the example video of the lib­rary cor­ridor shows. With the addi­tion of the per­son, the video loses real­ity (see video 2). For abstract gim­micks, on the oth­er hand, it is worth try­ing out dif­fer­ent com­mands and set­tings (see video 3).

Changes to a video through a source image usu­ally work quite reli­ably. Chan­ging a blue sky to an impress­ive col­our spec­tacle, as in the fol­low­ing example, works in a few clicks.

Finally, the tool can also imit­ate well-known "looks" through descript­ive text input, which (should) trig­ger cer­tain asso­ci­ations in the tar­get audi­ence. For example, enter­ing "Wes Ander­son Style" res­ults in col­or-heavy sug­ges­tions that are cer­tainly remin­is­cent of the director's detailed and archi­tec­tur­al signature.

Usage in science communication

The struc­ture of the video edit­or is simple and intu­it­ive to use. It allows basic edit­ing in cut, text and audio. The edit­ing tem­plates offer visu­ally appeal­ing designs that can eas­ily be filled with your own texts and videos. This can be espe­cially help­ful for those inex­per­i­enced with edit­ing, as they are easy to use and provide reli­able res­ults. This is also where the tool's poten­tial lies, that it can also sup­port inex­per­i­enced users in the pro­duc­tion of audi­ovisu­al con­tent. And even if Gen‑2 is primar­ily inten­ded for film­makers, it can also be used in sci­ence com­mu­nic­a­tion to visu­al­ize com­plex con­tent, for example, to cre­ate attract­ive present­a­tions or to cut short clips.

How­ever, try­ing to fur­ther adapt applied tem­plates or to change indi­vidu­al edit­ing steps of the AI can get tricky. This also applies to oth­er func­tions in which the AI takes over entire edit­ing steps after just one click. Wheth­er remov­ing noise in the sound or col­or cor­rect­ing videos: There is no log that shows the changes made by the AI and thus there is no way to pre­cisely track the pro­grams edit­ing steps. Moreover, gen­er­ated changes can­not be weakened, strengthened or only applied in parts. It is pos­sible to reset the edit­ing and gen­er­ate it again with changed para­met­ers, but this is time-con­sum­ing and costs one gen­er­a­tion each time, which are lim­ited in the free account anyway.

A self-exper­i­ment with the blue but­ter­fly, which was pro­cessed with the image-to-image func­tion, showed how many gen­er­a­tions may be neces­sary until the desired res­ult is achieved. The com­bin­a­tion of image and prompt was sup­posed to cre­ate a new image accord­ing to the specifications.

The attempt to place the but­ter­fly on a horse by prompt was not suc­cess­ful. Instead of a second anim­al, the AI gen­er­ated a horse pas­ture and a stable in the image back­ground. The recol­or­a­tion of the but­ter­fly worked well — even though it was not asked for in the prompt.

The "Infin­ite Image" func­tion, in which images are sup­ple­men­ted by tex­tu­al descrip­tions of indi­vidu­al objects, often deliv­ers con­vin­cing res­ults. Although there are still minor logic errors such as false shad­ows and partly sur­real com­pos­i­tions, most of the res­ults are almost cred­ible, espe­cially in com­par­is­on to the gen­er­ated videos.

What was part of the ori­gin­al pic­ture can still be determ­ined quite well here, because the added objects show incon­sist­en­cies. For more real­ist­ic addi­tions, for example of the dog, post-pro­cessing of light, shad­ow, col­or and high­lights would be help­ful, but these are not pos­sible with the tool at the moment. Instead, sev­er­al images can be out­put per prompt, from which the best image can be selected.

Due to the mul­ti­tude of dif­fer­ent func­tions, the tool offers the most diverse pos­sib­il­it­ies of use in the pro­duc­tion of audi­ovisu­al con­tent, whereby the cre­ation of arti­fi­cial pho­to­graphs and videos in par­tic­u­lar is in the fore­ground. The image manip­u­la­tion func­tions make it pos­sible to illus­trate ideas quickly, as authen­t­ic res­ults are gen­er­ated through text or image input. Espe­cially in the com­mu­nic­a­tion of sci­entif­ic con­tent, this opens up new pos­sib­il­it­ies for visu­al­iz­ing con­tent. For example, cli­mate research­ers could use the tool to draw atten­tion to the con­sequences of cli­mate change by gen­er­at­ing photo-real­ist­ic images that already show the eco­lo­gic­al effects of a rise in tem­per­at­ure. In par­tic­u­lar, social media con­tri­bu­tions or present­a­tions can be sup­ple­men­ted with appeal­ing visualizations.

Wrap-Up

Even if some res­ults are still expand­able, the tool offers a quick and easy way to visu­al­ize design ideas in a new way. It can shorten pro­duc­tion pro­cesses and allows edit­ing and image manip­u­la­tion without the need for pri­or tech­nic­al know­ledge. Of par­tic­u­lar interest for use in sci­ence com­mu­nic­a­tion could be the visu­al­iz­a­tion of abstract or future scen­ari­os, such as the impact of cli­mate change on our eco­sys­tem. How­ever, not all pro­jects suc­ceed right away, which is why work­ing with Gen‑2 by Run­way Research requires a lot of tri­al and error, and espe­cially per­son­al­iz­ing and adapt­ing the res­ults requires some patience and cre­ativ­ity in choos­ing the right prompts. The simple oper­a­tion invites exactly that and the gen­er­ated res­ults are fun, even if a good res­ult is not guar­an­teed with every gimmick.