Testing AI-Tools: Runway Research Gen‑2

With the "Deus Ex Mach­ina? — Test­ing AI Tools"-series we want to show you dif­fer­ent tools, that aim to sim­pli­fy writ­ing, design and research by using Arti­fi­cial Intel­li­gence. More on the "Deus Ex Machina?"-series can be found here.


Last March, New York-based com­pany  Run­way released the second gen­er­a­tion of its online tool Run­way Research. Sim­il­ar to the text-to-image tools Mid­jour­ney and DALL‑E, Run­way Research Gen­er­a­tion 2 gen­er­ates new images and videos from text inputs and image templates.

In addi­tion to the browser-based video edit­or, which is strongly remin­is­cent of Adobe's Premi­er Cut or the free edit­ing pro­gramme DaV­inci, there are also usable tem­plates for audi­ovisu­al clips and present­a­tions. These can be per­son­al­ized with stock mater­i­al or your own record­ings. The edit­or does not offer any func­tions for post-pro­cessing videos in terms of their bright­ness and col­our; it is more suit­able for arran­ging and cut­ting fin­ished clips and bring­ing them togeth­er with text and audio.

Ein Screenshot des Video-Editors, als Übersicht über die verschiedenen Funktionen. Neben der Vorschau in der Mitte sind die verschiedenen Funktionen angeordnet: Links die Basisfunktionen wie Upload, Texteingabe, Vorlagen und rechts clipspezifische Funktionen wie Transformieren, Zuschneiden oder vom Hintergrund freistellen. Die verschiedenen Spuren eines Videos sind im unteren Bereich angeordnet und enthalten in diesem Beispiel Vorlagen von einem Schreibtisch und einem Laptop.
Image 1: The user inter­face of the browser-based video edit­or is strongly remin­is­cent of already famil­i­ar edit­ing and design tools, but dif­fers in its functions.

Func­tions to fur­ther edit the clips are offered by the vari­ous AI Magic Tools, which can be opened in new tabs.  These func­tions can be used to auto­mat­ic­ally add sub­titles to your own videos, optim­ize the sound or cor­rect the col­or. In addi­tion, objects and people can be removed or sep­ar­ated from the background. 

Ein Screenshot zur Auflistung der verschiedenen KI-getriebenen Funktionen, den sogenannten AI Magic Tools, die mit ihrer Bezeichnung und einem für die Funktion aussagekräftigen Vorschaubild kurz beschrieben werden.
Image 2: Over­view of some of Run­way Research's AI-powered func­tions for image and video editing.

All the func­tions men­tioned up to this point can be used after stat­ing an e‑mail address. With the free sub­scrip­tion, the num­ber of total seconds that can be used for edit­ing the mov­ing image are severely lim­ited and there are restric­tions in the qual­ity of the export, as well as a lim­ited num­ber of gen­er­a­tion attempts. The tool can be upgraded with paid sub­scrip­tions, which cost either $12 or $28 a month and allow for high­er qual­ity res­ults with no time lim­it on down­load­able media.

Also free to use and in tra­di­tion with the first gen­er­a­tion of the tool is the func­tion to gen­er­ate videos with a trained intel­li­gence. Using an image or text as a prompt, new videos can be com­piled and com­pletely new scenes can be cre­ated and visu­al effects incor­por­ated by using already exist­ing videos. Those who have seen this year's sev­en-time Oscar win­ner Everything Every­where All at Once may remem­ber the scene with the two stones, whose move­ments were anim­ated using this tool (among oth­ers). In an inter­view pub­lished on Runway's web­site, Evan Hadleck, one of the film's visu­al effect artists, talks about how the tool helps him pro­duce music videos and commercials.

The AI behind the application

In addi­tion to this inter­view, Run­way not only pub­lished oth­er reports on film­makers who use their tool, but also refers to vari­ous papers with inform­a­tion on the pro­gram­ming and devel­op­ment of the tool. These are freely access­ible via the Cor­nell Uni­ver­sity web­site. How­ever, the doc­u­ments on Run­way Gen­er­a­tion 1 provide little inform­a­tion on how it works, espe­cially for those with little to no pre­vi­ous pro­gram­ming knowledge.

The paper on Gen­er­a­tion 2 is said to be pub­lished later this year and will com­mu­nic­ate the devel­op­ment and train­ing of the tool. In the six months since pub­lic­a­tion, how­ever, no such inform­a­tion has been pub­lished and no inform­a­tion on the applic­a­tion and its train­ing can be found. It is also not known yet wheth­er the videos and images uploaded by users are used to train Gen‑2 and may appear in oth­er users' work (as of August 2023).

The rhetorical potential of the tool

More explan­a­tion-friendly, on the oth­er hand, are the tutori­als that can be found on You­Tube for the indi­vidu­al func­tions. On a sep­ar­ate chan­nel, sev­er­al videos show how to use the tools and present the "next-gen­er­a­tion con­tent cre­ation with arti­fi­cial intel­li­gence". The motto is often com­bined with the prom­ise of sup­port­ing cre­at­ive work and redu­cing costs for the pro­duc­tion of visu­al media, as Designs.ai can be used to cre­ate entire films without the need for the often expens­ive record­ing of film material.

How­ever, this requires some prac­tice and patience, as authen­t­ic-look­ing res­ults are not always guar­an­teed and depend on source mater­i­al and prompt. Rep­res­ent­a­tions of non-exist­ent people in a video work less con­vin­cingly, as the example video of the lib­rary cor­ridor shows. With the addi­tion of the per­son, the video loses real­ity (see video 2). For abstract gim­micks, on the oth­er hand, it is worth try­ing out dif­fer­ent com­mands and set­tings (see video 3).

Video 1: video prompt "lib­rary corridor"

Video 2: Prompt "add a person"

Video 3: Abstrac­tion of the ori­gin­al video

Changes to a video through a source image usu­ally work quite reli­ably. Chan­ging a blue sky to an impress­ive col­our spec­tacle, as in the fol­low­ing example, works in a few clicks.

Screenshot des Programms mit dem Ausgangsvideo eines Schiffs in einer blauen Lagune und rechts die Eingabefläche mit einer Aufnahme eines Himmelspektakels und unten vier Vorschlagsvideos in der Vorschau zu sehen sind.
Image 3: The out­put video of a ship in a blue lagoon on the left and the input image of an aer­i­al spec­tacle on the right, from which four sug­ges­tions were generated.

Finally, the tool can also imit­ate well-known "looks" through descript­ive text input, which (should) trig­ger cer­tain asso­ci­ations in the tar­get audi­ence. For example, enter­ing "Wes Ander­son Style" res­ults in col­or-heavy sug­ges­tions that are cer­tainly remin­is­cent of the director's detailed and archi­tec­tur­al signature.

Vier Kacheln, die jeweils einen Ausschnitt aus einer hellen und sehr farbenfrohen Bibliothek zeigen. Die Kacheln sind an den Stil des Regisseurs Wes Anderson angelehnt in Farbauswahl (kräftige, warme Farben) und Ausrichtung (Anderson ist für seine Symmetrie, klare Linienführung und Einheitlichkeit bekannt).
Image 4: Four sug­ges­tions of a lib­rary cor­ridor under the keyword "Wes Ander­son Style".

Usage in science communication

The struc­ture of the video edit­or is simple and intu­it­ive to use. It allows basic edit­ing in cut, text and audio. The edit­ing tem­plates offer visu­ally appeal­ing designs that can eas­ily be filled with your own texts and videos. This can be espe­cially help­ful for those inex­per­i­enced with edit­ing, as they are easy to use and provide reli­able res­ults. This is also where the tool's poten­tial lies, that it can also sup­port inex­per­i­enced users in the pro­duc­tion of audi­ovisu­al con­tent. And even if Gen‑2 is primar­ily inten­ded for film­makers, it can also be used in sci­ence com­mu­nic­a­tion to visu­al­ize com­plex con­tent, for example, to cre­ate attract­ive present­a­tions or to cut short clips.

How­ever, try­ing to fur­ther adapt applied tem­plates or to change indi­vidu­al edit­ing steps of the AI can get tricky. This also applies to oth­er func­tions in which the AI takes over entire edit­ing steps after just one click. Wheth­er remov­ing noise in the sound or col­or cor­rect­ing videos: There is no log that shows the changes made by the AI and thus there is no way to pre­cisely track the pro­grams edit­ing steps. Moreover, gen­er­ated changes can­not be weakened, strengthened or only applied in parts. It is pos­sible to reset the edit­ing and gen­er­ate it again with changed para­met­ers, but this is time-con­sum­ing and costs one gen­er­a­tion each time, which are lim­ited in the free account anyway.

A self-exper­i­ment with the blue but­ter­fly, which was pro­cessed with the image-to-image func­tion, showed how many gen­er­a­tions may be neces­sary until the desired res­ult is achieved. The com­bin­a­tion of image and prompt was sup­posed to cre­ate a new image accord­ing to the specifications.

Ein blauer Schmetterling sitzt im Vordergrund des Fotos auf einem schattigen Steinboden.
Image 5: Pho­to­graph of a but­ter­fly in which the back­ground is to be changed in the attempt.

The attempt to place the but­ter­fly on a horse by prompt was not suc­cess­ful. Instead of a second anim­al, the AI gen­er­ated a horse pas­ture and a stable in the image back­ground. The recol­or­a­tion of the but­ter­fly worked well — even though it was not asked for in the prompt.

Screenshot von vier Generierungen mit jeweils einem Schmetterling vor verschiedenen Hintergründen.
Image 6: Res­ults of the attempt to put the but­ter­fly on a horse.

The "Infin­ite Image" func­tion, in which images are sup­ple­men­ted by tex­tu­al descrip­tions of indi­vidu­al objects, often deliv­ers con­vin­cing res­ults. Although there are still minor logic errors such as false shad­ows and partly sur­real com­pos­i­tions, most of the res­ults are almost cred­ible, espe­cially in com­par­is­on to the gen­er­ated videos.

Eine Fotografie von zwei Gebäuden mit einer kleinen Treppe und Fenstern, bei der durch das Tool Blumentöpfe, ein Mops mit Ball, ein Fahrrad und eine Regenbogenflagge ergänzt wurden.
Image 7: Tübingen's Bursagasse, at least in the core of the image. Because some objects were added from all sides by text input.

What was part of the ori­gin­al pic­ture can still be determ­ined quite well here, because the added objects show incon­sist­en­cies. For more real­ist­ic addi­tions, for example of the dog, post-pro­cessing of light, shad­ow, col­or and high­lights would be help­ful, but these are not pos­sible with the tool at the moment. Instead, sev­er­al images can be out­put per prompt, from which the best image can be selected.

Due to the mul­ti­tude of dif­fer­ent func­tions, the tool offers the most diverse pos­sib­il­it­ies of use in the pro­duc­tion of audi­ovisu­al con­tent, whereby the cre­ation of arti­fi­cial pho­to­graphs and videos in par­tic­u­lar is in the fore­ground. The image manip­u­la­tion func­tions make it pos­sible to illus­trate ideas quickly, as authen­t­ic res­ults are gen­er­ated through text or image input. Espe­cially in the com­mu­nic­a­tion of sci­entif­ic con­tent, this opens up new pos­sib­il­it­ies for visu­al­iz­ing con­tent. For example, cli­mate research­ers could use the tool to draw atten­tion to the con­sequences of cli­mate change by gen­er­at­ing photo-real­ist­ic images that already show the eco­lo­gic­al effects of a rise in tem­per­at­ure. In par­tic­u­lar, social media con­tri­bu­tions or present­a­tions can be sup­ple­men­ted with appeal­ing visualizations.


Even if some res­ults are still expand­able, the tool offers a quick and easy way to visu­al­ize design ideas in a new way. It can shorten pro­duc­tion pro­cesses and allows edit­ing and image manip­u­la­tion without the need for pri­or tech­nic­al know­ledge. Of par­tic­u­lar interest for use in sci­ence com­mu­nic­a­tion could be the visu­al­iz­a­tion of abstract or future scen­ari­os, such as the impact of cli­mate change on our eco­sys­tem. How­ever, not all pro­jects suc­ceed right away, which is why work­ing with Gen‑2 by Run­way Research requires a lot of tri­al and error, and espe­cially per­son­al­iz­ing and adapt­ing the res­ults requires some patience and cre­ativ­ity in choos­ing the right prompts. The simple oper­a­tion invites exactly that and the gen­er­ated res­ults are fun, even if a good res­ult is not guar­an­teed with every gimmick.