Nicholas Ray Quiz

Over on twitter dot com @mimbale asked me about what my summed frames process would look like in the case of director Nicholas Ray. But first, a quiz: can you match up (any of) the frames below to the list of films?

MontageSm

 

title year process cinematographer
Bigger than Life 1956 Technicolor Joseph MacDonald
Bitter Victory 1957 Black & white Michel Kelber
Flying Leathernecks 1951 Technicolor William E. Snyder
Hot Blood 1956 Technicolor Ray June
Johnny Guitar 1954 Trucolor Harry Stradling
King of Kings 1961 Technicolor Manuel Berenguer, Milton Krasner, Franz Planer
On Dangerous Ground 1951 Black & white George E. Diskant
Party Girl 1958 Metrocolor Robert Bronner
Rebel Without a Cause 1955 Warnercolor Ernest Haller
Run for Cover 1955 Technicolor Daniel Fapp
The Savage Innocents 1960 Technicolor Peter Hennessy, Aldo Tonti
The True Story of Jesse James 1957 Technicolor Joseph MacDonald

Once you’ve taken a guess, hop over to filmvis.tumblr.com. I’ll post the correct answers tomorrow morning.

Posted in Typecast | Leave a comment

We’ve Got Company!

Over on Twitter, @chopperfireball asks:

My thought: perfect question for a digital humanities approach to cinema and media studies.

I’ve been playing with the data dump of subtitles from OpenSubtitles.org, and in particular the English corpus files from here: a 6.5GB folder of subfolders of subfolders of zipped xml files. So, why not search the OpenSubtitles corpus to see what matches we get for “We’ve got company,” and then crosscheck with IMDb to find the earliest?

I decided to just search “got company,” on the assumption that “We’ve” might limit the results too much: it would exclude “I’ve got company,” “you’ve got company,” “you have got company,” “Weve got company,” and other subtitle font encoding weirdness that a contraction might introduce.

My first problem was that the subtitles folder has 119 subfolders (for years), each with more subfolders and files within. So I needed to search recursively. The second problem is that I need to search these files without unzipping each one first. I used this command in Terminal to do so:

find . -iname ‘*.gz’ -exec zgrep “got company” {} +

Here are what the results looked like:

./0/26801/3470214_1of1.xml.gz:We got company.
./0/26801/3491073_1of1.xml.gz:- We got company.
./0/26801/3573503_1of1.xml.gz:We got company.
./0/26801/3599826_1of1.xml.gz:- We got company!
./0/26801/3631978_1of1.xml.gz:Guys, we’ve got company.
./0/26801/3920449_1of1.xml.gz:Wake up, you’ve got company.
./0/26801/3951555_1of1.xml.gz:We got company.
./0/26801/3951555_1of1.xml.gz:We got company.
./1901/11382/128016_1of1.xml.gz:You’ve got company, Trip.
./1909/10696/118940_1of12.xml.gz:- You got company?
./1911/63709/4026450_1of1.xml.gz:Santiago has got company
./1912/11913/216415_1of1.xml.gz:So, you’ve got company.
./1913/37828/3308112_1of1.xml.gz:We’ve got company.
./1929/64395/4061542_1of1.xml.gz:we got company.
./1930/27386/3248842_1of1.xml.gz:Oh, you got company.
./1930/27386/3440567_1of1.xml.gz:Oh, you got company.
./1933/1234/2102_2of2.xml.gz:I’ve got company right now.
./1933/1234/3122453_1of1.xml.gz: I’ve got company right now.
./1933/1234/3133430_1of3.xml.gz:I’ve got company right now.
./1933/1234/3133430_3of3.xml.gz:I’ve got company right now.
./1933/1234/3137076_2of2.xml.gz:I’ve got company right now.
./1933/1234/3211259_2of2.xml.gz:I’ve got company right now.
./1937/47447/3512534_1of1.xml.gz:but we got company.
./1937/7104/240514_1of1.xml.gz: I’ve got company.
./1937/7104/3094233_1of1.xml.gz: I’ve got company.
./1937/7104/3223919_1of1.xml.gz: I’ve got company.
./1937/7104/93813_1of1.xml.gz: I’ve got company.
./1938/1000/3148641_1of1.xml.gz:I see we’ve got company.
./1938/1000/3344629_1of2.xml.gz:I see we’ve got company.
./1938/1000/3506669_1of1.xml.gz:I see we’ve got company.
./1938/1000/68331_1of1.xml.gz:I see we’ve got company.
./1938/38647/3478797_1of1.xml.gz:But you got company.
./1938/38647/3478802_1of1.xml.gz:But you got company.
./1938/7760/3087673_1of1.xml.gz:We’ve got company tonight.
./1938/7760/3213587_1of1.xml.gz:We’ve got company tonight.
./1938/7760/3213588_1of1.xml.gz:We’ve got company tonight.
./1938/7760/3361406_1of1.xml.gz:We’ve got company tonight.
./1938/7760/3822692_1of1.xml.gz:We’ve got company tonight.
./1938/7760/79317_1of1.xml.gz:We’ve got company tonight.
./1938/7760/97751_1of1.xml.gz:We’ve got company tonight.
./1940/21299/3090625_1of1.xml.gz:- Looks like we got company, Shorty.
./1940/21299/3090625_1of1.xml.gz:- You got company.
./1940/21299/3092212_1of3.xml.gz:- Looks like we got company, Shorty.
./1940/21299/3092212_1of3.xml.gz:- You got company.
./1940/21299/3092212_3of3.xml.gz:- Looks like we got company, Shorty.
./1940/21299/3092212_3of3.xml.gz:- You got company.
./1940/24924/3121380_2of3.xml.gz:We got company.
./1940/24924/3121380_3of3.xml.gz:We got company.
./1941/35861/3283752_1of1.xml.gz:Oh, he’s got company.
./1941/53816/3608420_1of1.xml.gz: He got company
./1941/53816/3608422_1of1.xml.gz: He got company
./1941/9150/101449_1of1.xml.gz:We’ve got company.
./1941/9150/158082_2of2.xml.gz:We’ve got company.
./1941/9150/94986_2of2.xml.gz:We’ve got company.
./1942/1005/1738_1of1.xml.gz:Looks like we’ve got company.
./1942/1005/3592893_1of1.xml.gz:we’ve got company.
./1942/1005/97106_1of1.xml.gz:Looks like we’ve got company.
./1942/1005/97121_2of2.xml.gz:Looks like we’ve got company.
./1942/40517/3359340_1of1.xml.gz: We got company.
./1942/40517/3368050_1of1.xml.gz: We got company.
./1942/40517/3428769_1of1.xml.gz: We got company.
./1942/40517/3437623_1of1.xml.gz: We got company.
./1944/2230/132133_1of1.xml.gz:You got company?
./1944/2230/28273_1of1.xml.gz:You got company?
./1944/2230/3081278_1of1.xml.gz:You got company?
./1944/2230/3284537_1of1.xml.gz:You got company?
./1944/2230/3305330_1of1.xml.gz:You got company?
./1944/2230/3671542_1of1.xml.gz:You got company?
./1944/2230/48650_1of1.xml.gz:You got company?
./1944/2230/48746_1of1.xml.gz:You got company?
./1944/2230/87194_1of2.xml.gz:You got company?
./1944/2230/87194_2of2.xml.gz:You got company?
./1944/789/3119319_1of1.xml.gz:He’s got company!
./1945/29969/3166121_1of1.xml.gz:We’ve got company, take off cloth now
./1945/37749/3369358_1of1.xml.gz:We’ve got company.
./1946/27947/3163942_1of1.xml.gz:Look, we’ve got company.

There are 76 hits, just going up to 1946, although the repeated entries would be from multiple releases of the same film. Had I an actual background in computer science, I would like to try doing something such as calling on IMDb’s API (or better, OMDB’s API) to parse these results, or better wrangle the XML to call back OpenSubtitles webpages, but my hope is that to answer the question, it’s easy enough to look at the earliest entry (since the first set of subfolders seem to be organized chronologically) to determine in what sense “got company” was being used. But I need to read the OpenSubtitles documentation a little better to understand what the “0” folder is; turns out the first entry in our list is from Terminator 2: Judgment Day (James Cameron, 1991), so I’m guessing the 0 folder is for films they do not have organized in folders yet.

But the next result is from 1901?! Amazingly early result. Let’s see if the sense matches the action trope, or if this is just a butler introducing a guest or something:

Captain, there’s a small craft closing on the shuttlepod.
It’s a patrol ship.
You’ve got company, Trip.
I see them.
Bearing 1-8-4, mark 2-7.

Um, 1901? Wait a minute, why are there even subtitle results for 1901? Turns out this is another mis-filed result; the dialogue is from “The Breach,” a 2003 episode of Star Trek: Enterprise.

Sacrificing thoroughness’s for time’s sake, I’ll skip ahead a bit:

1913’s result is from the 2008 television show In Plain Sight, s1e4, but 1929 is promising: Applause (Rouben Mamoulian, 1929), a grim-sounding film I haven’t yet seen about “a burlesque star [who] seeks to keep her convent-raised daughter away from her low-down life and abusive lover/stage manager.”

Come on in, boys and girls, we got company.
Gee, Kitty, the baby looks great.
Oh, isn’t it cute?
It looks just like Kitty.

You can watch the scene here:

This is certainly not an action film, and “we got company” does not seem to have the same euphemistic or ironic overtones as in later uses, but the menace and threat of just this scene is apparent. Interestingly, the word “company” appears five times in the subtitles for this film, and the last instance is in fact a parsing of the meaning of the word “company” (44 minutes in):

Just looking for somebody to talk to.
Sailors don’t generally have much trouble finding company.
Oh, I don’t want the kind of company you mean.
Listen, all sailors ain’t a bunch of bums.
But you got no idea how tough it is to be all alone and lonesome in a city like this.
Maybe I have.

If you’re interested in further investigation, 1937’s “but we got company” is from the Bing Crosby vehicle Waikiki Wedding (Frank Tuttle, 1937). The next one that looks most promising is 1940 “Looks like we got company, Shorty,” which is from Boom Town (Jack Conway, 1940). If, unlike me, you have a copy at home, fastforward to 10:19-ish and let me know the sense of this exchange:

Well, I didn’t…
Looks like we got company, Shorty.
You got company.
That’s what I thought.
I beat you by four feet, Shorty.
Wait a minute.
I got another chance.

Hopefully by now you’ve realized the major problem with this approach: that the value of this answer is limited by the faith we have in OpenSubtitles’ completeness: both the range of films they have and the thoroughness of the subtitling. But, like any good digital humanities project, this initial result can help confirm or deny our first guesses (no, it’s not Star Wars), and put us on the right track to ask, and answer, even better questions.

Last, for even more of the firehose: see this link.

Posted in Typecast | 2 Comments

Western Roundup

In September, I watched 50 Westerns. Here’s one of them:

The Outlaw Josey Wales, summed frames

I’ve been trying to keep my viewing habits more focused this year, picking a director or an actress to watch for a month. No problem with someone like Andy Sidaris (a dozen films) or Alex Cox (thirteen), but I really stepped into it with “Western Month.” In truth I had just wanted to rewatch Once Upon a Time in the West (Sergio Leone, 1968), but I found myself adding and adding to the list until, well, I watched more than I had intended to but not enough to satisfy me.

I was already on to October (gialli!) when @wormatwork asked me if I was going to do a  “roundup.” And so this is what I did:

ImageJ=1.48d<br /><br /><br /><br /> unit=pixels

This is a montage representing 48 of the westerns I watched (click for a larger view). Each image in the montage is a sum image of every 10th second of each film (that is, one frame from every 10 seconds was extracted and summed with the others to create a real image; math, basically). At the top of this post is one of those summed images, showing one way of seeing the overall “look” of The Outlaw Josey Wales (Clint Eastwood, 1976).

Only occasionally might one make out actual features of these films–perhaps endtitles or credits or changes in aspect ratio, but certainly not characters or objects or much in the way of sets. Rather, these summed images show us patterns and colors: blobby shapes occupying center frame, a shadowy vignette ringing the corners, or mottled concentrations of saturation bleeding into one another. These shapes and colors are evocative in a way that tea leaves and tarot are: they don’t actually tell you all that much about what you’re looking at, but instead allow you an emotional response confirmed or denied once you come to discover what the image “really” is. Consider these three:

The Godfather, Part I, summed frames

The Matrix, The Matrix

The Wizard of Oz, summed frames

Actually, I tricked you there–those aren’t Westerns, but are sum images respectively of The Godfather, The Matrix, and The Wizard of Oz. In hindsight, The Matrix’s green code color palette and fixed-width font scroll is recognizable, but The Godfather isn’t nearly as dark as we might expect and The Wizard of Oz not as bright.

So, alone, these summed images can only tell us much. But what if we compare a large group of them? Do Westerns “sum” more or less the same? Is there a constitutive color palette? Is there a noticeable shift in their “average look” over time?¹

To begin a comparative analysis of the set of filmic, washed-out color fields I had generated, I used the program ImageJ,² which I’ve written about before, and in particular a plugin that helps visualize the relationship between the summed images by plotting thumbnails of those images along two axes. I worked with some simple interrelated features of images that are easily calculated (math again!) by computer: brightness, hue, and saturation (the last two not as useful for black and white films). Here is what it looks like to plot median hue by median saturation (in other words, from left to right the hue changes, where up to down we move from less saturated to more saturated). Click for a closer look:

Westerns hue_median vs saturation_median

In the bottom left, eleven of the black-and-white films are stacked on top of each other, occupying one space (not a whole lot of hue or saturation in black and white), and then above them we see a vertical line of films with similar hues but varying saturations. Thus we can easily see that by this measure Seven Men from Now (Budd Boetticher, 1956) is the most saturated film I watched. Outside of that tall left-most column, there are a few lower-saturation blue-hued films that stand out at the bottom middle; Decision at Sundown (Budd Boetticher, 1957) is the one from that group furthest to the right, just above and to the right of Johnny Guitar (Nicholas Ray, 1954). Before you look for yourself: yes, the two Boetticher films had different cinematographers.

Then, most obvious, is that horizontal line of four films in the middle. Here are those four up close, from left to right:

The Naked Spur
The Naked Spur
, Anthony Mann, 1953

Once Upon a Time in the West
Once Upon a Time in the West
, Sergio Leone, 1968

The Missouri Breaks, summed frames
The Missouri Breaks, Arthur Penn, 1976

Ride the High Country
Ride the High Country, Sam Peckinpah, 1962

Of these four, The Naked Spur is the most interesting to me; first, it doesn’t show nearly the same amount of lighter blue sky that the others do, and second, it very clearly shows the film’s pattern of framing large red-hued shapes in the center with blue on the right and blue-green on the left. The Naked Spur was filmed in Technicolor; it’s not the only I watched in that process, but it’s the one whose summed frame most obviously shows off the deep saturation that Technicolor is known for. (Off the top of my head, others I saw in Technicolor: The Wild Bunch [Sam Peckinpah, 1969], The Professionals [Richard Brooks, 1966], and Silver Lode [Allan Dwan, 1954]. A future research project for me is to use IMDb’s API to quickly sort for details like color process.) Another consideration is that The Naked Spur was one of the few I watched in the Academy ratio, and so we might expect the film’s “average look” to be concentrated in one central place.

Ride the High Country has the most unique hue of the group, somewhere in the magenta (300°) range. This is surprising to me, as the film doesn’t seem particularly garish, and has a number of nighttime camp scenes (although perhaps those campfires are showing up as magenta-hued?). Ride the High Country was filmed in Metrocolor (MGM’s trade name; in reality, Kodak Eastmancolor). Here are the frames from Ride the High Country used to generate the summed image:

Montage Ride the High Country

What is visible in Ride the High Country that is even more visible in The Missouri Breaks is the obvious blue sky at top of the frame. I’ll note the best example of this below, but in these two films it is clear that a number of scenes were filmed outdoors, with a horizontal or up-looking camera framing some part of the sky (invariably with some gorgeous cloud patterns in the distance). The Missouri Breaks also demonstrates another feature visible in a number of these films: a darker magenta-hued shape center frame, with a yellow-red-hued falloff to the right and left. While it’s difficult to make out characteristics, it is apparent that this shape is a result of the film’s repeated use of shots that only show one character. Whether closeups or medium shots, we’re more often looking at one person than a group. The Missouri Breaks is about a gang of thieves, but its strength comes from its emphasis on a battle of individual wills: Jack Nicholson’s rustler, Marlon Brando’s regulator, John McLiam’s land baron, and his love-interest daughter Kathleen Lloyd. (For comparison, look again at the image of The Godfather above and see if you can’t make out that film’s preferred two-shot.)

With that idea in mind, look again at Once Upon a Time in the West‘s widescreen (2.35:1) frame, which rather than show one darker shape in the center instead has a few dark shapes: one darker bluish one taking up the left third of the frame, then two smaller ovals on the right half. One explanation could be a preference for framing closeups on the left, mixed with repeated long shots of evenly spaced groups of characters across the frame. Such as these two shots, from the opening scene:

xOnce.Upon.a.Time.in.the.West 031

xOnce.Upon.a.Time.in.the.West 073

Another pattern shows up when we plot median hue vs. median brightness (in other words, from left to right the hue changes as it did in the first plot, where up to down we move from darker to brighter).

Westerns - median hue vs median brightness

Looking at this visualization, a few different films stand out. First, I can see (as with the first plot) that the majority of the films I saw are concentrated around one particular median hue value on the left, while this grouping of films is still differentiated by brightness and saturation. I would wager that a sampling of most color films of any genre would result in a similar spread: the soft, warm hue seems to me to be the color of light, mostly reflected off of human skin.

What is interesting, though, are the outliers: far on the top right is the magenta Ride the High Country, which we’ve already seen is unique in its hue. In the middle on the right is The Missouri Breaks. The brightest film is Johnny Guitar (in the center top, only a touch higher on the y-axis than Two Rode Together), and locked solidly in the bottom left corner is The Gunfighter. Here are those last two up close:

Johnny Guitar, summed frames
Johnny Guitar, Nicholas Ray, 1954

The Gunfighter, summed framesThe Gunfighter, Henry King, 1950

In The Gunfighter, Gregory Peck’s character spends much of the film somberly waiting in a saloon for a chance to leave town, and the dark tone and vignetted corners reflect the oppressive interior locations. On the other hand, Johnny Guitar is bright and evenly lit, particularly in its interiors. Here’s a typical shot; while we can see some occasional dark areas, a glance at the actors’ shadows on the floor shows us the numerous light sources on set:

xJohnny.Guitar.1954.BluRay.720p 097

Another interesting pattern from the above visualization is the cluster of films that occupies the upper left corner. They include Two Rode Together, Comanche Station, Unforgiven, Young Guns, High Plains Drifter, and The Quick and the Dead among others. These seem to be the most “Western” in look, and out of all 50, one from this group–The Searchers–most clearly demonstrates to me the genre’s basic look: a wide desert and a quarter of blue sky.

The Searchers, summed frames
The Searchers, John Ford, 1956

The sharp contrast between the two colors, limned by the faintest jagged suggestion of mountains, is compelling. Below are the sampled frames used to generate this image. We can see that even in the nighttime scenes there are either warm, fire-lit tones in the bottom two-thirds or blue silhouetting skies in the upper third.

Montage of The Searchers

Another mode of looking at The Searchers further demonstrates how (and where) this foreground-sky demarcation works. Here are a few “barcode” views of the film. In each case, a number of frames are extracted from the film, and then compressed horizontally and presented left to right. Here is a sample of 1200 frames (at 24 frames per second, The Searchers works out to about 171,000 frames):

Searchers barcode 1200

And here (respectively) are 400, 200, 100, 50, 25, and 10 frames (so you can see how this works):Searchers barcode 400

Searchers barcode 200

Searchers barcode 100

Searchers barcode 50

Searchers barcode 25

Searchers barcode 10

Last, I want to touch on Heaven’s Gate (Michael Cimino, 1980). Gauzy, misty, soft, indistinct: it was the film that made me first want to explore this technique for trying to capture a film’s overall “look.” And, I expected it to look like exactly like it does.

Heavens Gate, summed frames
Heaven’s Gate
, Michael Cimino, 1980

This image perfectly matches my visual conception of the film–almost as if a scrim just like this image had been laid over the lens during filming. If I were dreaming the film, here is what it would look like to me. If I were making a photography app, here would be a set of readymade photo filters. If I were a fan, here would be a nice poster or computer wallpaper. As a scholar, though, what use are these average looks, which strip out all (or almost all) narrative, characterization, plot, sound, dialogue, action? I don’t yet have a cogent answer to that question, but I do have a strong suspicion that film studies will benefit from new modes of visualization such as this one, which represent film texts from an otherwise impossible perspective–in this case, along the z-axis that compresses the film’s time into a single frame of form and light.

—————————

tl;dr: watch more Budd Boetticher.

—————————

caveat I don’t know where else to put: I did not attempt to correct for discrepancies in the quality of my digital source material, which will have an unavoidable affect on comparisons between the films.

—————————

To see all the summed images from the Westerns I watched, visit my new film visualization tumblr: filmvis.tumblr.com. I’ll be posting new Westerns over the next month (and probably gialli after that, among other things).

—————————

¹ (nope, no, and not really, as it turns out)

² I also used ImageJ to generate the summed film frames, using the Z-Projection feature.

Posted in Typecast | 2 Comments

Some Typewriters I’ve Watched Recently

as always, click on the images for a larger view


Death and the Compass, Alex Cox, 1992


Edge City, Alex Cox, 1980


El Patrullero, Alex Cox, 1991


Elephant , Alan Clarke, 1989


Fear City, Abel Ferrara, 1984


Mondo Cane, Paolo Cavara, Gualtiero Jacopetti, Franco Prosperi, 1962


Repo Man, Alex Cox, 1984


Sid and Nancy, Alex Cox, 1986


The Bullet Vanishes, Law Chi-leung, 2012


The Paperboy, Lee Daniels, 2012


The Wonderful and Frightening World of Mark E Smith, Dione Newton, 2005


To Live and Die in L.A., William Friedkin, 1985


Twixt, Francis Ford Coppola, 2011


Zeder, Pupi Avati, 1983

Posted in Typecast | 5 Comments

Using ImageJ to Re-Visualize The Iron Giant

[apologies to the typosphere for forsaking the Selectric for some o’ this digital ink. I’ll soon make it up to you with typewriter-movie-eye-candy.] 

Intro

This post describes my efforts to use software that analyzes visual images in order to “re-visualize” Brad Bird’s 1999 animated film The Iron Giant, which was adapted from Ted Hughes 1968 children’s novella. By “re-visualize” I have in mind two things. First, it does not make quite as much sense to talk of “visualizing” a film text as it does to talk about other visualizations such as visualizations of census data, or of place names mentioned in novels, or of word frequencies in political speeches. Since a film (along with, most often, its audio track) operates primarily by visual means, we should recognize the film itself as already a “visualization.” Just as with data visualization, which demonstrates according to Manuel Lima a

broad palette of visual elements and variations that consider color, text, imagery, size, shape, contrast, transparency, position, orientation, layout, and configuration [and yet still] follow noticeable trends and common principles . . . in a type of emergent taxonomy,

film from its earliest moments has been consistently spoken of as having a language and grammar that serves to isolate and emphasize details in order to make sense of a complex world; indeed, almost all of the words Lima uses to describe principles of data visualization are the same ones used in introductory film texts to describe mise-en-scene (barring, perhaps, “transparency,” although that wouldn’t be out of place in discussing montage).

Second, by “re-visualization” I have in mind critic Victor Shklovsky’s discussion of ostranenie, which grounds his exhortation that:

The purpose of art is to impart the sensation of things as they are perceived and not as they are known. The technique of art is to make objects ‘unfamiliar,’ to make forms difficult, to increase the difficulty and length of perception because the process of perception is an aesthetic end in itself and must be prolonged.

In this, I am endorsing a claim that interpretation is art: that (re-)visualizing critics are, like the texts they study, involved in an effort to “impart the sensation of things as they are perceived” by taking the text and making it unfamiliar (just as for Shklovsky the text should take the too-familiar world and render it unfamiliar through artistic presentation). I am coming to realize that the aesthetic dimension of contemporary data visualizations comes precisely from the fact that, rather than return a defamiliarizing art back to reality (that is, to return our perceived “sensation of things” back to an intellectualized “how things are known”), visualization interprets art into a third direction (or “third meaning,” shades of Barthes), where the real world, having been presented in defamiliarized terms through art, is once again defamiliarized through a method of art/criticism which at once extends originally from the work of art but at the same time spins back out from the real world with which it is connected.

OK, how about a gif?

gif-schema-des-relations

Mathilde Lesueur, animation. “Evolutionary schema showing relationships between characters in Eric Rohmer’s Pauline à la plage.” <http://mathildelesueur.com/spip.php?article20>.

Back to The Iron Giant. I became interested in this film and presented a paper on it at MLA 2009. At the time, I was using a psychoanalytic model to account for film adaptations of children’s literature. Children’s stories are often filled with “living dolls” and other inanimate objects that come to life, move, or talk (e.g., Pinocchio, the Tin Woodman, Toy Story, or WALL-E). While a welcome feature of childish narratives, such uncanny figures are horrific in adult literature and film (e.g., Child’s Play, the Puppet Master series, and most sci fi involving robots). In adapting children’s stories to cinema, filmmakers actualize children’s fantasies of “living dolls,” and I wondered in general how this process worked. Freud puts the puzzle like this: “Children have no fear of their dolls coming to life, they may even desire it.” I was also specifically curious about one particular narrative change (of many) that The Iron Giant made to the source text: in both, the Iron Giant’s body is fragmented and he must put himself back together, however while the novella begins with a fragmented Giant, the film runs this process in reverse, beginning with an intact Iron Giant and ending with the robot in pieces. This theme of the fragmented body and the image of body parts in autonomous motion are at the heart of cinematic practice, which also constructs the illusion of coherent space and time by putting together discrete fragments, and in order to escape an uncomfortable, uncanny meaning, it seems that the film must avoid a discussion of the machine being’s fragmentary origins and begin instead from an illusion of coherence.

That was as far as I got trying to turn my conference paper into a publishable article (new job, new house, new baby, etc.). Now, I’m hoping that by re-visualizing the film, I can take better account of the fragmentation of the Giant and of the space of the film (viz., the various locations where actions occur). Hughes subtitled his novella “A Children’s Story in Five Nights,” and I likewise wanted to see if I could take the theme of temporal fragmentation as a way to account for the process of adaptation. I have already seen the film one way (a number of times): chronologically frame-by-frame. Now I want to see it again in new ways (in literally new ways, not just new perceptions via the old way). Imagining the film arranged not as a succession of frames on a strip, but stacked up in a cube,[1] I want to watch the film sideways, and see what comes of it.

Method

Here is a step-by-step account of what I did to prepare the film for analysis. It is a combination of following instructions and being dunderheaded, with one very useful insight born of trial and error.

1) Working from a digital copy of the film, I used Quicktime’s “Export” feature to convert the movie into still images. The digital copy had a frame rate of 29.97 frames per second, so I chose this same ratio when exporting, resulting in 139,517 png image files. This was undoubtedly overkill. I could have chosen less (such as one frame per each second) or more (such as 100 frames per second), but the latter would have simply resulted in redundant frames and the former might have missed shot changes that occurred in less than one second. I should note that while this method approximates the conventional technique of scanning individual frames from a long strip of film, in reality my source material was a compressed digital intermediary that works by comparing reference frames to changes between frames, rather than what happens with film-based materials, where each new frame is an entirely, well, new frame. This poses a subtle philosophical problem for scholars working on visual analyses of films, since issues of compression and interpolation, although perhaps not noticeable or statistically significant, are nonetheless present. Having said all that, I have to turn right back around and note that The Iron Giant is an animated film and thus was itself never “real” (in the limited sense of the word)–the film was animated using a combination of 2D and 3D animation. [I should also note, in case you could not tell, that I lack a fundamental understanding of how Quicktime’s “Export” feature works and what it actually does when I ask it to generate “29.97 frames per second.” No doubt I should have just picked “30 frames per second,” which was the number I later used when calculating shot lengths.]

Folder of Images

2) After removing the end credits and initial frames announcing production and distribution companies, I manually went through the files to identify shot changes. Stepping through 130,000-plus files was as time-consuming as it sounds,[2] but in holding the “down arrow” as I browsed through the image folder I essentially watched the film in slow motion. When I noticed a shot change, I marked the first and last frame of each shot by appending an “a” or “b” respectively to the file name.

3) While I was scrolling through the film, I also noted some basic information about each shot in a separate spreadsheet: whether it was XCU, CU, MS, LS, or XLS (or other); whether it was Still, Track, Pan, Zoom (or other), and what the Transition out of the shot was (cut, dissolve). Last, I kept track of two other parameters that I hoped to use to redirect the film’s chronological organization: the Location of the shot (i.e., “The Town,” “At Sea,” “Barn,” “The Woods,” etc), and which characters were in each shot. While it was simple to predetermine the choice of basic shot information, I needed to expand the categories for Location and Character as I went along.

4) A quick sort enabled me to create a new folder with just the first frame of each shot in the film (I will going to get to the last frames later, so they’re in another folder). I ended up with 1,174 shots, and because the files retained the sequential numbering from the original 130,000-plus images, a quick Excel formula gave me a list of all the shot lengths (and the average shot length: 4.04 seconds).

Thus far I have actually followed a very traditional method for “marking up” a film. Other than working with a digital text, I have done very little that could not have been accomplished before the advent of computers. My next step, however, is novel and takes advantage of new means of computer-aided data visualization.

ImageJ=1.47h

5) To analyze and display the sequence of film shots, I used the image processing program ImageJ and in particular custom macros developed by Lev Manovich and Software Studios along with QTIP. All of this software is freely distributed online, and links to download and other help documentation is here: http://lab.softwarestudies.com/p/software-for-digital-humanities.html. The central feature for my purposes was the ImagePlot macro, which creates a sort of visual bivariate spreadsheet of data–a scatterplot, say, but with actual images instead of just circles for data points.

ImageJ works by referring to a folder of images (whether film stills, paintings, or manga) and to a database stored as a CSV or excel file. Accompanying the ImageJ package linked to above, there are additional pieces of software that help create this database, so that users can automatically generate information about a batch of images, such as their hue, saturation, and brightness values; their contrast and intensity/brightness; and even more exotic measures like “number of shapes” and “entropy.”

Importantly for my purposes, users can easily modify the resulting database in a spreadsheet program. So, after using the ImageJ program and QTIP to automatically analyze my folder of frames, I combined those results with the results I had previously come up with regarding Shot Parameters, Characters, and Locations. The result was 122,000+ cells of Excel data:

Excel Data

6) ImageJ works with numbers, not letters; it wasn’t going to know what to do with “CU” or “Diner” or “Annie.” Thus, I needed a system to code the variables I had myself identified into the database. I did this simply by assigning a whole number to each of the 13 Characters I identified and to each of the 17 Locations.

One additional limitation of ImageJ was that each cell must only have one number in it. Easy enough for “Locations,” since all shots by necessity take place in only one location (OK, there is actually one shot when the Giant shoots his weapon which begins in the town and ends at sea, and it is not difficult to imagine a film with many complex shots that begin and end in different locations, much less films that use split-screen effects à la lots of De Palma or Figgis’s Timecode). But I ran into a problem with Characters, since naturally many shots have more than one character, and thus I would need to have more than one number associated with the data row for that shot. Luckily, however, ImageJ has no problem passing through data that is repetitive. So, in the instances where there were, say, two characters in one shot that was analyzed in Row 100, I simply marked one character for Row 100, and then copied the contents of that row at the bottom of the spreadsheet and marked the second character there. Via cut-and-paste and paying careful attention, I created a sort of a “manual recursivity” (and a ridiculously complex database). Since ImageJ works its way down the database row by row, once it reaches the end of the original data set, it simply starts again on the second (or third or more) Characters/Locations/whatnot.[3]

Some Initial Results

Here are some of the resulting visualizations. (Note, the original files are TIFFs each well over 100MB, so I have reduced them to JPEGs for this blog. Still, by clicking on the image you should be able to explore each visualization and make out the shot.)

First are visualizations where the film’s chronology is more-or-less maintained; the frames are ordered in sequential order along the x-axis, and thus changes or patterns in the y-axis variable are easy to spot.

Changes in Median Hue:

imageID-vs-hue_median.jpg

Changes in Median Brightness:

imageID vs brightness_median

Changes in Shot Length:

imageID vs ShotLengthFrames

Changes in Entropy:

imageID vs entropy

These are all examples of a kind of time-series plot. In the Visual Display of Quantitative Information, Edward Tufte discusses the problem with time-series: “the simple passage of time is not a good explanatory variable: descriptive chronology is not causal explanation” (37). Narratologists might debate this point regarding the above visualizations, where a film’s narrative and visual unfolding is controlled in a way that is calculated to produce recognizable effects. This is most apparent in the second visualization (median brightness), where there is a noticeable periodic alternation between light and dark.

Next are visualizations where the x-axis variable is not chronological, but simply another data point. Tufte calls these “relational graphics,” “the greatest of all graphical designs” since they “encourag[e] and even implor[e] the viewer to assess the possible causal relationship between the plotted variables” (47). Unlike the time-series plots which present causal arguments, these next visualizations emphasize relationships between variables.

Median Brightness (x-axis) vs. Median Hue (y-axis):

brightness_median vs hue_median

Shot Length (x-axis) vs. Entropy (y-axis):

ShotLengthFrames vs entropy

Many of these relational visualizations are not all that surprising for anyone with a passing familiarity with Hollywood films–of course there will be limited clusters of similarly-hued shots since narrative films strive to create a series of visually coherent spaces, and hue (background color, set design, costuming, color palette) is an expression of that.

Last are the visualizations I am most interested in further exploring–those that plot character or location on one axis in an effort to identify patterns or outliers along the other. I would expect there to be a noticeable difference in, say, “hue” when looking at shots taking place in The Woods (green) vs. The Barn (brown), but what about brightness, or shot length, or even a more general idea of how the film moves chronologically through various spaces?

Location (x-axis) vs. Entropy (y-axis):

Locations vs entropy

Location (x-axis) vs. Shot Length (y-axis):

Locations vs ShotLengthSeconds

Shot number, i.e. chronology (x-axis) vs. Location (y-axis):

imageID vs Locations

Looking at these graphs, particularly the last one (whose y-axis I forgot to label), I see the beginnings of a new kind of mapping of film space. While I am suspicious of the value of, say, Standard Deviation of Image Saturation in developing a critical argument about a particular film, the ability to use even an artificial-seeming method of sorting a huge number of images and identifying patterns or outliers is valuable. By hand-identifying the parameters I am interested in–locations in this case–these kinds of visualizations make it easier to see how a film’s use of a locations is or is not internally consistent (well, maybe).

A Few Future Questions

I need to figure out a better method for handling the co-presence of multiple qualities in a single shot (i.e., multiple entries in a single cell, like characters).

Are the shots “true”? 624 of the 1,174 shots are static, which means that just under half of the shots “ended up” someplace different than they began. By just analyzing the first frame of each shot, I am looking at a different picture than if I were to analyze the last frame (or for that matter to analyze every frame). Analyzing every frame might give a truer “picture” of the film in a conventional sense, but such a method would obscure a film that had a more ambulatory camera; comparing analyses of first and last shots might draw out differences in such films. I need to imagine the best way to do that. I also imagine that this kind of analysis might be just as useful (albeit in a different way) as the old-fashioned marking of camera movements with “dolly right,” “pan left,” or “static.” In other words, if the first and last shot differ significantly, camera movement can be inferred.[4]

One of my students, Matt Power, imagined using ImageJ to trace a line where the Giant and Hogarth intersect over time throughout the film. My first response was “impossible, since that’s three variables–time, character X, and character Y–and we can only work with two.” On the car ride home, though, I realized it would be possible to pass three variables through ImageJ. I need to look into that further.

I need to do some simple, analogous, processing of the text of Ted Hughes The Iron Giant. The next step of my research will involve identifying and pursuing an appropriately analogous method for analyzing the text so as to draw comparisons between the film and novella. The conceptual artist side of me wants to fire up my scanner and see what QTIP can do with scanned pages.

This would be infinitely more useful if I were able to compare visualizations of The Iron Giant with data sets from other films (sorted, perhaps, into categories like “animated films,” children’s movies, films from 1990s). I knew that before I started, so I’m not sure it’s a future question so much as a problem of scale.

Takeaway[5]

This will never be a truly digital project, and the thought of scaling up to an analysis of even a dozen films is daunting considering the amount of labor still required to identify film components such as character or location. However, it is my belief that a hybrid method of hand and digital film analysis is worth the effort. Personally, as a stickler for details, I still do not have a good enough grasp on the principle of “good enough,” or how much human effort is required in order to achieve meaningful results. On the one hand, there is an aesthetic dimension to knowing I have reverse-engineered the film’s editing pattern; on the other, it took me quite a few weekends to finish stepping through every film frame.


[1] I came to this idea considering digital solutions to achieving slit-scan techniques in video. This is the most radical technique for “re-visualizing” films, and had I more intellectual fortitude (and tenure?) I might pursue this concept instead. For examples, see “An Informal Catalogue of Slit-Scan Video Artworks and Research” and in particular “Video Cubism.”

[2] I cannot get it to work, and not sure I would trust it if I could, but here is a link to “Shotdetect . . . a free software (LGPL) which detects shots and scenes from a video.”

[3] Again, if you’re not able to tell yet: I actually have no real understanding of how ImageJ “works.”

[4] A million qualifications to this thought–the shot whose camera begins and ends in the same place but roves throughout the shot; the shot which begins one place and ends another, but settles for a time on a third or fourth place.

[5] Takeaway” is a corporate word my wife has infected our home with–she says they would always end meetings with a “takeaway,” an answer to the “what’s the point” question.

Posted in Typecast | 10 Comments