[HN Gopher] Pixar almost deleted Toy Story 2 (2012)
       ___________________________________________________________________
        
       Pixar almost deleted Toy Story 2 (2012)
        
       Author : feross
       Score  : 123 points
       Date   : 2022-10-26 14:47 UTC (8 hours ago)
        
 (HTM) web link (thenextweb.com)
 (TXT) w3m dump (thenextweb.com)
        
       | dingosity wrote:
       | This is a great example of why it's a good idea to always test
       | your backups.
        
         | pstuart wrote:
         | In the olden days when I was the admin for a small shop we
         | would immediately restore the backup to a second system to be
         | used as a playground. Win/win: verified restore and a place to
         | test/train with production data.
        
         | ltbarcly3 wrote:
         | I came to make this snide comment as well! Nobody tests their
         | backups though, I've only ever seen it done inadvertently when
         | the backups were used as part of the development process.
        
           | yakubin wrote:
           | BTW how do you test a personal workstation backup? I don't
           | have backup workstations to rotate between by backing one up,
           | restoring on the other, working and then cycling between them
           | in this manner. How do you test the restore procedure without
           | risking destruction of the original?
        
             | opencl wrote:
             | You can buy an extra storage device and swap them out
             | (either physically or swapping the boot order), as long as
             | your workstation doesn't have soldered in storage.
        
               | jen20 wrote:
               | Even if it does, you can usually boot from an external
               | volume anyway.
        
             | bbarnett wrote:
             | md5sum using a bash script and find, then compare on live.
        
             | l33tman wrote:
             | Most backup systems allow you to explore backups and
             | extract individual files. Testing a bunch of files for
             | restoring is at least better than completely ignoring it..
             | however it's difficult if it turns out part of your backup
             | is simply missing, maybe an erroneous exclusion entry or
             | some path miss (happened to me, I was without backup for a
             | home folder for a year before I noticed it, the /home/user/
             | was a symlink to another volume that mostly had tmp data
             | and so wasn't backed up). You might not even notice this
             | even if you restore the whole backup, because you can't
             | really be expected to go through everything every time
             | either..
        
         | vikingerik wrote:
         | You don't need backups. You need _restores_.
        
           | haswell wrote:
           | Generally speaking, I think this is what testing a backup
           | means.
        
       | [deleted]
        
       | geocrasher wrote:
       | My favorite part of this account isn't the "test your backups!"
       | lesson. It's the "Have an amazing team" lesson. In one weekend, a
       | team of actual humans sorted through _thirty thousand files_ by
       | hand.
       | 
       | Surround yourself with amazing people, and you will find amazing
       | solutions.
        
         | outside1234 wrote:
         | and then make sure those amazing people are testing their
         | backups. :)
        
       | brainwipe wrote:
       | It's 2022. Test your backups.
        
         | haswell wrote:
         | While yes, backups should be tested, this all happened over two
         | decades ago.
        
       | raldi wrote:
       | I'm reminded of the time I was on the InfoSec team at a company
       | you've all heard of where all software development was done on a
       | handful of Unix servers with a shared network file system.
       | 
       | I reached out to the Systems team to suggest they look into
       | mounting the home directories with the no-setuid option, and one
       | of them invited me over to pair at their desk. They explained
       | that there was a trick, where you could leave the existing mount
       | point as is, with its existing mount options, and then mount the
       | same device somewhere else with different options. So they got a
       | root shell, went into some scratch directory, mounted the
       | homedirs there, and we tested things out.
       | 
       | I forget how that went, or what happened next, but it seems that
       | we neglected to clean up after this experiment, and at some
       | point, another Systems engineer was looking around the scratch
       | directory and perhaps ran something like `du -ak` (forgetting to
       | specify the option that disables crossing onto new devices),
       | finding what appeared to be a gigantic copy of everyone's home
       | directories.
       | 
       | So they started `rm -rf`'ing it, and made a pretty huge dent
       | before the phones started ringing and they realized that what
       | they were erasing was not a copy on a local scratch space, but
       | rather the real thing over on the NFS drive.
       | 
       | Fortunately, this company _did_ have great backups.
        
         | LanceH wrote:
         | I was one ROLLBACK away from the Users table in production. Mid
         | 90's, we fixed things in production all the time, and I left
         | the where clause out. There were no soft deletes back then, we
         | just deleted the row, and that would trigger the deletion of
         | user resources elsewhere...
         | 
         | I'm sure we had backups, but it would have been bad and
         | resulted in a lot more fixes in production. I guess the upside
         | of the "seat of the pants" days is that there was poor
         | referential integrity, so it wouldn't have taken out anything
         | else like billing history.
         | 
         | edit: I never typed anything so carefully as I typed that
         | rollback. I almost want to say there was an autocommit when
         | closing the session, but my memory on that is fuzzy. MS SQL
         | Server, if anyone knows. From then on, when called on for
         | sketchy but necessary actions, I write the where clause first,
         | run it as a select, see the number of results, wrap it in a
         | transaction, etc...
        
           | tomcam wrote:
           | Good story although there were indeed soft deletes in the
           | 90s.
        
       | bluedino wrote:
       | >> Galyn mentioned she might have a copy at her house. So we went
       | home to get that machine, and you can watch the video for how
       | that went...
       | 
       | DLP software hadn't yet been invented
        
       | dang wrote:
       | Related:
       | 
       |  _A baby saved 'Toy Story 2' from near complete deletion (2021)_
       | - https://news.ycombinator.com/item?id=30330740 - Feb 2022 (121
       | comments)
       | 
       |  _Toy Story 2 files wiped by accident, restored thanks to work-
       | from-home employee_ -
       | https://news.ycombinator.com/item?id=23022717 - April 2020 (2
       | comments)
       | 
       |  _How Pixar's Toy Story 2 was deleted twice (2012)_ -
       | https://news.ycombinator.com/item?id=20271179 - June 2019 (2
       | comments)
       | 
       |  _Toy Story 2 Got Deleted Twice, Once on Accident, Again on
       | Purpose (2012)_ - https://news.ycombinator.com/item?id=20265328 -
       | June 2019 (4 comments)
       | 
       |  _How One Line of Text Nearly Killed Toy Story 2_ -
       | https://news.ycombinator.com/item?id=16621848 - March 2018 (1
       | comment)
       | 
       |  _How Pixar's Toy Story 2 was deleted twice [] (2012)_ -
       | https://news.ycombinator.com/item?id=15978145 - Dec 2017 (1
       | comment)
       | 
       |  _Did Pixar accidentally delete Toy Story 2 during production?
       | (2012)_ - https://news.ycombinator.com/item?id=13417037 - Jan
       | 2017 (253 comments)
       | 
       |  _How Toy Story 2 Got Deleted Twice, Once on Accident, Again on
       | Purpose_ - https://news.ycombinator.com/item?id=11500467 - April
       | 2016 (1 comment)
       | 
       |  _Rm -rf nearly killed Toy Story 2_ -
       | https://news.ycombinator.com/item?id=11326831 - March 2016 (1
       | comment)
       | 
       |  _How Toy Story 2 was nearly lost_ -
       | https://news.ycombinator.com/item?id=9739675 - June 2015 (1
       | comment)
       | 
       |  _Pixar's Toy Story 2 was deleted twice (2012)_ -
       | https://news.ycombinator.com/item?id=8786790 - Dec 2014 (2
       | comments)
       | 
       |  _How Pixar's Toy Story 2 was deleted twice once by tech & again
       | for its own good_ - https://news.ycombinator.com/item?id=4001714
       | - May 2012 (1 comment)
       | 
       |  _How Pixar Almost Lost Toy Story 2 To A Bad Backup_ -
       | https://news.ycombinator.com/item?id=3972798 - May 2012 (75
       | comments)
        
         | joezydeco wrote:
         | I used to flag duplicates from the new page, but realized it
         | goes nowhere. Maybe a different flag to downgrade dupes?
        
           | nathanvanfleet wrote:
           | I kind of wonder if deduping posts would be good. I find it
           | really annoying to see reposts and if definitely makes it
           | feel like there really isn't all that much content out there,
           | at least much content that has enough viral potential. I
           | suppose it's possible for a company to fingerprint content
           | easily enough to stop dupes. Maybe it would block the post
           | but allow a repost of the original content? But is that
           | really good for the company? Plenty of people haven't seen
           | that content anyway, so maybe it's working for those people,
           | and sometimes you do want to see the repost if it's been long
           | enough. I personally think it might just seize the entire
           | ecosystem if you prevented it because the volume of content
           | would go down, and the vitality of things would go down too.
           | Not to mention the people reposting are kind of keeping the
           | whole thing alive for their virtual points, and if they can't
           | do it at scale by posting the same video in 20 different
           | places anymore things might just stop. I feel like a lot of
           | social networks are relying on a an orbiting cloud of garbage
           | that just recirculates constantly. I'm surprised no one has
           | started uploading, upscaling and posting America's Funniest
           | Home Videos clips.
        
             | joezydeco wrote:
             | I don't think people are karma farming here. They're just
             | too new and/or don't recall when certain posts have been on
             | HN before.
             | 
             | Dang has a pretty good list of links that keep showing up
             | every 90-180 days on a continual basis (e.g. "The Story of
             | Mel")
        
             | vlunkr wrote:
             | > I feel like a lot of social networks are relying on a an
             | orbiting cloud of garbage that just recirculates constantly
             | 
             | This is definitely true of mainstream social media, but I
             | don't think it's that bad on HN. Based on the list of
             | duplicates, this has been posted 13 times in 12 years,
             | that's really not that bad. Things don't live on the front-
             | page that long, someone reasonably could have missed it
             | every time. If people are engaging with the post, what's
             | the problem?
        
           | ChrisArchitect wrote:
           | Yes please.
           | 
           | This is hardly news, even if it prompts discussion, most of
           | it is repeating the same discussion from all the previous
           | discussions
        
           | dang wrote:
           | Flagging from the new page doesn't go nowhere - it's handled
           | the same way as flagging from any other page.
        
             | joezydeco wrote:
             | Of course. But I'm guessing that dupes aren't flagged as
             | heavily as something illegal/inappropriate/ads/etc. So they
             | survive more often, correct?
        
         | onionisafruit wrote:
         | It looks like you're saying you've had this conversation
         | before.
        
           | dang wrote:
           | 'Related' lists are just for people who might be curious to
           | read past related threads.
           | 
           | (Arguably this one is dupey because the last major discussion
           | was already this year, but I decided to let it slide in this
           | case)
        
       | cnorthwood wrote:
       | Early in my career I was working for a well-known anti-virus
       | firm. They had a crash of their Perforce servers and it turns out
       | the backups were not good and the most recent good one was
       | several months old. They painstakingly restored it from the good
       | backup and forensic recovery of the drives, but there were
       | several files that were still missing to fully restore the
       | archive. They did know the hashes of the files, so what they did
       | was push out a bunch of custom virus signatures to the dogfooding
       | (internal testing) environment which flagged any files with those
       | hashes as viruses, and then asked anyone who had a hit to send
       | the file into central IT to get it resolved. It was fairly clever
       | I thought!
        
       | jfb wrote:
       | I remember learning that when Pixar did a pan-and-scan version of
       | one of their movies, they wouldn't just use a floating 4:3 frame
       | over the 16:9 original -- they'd _re-render the movie_ , moving
       | items around as necessary, to get a better framing. The
       | perfectionism was palpable when we'd meet with them.
        
         | acuozzo wrote:
         | Disney did this with "Lady and the Tramp". Cinemascope and an
         | academy ratio versions were animated individually in order to
         | make the best use of the frame shape.
        
         | bee_rider wrote:
         | This seems like a huge advantage with mostly-digital movies.
         | 
         | I wonder how easily a Marvel or Starwars movie could be re-
         | rendered like this. Technically there are actors of course, but
         | they are usually the main focus of the scene anyway...
        
           | hondo77 wrote:
           | > I wonder how easily a Marvel or Starwars movie could be re-
           | rendered like this.
           | 
           | Fortunately we don't have to worry about 4:3 anymore.
        
             | Keyframe wrote:
             | There's wider adopted 16:9 aka 1.78 aspect target and then
             | there's everything from 1.85 to 2.39 and ridiculousness
             | like 2.44. Most common three today being 1.78 (aka 16:9),
             | 1.85 and 2.39 (2.35 instead sometimes). 1.85 to 1.78 is
             | usually a fit frame type of crop and call it a day, but
             | 2.39 to 1.78 for home/TV release is same story all over
             | again - do you do the letterbox or are we gonna play re-
             | frame this shot game?
        
           | gabereiser wrote:
           | For green screen shots this would be fine. For VFX LED wall
           | shots, this wouldn't work. However, movies and shows today
           | are shot in a high resolution that you can crop/edit into
           | most formats.
           | 
           | Green screen shots lend itself to resizing better due to the
           | ability to fill in more of the scene while pulling the
           | subject matter away a bit to fit the format.
        
           | jonny_eh wrote:
           | VFX shots are not pure CGI, it's a whole lot of layers
           | composited on top of each other, think Photoshop but video.
           | So making tweaks to any given shot would not be as easy.
        
       | [deleted]
        
       | DefNotMe wrote:
       | Is kottke.org operated by Daniel Kottke?
        
         | jonhohle wrote:
         | Jason Kottke, unrelated - https://kottke.org/99/06/dan-kottke
        
       | helsinkiandrew wrote:
       | I think there's more background and detail in this version:
       | 
       | https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...
        
         | dang wrote:
         | We've changed the URL to that from
         | https://kottke.org/12/05/how-pixar-almost-deleted-toy-
         | story-.... Thanks!
        
       | naikrovek wrote:
       | this is insane to me because individual frames for a Pixar film
       | are around 1GB in size.
       | 
       | recall how large laptop hard drives were before Toy Story 2
       | (2002)...
       | 
       | I have an extremely small amount of experience writing shaders
       | for RenderMan (BMRT, actually), and those aren't large. animation
       | data typically isn't HUGE, but models sure as heck are, and a
       | frame you send to RenderMan is concrete (it used to be, maybe it
       | isn't anymore?) meaning that all geometry was present in the file
       | you gave to RenderMan, only the compiled shaders were referenced
       | from it, because they were procedural and could not be statically
       | included. so I guess the frames the renderer sees are sort of
       | like the binaries of what I would now view as a software
       | compilation process.
        
         | raldi wrote:
         | I think this story is about the models, not the rendered
         | frames.
        
           | naikrovek wrote:
           | well that's my point. I always thought that what was lost
           | were frame definition files, and not the models. surely every
           | animator had copies of the models; you can't animate without
           | them, and the movie was in production when the data was lost,
           | so they had those models, or certainly most of them.
           | 
           | I think what was lost was the animations, and not the models
           | or the frame/scene files.
        
       | sdevonoes wrote:
       | > In order for her to work on the movie while out, they had
       | plugged the machine up to the local network and copied the whole
       | file tree over.
       | 
       | Now image the same situation in 2022:
       | 
       | - there is no acceptable backup for the whole movie (or let's
       | say, app)
       | 
       | - only you know there is one acceptable copy... but it's on your
       | personal laptop. You did copy it back in the day because, well,
       | let's say you wanted to run the app in our local machine
       | 
       | There is no way you would tell your team/boss: "Hey guys, no
       | worries! I have a perfect copy of it!", because why on earth did
       | you copy work material into your personal laptop?! That's the
       | rule number 1 of every modern tech company: don't mix work stuff
       | with personal laptop (and the other way around: don't use work
       | laptop for personal stuff).
        
         | asciimov wrote:
         | - If you can trust an employee with a $25K ($50K today)
         | computer, trusting them with a complete source tree isn't a
         | stretch.
         | 
         | - I'd be more pissed off that somebody didn't have a complete
         | backup somewhere, even if it was a personal laptop they used
         | for work.
         | 
         | - Back then, most people didn't have home computers. Hell
         | nowadays a home pc isn't a guarantee, as most of my (non-
         | engineering) coworkers only have cellphones and ipads at home.
         | And while I agree don't mix work and play, most of these people
         | do.
        
           | serf wrote:
           | > Back then, most people didn't have home computers.
           | 
           | you had a much different perspective of 99 than I did.
           | 
           | That factoid might have been true _for-the-public-at-large_ ,
           | but certainly not for the offices of large CGI or dev firms.
        
             | hinkley wrote:
             | Animated movies take years to make.
             | 
             | > The story likely takes place in 1998, though Jacob admits
             | he's foggy on the exact date.
             | 
             | In 1998 we were living in Internet Time. The amount of
             | stuff that changed from '97 to '98 and from '98 to '99 made
             | people's heads spin. Google incorporated in September of
             | 1998. Napster had been out for 5 months when Toy Story 2
             | debuted in theaters, and Java 1.2 was just about to exit
             | from beta into general availability.
             | 
             | People were still introducing each other to WinAmp when
             | they started storyboarding this movie.
        
             | asciimov wrote:
             | > for-the-public-at-large
             | 
             | True. Most of the people I knew with computers back then
             | had them provided by their work, or bought them for
             | university.
             | 
             | But then I lived in a small city in Texas not near a major
             | metro. Most people there didn't buy a home pc until the
             | DotCom boom brought those cheap emachines.
        
               | Consultant32452 wrote:
               | I decided to look it up. In 2000, 51% of households had a
               | computer a 41% had internet.
               | https://www.infoplease.com/math-science/computers-
               | internet/u...
        
         | 2muchcoffeeman wrote:
         | Susman was the Supervising Technical Director and the machine
         | was a Silicon Graphics workstation. Certainly sounds like she
         | was given a work machine to do her job.
        
       | jefftk wrote:
       | More details on Quora: https://www.quora.com/Pixar-company/Did-
       | Pixar-accidentally-d...
       | 
       | The ending is sad -- that intrepid recovery mostly didn't end up
       | used:
       | 
       |  _And then, some months later, Pixar rewrote the film from almost
       | the ground up, and we made Toy Story 2 again. That rewritten film
       | was the one you saw in theatres and that you can watch now on
       | Blu-Ray._
        
         | diskzero wrote:
         | I think we made How to Train Your Dragon three times at
         | DreamWorks. So many rewrites, director changes, character
         | redesign, etc. If you ever wonder why a feature length animated
         | film can cost over $300 million dollars, this is one reason.
        
           | mmcgaha wrote:
           | It is good to hear that big up front design sucks as much for
           | movies as it does for software.
        
           | coding123 wrote:
           | The dragon looks exactly like one of my cats. Thank you for
           | iterating.
        
             | hinkley wrote:
             | I have since seen a number of voids named Toothless. Some
             | of them weren't even bitey (which seems like false
             | advertising to me).
        
           | davidferguson wrote:
           | Oh you worked on _How to Train Your Dragon_? That 's my
           | favourite movie in the world!
           | 
           | Really random follow up question - there's a How to Train
           | Your Dragon advert that I assume played before the first film
           | in cinemas, advertising the cinema's popcorn and other
           | concessions, but the only two copies on YouTube are in really
           | terrible quality. (one in English
           | https://www.youtube.com/watch?v=f37fI14Ne0I, the other in
           | Finnish I think https://www.youtube.com/watch?v=W0Hr4mD3G1Q).
           | I really really want a high quality version of this, and I'm
           | wondering if you know of anyone at DreamWorks who I can
           | contact about this? I've tried tweeting at a couple of people
           | but unsurprisingly heard nothing back, so I'm maybe looking
           | for someone in marketing?
        
             | diskzero wrote:
             | You could try reaching out to Kate Swanborg. That sounds
             | like some sort of partnership deal that she may have been
             | involved with. Worth a try!
        
           | hinkley wrote:
           | Thanks for nothing, you useless reptile.
           | 
           | Seriously though, did they have to record the voice acting
           | three times, or partially again several times?
        
             | diskzero wrote:
             | Yes, the recordings get done over many times. Sometimes we
             | lose someone due to story changes or some other
             | circumstance. This is a bummer, as there is more than the
             | voice that is lost. While it is easier for some stars to
             | drive over to Glendale from wherever they live in LA, they
             | still "perform" the part and that performance is recorded.
             | The video of the voice over work is referred to by the
             | animators and influences a lot about the how the animated
             | character behaves. I suppose in less expensive productions,
             | you could just try and plop a voice over in, but all the
             | top studios will use the video reference and ask the talent
             | back for voice fixes.
             | 
             | In general, I found the talent to be really fun to be
             | around and no-one just phoned it in. They really liked to
             | work on the films. I ended up sitting next to Gerard Butler
             | at the employee screening of Dragon at the now non-existent
             | Universal Amphitheater. He stayed for the whole film.
             | Unlike me, he hadn't seen it in thousands of tiny chunks
             | during dailies, and had a blast.
        
           | pkulak wrote:
           | Worth it. Love that movie.
        
           | skunkworker wrote:
           | In college I remember hearing from one of my animation
           | professors that DreamWorks switched between NURBS and
           | polygons for How To Train Your Dragon, I'm unsure which
           | direction it was but I don't envy the artists tasked with
           | that if true.
        
             | diskzero wrote:
             | I don't know about that. I would need more info. The PDI
             | tool chain used for Dragon used the same clamped p-blend
             | curves that had been used for several generations of
             | movies. Technology was almost never the issue for delays.
             | It was usually story, direction, editorial, etc. If you get
             | more context, maybe I can find out more.
        
         | haswell wrote:
         | I don't know about sad necessarily - if the earlier iteration
         | was necessary to realize they needed to make the changes that
         | brought us the final version, it's all part of a larger
         | journey. I can see how this could be frustrating for people
         | involved, but reshoots / rewrites are part of the process.
         | 
         | And I suspect that the recovery of character models and other
         | assets was still instrumental in making the final film, if not
         | the actual shots/sequences themselves.
        
         | AdmiralAsshat wrote:
         | This seems to be par for the course for Pixar. I remember many
         | of the deleted scenes from the _Inside Out_ DVD were prefaced
         | with explanation from Pixar that they were from a much earlier
         | draft. They realized after working through the first draft that
         | they now had a much better understanding of the characters, and
         | then rewrote the narrative to match.
        
           | chx wrote:
           | Zootopia was also rewritten.
           | https://lostmediawiki.com/Zootopia_(partially_found_pre-
           | rewr...
           | 
           | > This decision came fairly late in the process, as animators
           | had been working on the film for almost a year with only 17
           | months until it needed to be completed.
        
             | hinkley wrote:
             | Nobody comes into a digital movie with all of their tools
             | already written and running at top speed, right? Isn't this
             | typically a process where frames per hour is pretty low at
             | first and ramps up as you go?
        
         | trynewideas wrote:
         | Considering how good Toy Story 2 was, "sad" is an odd way of
         | putting it, unless you're talking about the ensuing crunch.
         | 
         | There are a few more details about that side of the story
         | elsewhere:
         | 
         | https://www.independent.co.uk/arts-entertainment/films/news/...
         | 
         | https://www.denofgeek.com/movies/toy-story-2-disasters-almos...
        
           | nathanvanfleet wrote:
           | The sad part is that the thing they saved didn't actually get
           | used and was just thrown out. So it wasn't exactly the big
           | save it's made up to be. But if it wasn't framed as a big
           | save maybe it wouldn't be posted every week or so.
        
             | haswell wrote:
             | If the thing they saved hadn't been saved, it makes me
             | wonder if they would have remade it as written, and then
             | either have to throw _that_ out or release what was by all
             | accounts not a good first pass.
             | 
             | I suspect some character models were reused, and recovering
             | what they did enabled the rewrite that came after.
             | 
             | I think it's still fair to characterize this as a big save
             | even if it wasn't used as-is. They arguable needed the
             | first version to decide to move away from it.
        
       | [deleted]
        
       | locallost wrote:
       | I remember the stories from Ed Catmull's book Creativity Inc.
       | Although not that they tossed the movie out after almost deleting
       | it anyway, like it's said in the article. The story about
       | starting from scratch was that they left the team that did it
       | alone, but it was their first movie and it fell flat. Reading
       | about it then felt like a dick move and kind of questionable, but
       | even more so later when I'd read that Lassiter, who was the most
       | critical one, is a certified creep / jerk. So I don't know, but I
       | guess if you're successful in the end your opinion is never
       | questioned.
        
       | navels wrote:
       | Do people not search HN before posting?
        
       ___________________________________________________________________
       (page generated 2022-10-26 23:01 UTC)