[HN Gopher] A baby saved 'Toy Story 2' from near complete deleti...
___________________________________________________________________
A baby saved 'Toy Story 2' from near complete deletion (2021)
Author : walterbell
Score : 100 points
Date : 2022-02-14 11:25 UTC (11 hours ago)
(HTM) web link (insidethemagic.net)
(TXT) w3m dump (insidethemagic.net)
| CharlesW wrote:
| "Pixar Studio Stories: The Movie Vanishes", narrated by Oren
| Jacob and Galyn Susman:
| https://www.youtube.com/watch?v=8dhp_20j0Ys
|
| "Did Pixar accidentally delete Toy Story 2 during production?",
| Quora answer by Oren Jacob: https://www.quora.com/Pixar-
| company/Did-Pixar-accidentally-d...
| gxqoz wrote:
| The way I heard this story is that the movie being saved wasn't
| actually the theatrical Toy Story 2. Instead, it was Disney's
| version that was mostly scrapped when they resolved their dispute
| with Pixar.
| signal11 wrote:
| This story is oddly similar to the Maersk cyber-attack in
| 2017[1]:
|
| > The virus, once activated, propagated within just seven
| minutes. And for Maersk, as with other organizations, the level
| of destruction was enormous.
|
| > As well as affecting print and file-sharing capabilities,
| laptops, apps and servers, NotPetya severely damaged both
| Maersk's implementations of DHCP (Dynamic Host Configuration
| Protocol) and Active Directory, which, respectively, allow
| devices to participate in networks by allocating IP addresses and
| provide a directory lookup for valid addresses. Along with those,
| the technology controlling its access to cloud services was also
| damaged and became unstable while its service bus was completely
| lost.
|
| > Banks is candid about the breadth of the impact: "There was
| 100% destruction of anything based on Microsoft that was attached
| to the network."
|
| > In a stroke of luck, they were able to retrieve an undamaged
| copy of its Active Directory from the Maersk office in Nigeria.
| It had been unaffected by NotPetya purely because of a power
| outage in Lagos that had taken the service offline while the
| virus was spreading. A rather nervous local Maersk employee was
| tasked with transporting the crucial data to Denmark.
|
| What this says to me is that multiple offline backups (or failing
| that, copies) will save your bacon some day.
|
| [1] https://www.i-cio.com/management/insight/item/maersk-
| springi...
| monocasa wrote:
| That's one of the ideas behind use of tape storage today
| interestingly. It's pretty hard (although not impossible) for
| hackers to get at tapes sitting on a shelf.
| 999900000999 wrote:
| If I write a text file to a USB drive without any iot
| features, and then throw the USB drive in my closet I'm not
| seeing how anyone could hack it .
|
| I guess you could drop an emp pulse on my house and fry the
| drive, but absent that it's safe.
| tempnow987 wrote:
| They key is usually something that is pretty current but
| can't be corrupted.
|
| A drive in a closet doesn't get you there. The nigeria
| office offline due to power does.
|
| I feel good with S3 with an object lock and retention
| policy for backups.
| rectang wrote:
| Pretty current, can't be corrupted, _and complete_.
|
| Unless you're practicing continuous restoration to ensure
| that your backups are actually backing up what you need,
| the odds that you got everything critical are not good.
| tempnow987 wrote:
| The idea that throwing a USB in a drawer does better is
| absolutely ridiculous.
|
| I think this illustrates where folks can go wrong.
|
| Many folks and business TODAY are backing up to NAS /
| RAID arrays etc. Many are easily corruptible by
| ransomware. We see this REPEATEDLY with many infections
| of large orgs.
|
| The basic story here is that something not editable has
| great value even if not perfectly current. If you do a
| diff every 4 hours you have covered a LOT of ground
| already.
|
| I've sat through continuous DR architecturing discussions
| - the massive cost / complexity / scale can just be out
| of control for many users.
|
| In many use cases Active Directory for example behind by
| 4 hours is not game ending. With notice to users that
| recent edits may not be captured you may be able to get
| back up and running pretty quickly.
| rectang wrote:
| I'm not singing the praises of "a USB in a drawer" -- the
| medium doesn't matter, the number of copies doesn't
| matter. The only criteria I am considering is whether the
| backups are tested to prove that they actually work as
| intended.
|
| There _are_ shades of gray, here: continuous restoration
| is a best practice, but backups which are tested even
| once, and perhaps incompletely, are better than backups
| which are never tested at all.
| monocasa wrote:
| Well, as a backup, USB drives don't have that great of a
| lifespan compared to tapes. They're also not nearly as
| space dense or competitive for $/GB once you've amortized
| the drives.
|
| And for an org, there's still attacks in both cases. Do
| your backups get requested through an automated system?
| Well you've just made a very slow tape library with people
| rather than robotics. Are the tapes/USB sticks encrypted
| and the encryption keys stored in an online system? Then
| the hackers have a mechanism to deny you access to the
| contents of the tapes, which is probably their goal anyway.
|
| Yes a USB drive with a text file is pretty safe from such
| attacks, but it also doesn't scale to the backup concerns
| of even relatively small orgs.
| [deleted]
| Damogran6 wrote:
| It's not the backup, it's the -restoration-.
|
| TheBadGuy changes the encryption key used to make the
| backups, then waits %long enough%, you're still boned.
| monocasa wrote:
| There's no perfect scheme, but the best mechanisms I've
| seen for this in practice involve HSMs so that nobody, bad
| guy or good guy, can change the keys used for encryption
| without physically pulling the module, and it can be
| disconnected and stored at a safe location once the backup
| is complete. Each redundant set of a backup has an HSM, and
| the HSMs are cold stored in different physical locations.
|
| But yes, this is hard to get right as I hinted by the "but
| not impossible" quip.
| hughrr wrote:
| Many years ago I was working for a defence contractor and
| an idiot hit the TOTAL ERASE button on an HSM. This all
| went to hell when no one could work out how to load the
| keys into it again for nearly two weeks...
| jonny_eh wrote:
| > What this says to me is that multiple offline backups (or
| failing that, copies) will save your bacon some day.
|
| This is why distributed version control system like Git are so
| nice.
| notorandit wrote:
| A mom saved Ts2!
| diego_moita wrote:
| John Catmull (correction: sorry, his name is Ed Catmull, as tbab
| corrected me), Pixar co-founder and president wrote a book
| ("Creativity Inc") where he tells that story in detail.
|
| He wrote that everybody in the team had forgotten about Sussman's
| copy and she is the one that told in a meeting the copy existed,
| to the astonishment of everyone else. The van that went to pick
| up the computer on Sussman's house was equipped with pillows
| because they were scared even from the road's vibrations.
| tbabb wrote:
| Ed Catmull.
| [deleted]
| diego_moita wrote:
| You are right, I was wrong. Thank you.
| coldpie wrote:
| Would you recommend the book?
| filoleg wrote:
| If you are even slightly curious about the history of Pixar
| and the decision-making process behind their actions, then it
| is 100% worth it. It was written extremely well.
|
| If you are trying to read it purely for business advice or
| anything of that nature, you might find it a bit eh.
|
| It is very heavy on telling an interesting story, which
| should explain rather well why it might not be the best for
| the purpose of being an educational material. But if you want
| an interesting delivery of the story of Pixar, along with a
| look at behind the scenes in the industry (e.g., Steve Jobs
| was featured in there as one of the important personalities,
| given he was heavily involved with Pixar) and behind some of
| the decision-making/philosophy of Pixar as a company as it
| was growing, it is a great and entertaining read.
| diego_moita wrote:
| Depends on the subject of your interests.
|
| Roughly speaking the book validates the whole idea of "to
| build awesome stuff, hire the most creative and talented
| people you can find and create an environment for their
| imagination to run free". If you're interested in this
| perspective, I do very much recommend.
|
| But, obviously, he doesn't address issues like John Lasseter
| running free with sexual harassment and other uncomfortable
| issues.
| trimbo wrote:
| Does he address his own efforts to suppress wages for those
| employees?
|
| https://www.courthousenews.com/disney-to-settle-animators-
| wa...
| peter303 wrote:
| I have done forensic reconstruction of UNIX files. The directory
| node list is erased/lost, but the raw data remains on disk. This
| assumes the data has a sequential structure. The article didnt
| say whether this was the model data or the rendered data.
| abotsis wrote:
| This seems to be less about how a baby saved the day, and more
| about how an employee exfiltrated data from her employer. As a
| cybersecurity guy at a large financial institution, I wonder what
| Pixar's policy was about the matter then, and what impact it had
| on the same policy today.
| abotsis wrote:
| Why this got downvoted a bit is beyond me, I'll never
| understand hn.
|
| Oren, who worked at Pixar recounts the story, and said "She has
| an SGI machine (Indigo? Indigo 2?) Those were the same machines
| we had at all our desktops to run the animation system and work
| on the film, which is what she was doing. Yes, it was against
| the rules, but we did it anyway, and it saved the movie in the
| end."
|
| https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
| adl wrote:
| The article is wrong, it wasn't her 'personal computer' it was
| a Pixar Workstation that was at her home, not the same thing.
| ehPReth wrote:
| From what I recall from another telling of the story Pixar was
| the one who set her up with a home computer & storage system
| for that purpose - since the storage and computing power needed
| was higher than what most people would have at home - but I
| could be misremembering
| ckozlowski wrote:
| You're on the right track. As told in the Next Web story, the
| workstation in question was a Silicon Graphics Octane or
| there abouts. These were incredibly expensive and were
| provided by the company. In addition, according to the story,
| they were the ones who set up the file sync for her to be
| able to work from home.
|
| LGR and RMC have both done videos on the SGI Octane and are
| worth watching: https://www.youtube.com/watch?v=ZDxLa6P6exc,
| https://www.youtube.com/watch?v=VHsA8iq4N0s
|
| Edit: It was Retro Man Cave, not Adrian's Digital Basement as
| originally written.
| peter303 wrote:
| Home computers where pretty dinky and broadband slow at the time
| of Toy Story 2. I wonder how they'd get 90% of the movie
| transferred home and saved to disk/tape?
| fulafel wrote:
| This sounds like there's a whole other story in the aftremath,
| would be interesting to hear how they dealt with it (deciding
| between dropping scenes vs reproducing):
|
| > Jacob admits that about 10% of the film's files were never
| retrieved and are gone forever, but they were able to make the
| film work without the scenes.
|
| Also, this seems to be a second hand rereporting, the linked
| article has a little more detail
| (https://thenextweb.com/news/how-pixars-toy-story-2-was-delet...)
|
| It also has the real beef of the story:
|
| > Pixar, at the time, did not continuously test their backups.
| This is where the trouble started, because the backups were
| stored on a tape drive and as the files hit 4 gigabytes in size,
| the maximum size of the file was met. The error log, which would
| have told the system administrators about the full drive, was
| also located on the full volume, and was zero bytes in size.
| adl wrote:
| That link provides more info about the story, I remembered
| reading about it years ago, Also after they restored most of
| the files deleted, most of the movie was scraped and redone.
| frosted-flakes wrote:
| Scraped or scrapped? Can you explain more?
| kwhitefoot wrote:
| Many years ago I was responsible for backups in a mixed
| Unix/Windows system. I asked for funds and time to do a
| disaster recovery exercise in case a server system disk failed
| but was turned down flat. Luckily we never had to restore
| anything more than a few individual files.
|
| It's not enough to confirm that the backup has actually been
| created you need also to have the procedures necessary to do
| the restore and the personnel and hardware to do it.
| rectang wrote:
| Modern development practices -- idioms, frameworks, libraries,
| traditions -- make it difficult to test your backups. The
| priority on responsiveness means that you have to work extra
| hard when architecting the system to ensure that you can
| restore from backups without losing recent changes -- which
| makes it impractical to confirm that your backups actually
| work.
|
| Offering _Continuous Restoration_ as a feature should
| theoretically be a way of differentiating a development tool in
| the software marketplace. But short-termers hold power in so
| many organizations that it 's not clear to me whether it would
| actually win. Even if ransomware kills the company, executives
| still keep their money.
| ckozlowski wrote:
| At AWS, we use this story in some of our presentations to
| quickly illustrate the importance of backups and testing them.
| It also highlights the fact that DR scenarios can often arise
| from simple mistakes such as "rm -rf", and not the "hurricane"
| or other global event example that often gets overused. A good,
| cautionary tale.
| commandlinefan wrote:
| I've read that story a few times and I'm always struck by how
| aware management was when that _did_ happen. I suspect that, at
| least half of the places I 've worked, if I (or anybody else)
| happened to be that one employee who had a backup on their home
| computer, they'd find it impossible to communicate that to upper
| management in any way. The whole movie would be shut down while
| somebody was saying "I have a copy over here! I have a copy over
| here!"
| anyfoo wrote:
| Were you actually in the situation? Because at practically all
| the places that I worked for (some of them smaller, some of
| them very big), I don't think this would be a problem if so
| much was at stake...
| Kaibeezy wrote:
| The same genius manager who failed to hire someone to make
| backups must have gone on to run the graphic design team for
| _Avatar_.
| paulpauper wrote:
| backup your backups
| monocasa wrote:
| 3 - 2 - 1
|
| 3 backups
|
| 2 media formats
|
| 1 off-site
| dspillett wrote:
| Having multiple backups, on unconnected systems, is the general
| rule.
|
| Whether this is done in a chain (backing up the backups) or as
| multiple transfers from the sources is an implementation
| decision: ideally the latter as otherwise a break in the first
| backup system can stop the second updating, _but_ this may
| require notably more resources at the source end so compete for
| those resources with production systems so the former isn 't
| always a worse plan.
|
| More importantly: _test your backups_. Preferably in an
| automated manner to increase the change the tests are not
| skipped, but have humans responsible for checking the results.
| Ideally by doing a full restore onto alternate kit, but this is
| often not really practical. A backup of a backup is no extra
| help if the first backup is broken in a way you have not yet
| detected.
| CobrastanJorji wrote:
| More importantly: test that you can restore from backup. If
| you're not confident that you can restore from backup, you
| don't have a backup.
| closetohome wrote:
| Two is one, one is none.
| high_pathetic wrote:
| Clickbait. Did the baby actually and literally saved Toy Story 2?
| No.
|
| By that logic, another title could be "A passionate, unprotected
| sex saved Toy Story 2".
| tagoregrtst wrote:
| Lol, agreed, but common! It was cute.
| Kaotique wrote:
| If you follow the logic of the title of this article the new
| policy was that there should always be a mother with a baby on
| the team.
| tagoregrtst wrote:
| No. Instead you should always treat mothers, especially new
| moms, with extra respect and care. They have a lot to deal
| with.
| lanius wrote:
| How common was working from home in 1995?
| tagoregrtst wrote:
| Work for home was what they put in their briefcases before
| computers.
| jedberg wrote:
| It was 1998, and it wasn't that common, but in tech it wasn't
| impossible. For Pixar it was hard because of the size of the
| files, but otherwise wasn't all that hard.
|
| I occasionally worked from home in 1998. The biggest impediment
| was network speed -- at work we had 10mbps and sometimes even
| 100mpbs. But at home all we had was dail up or if you were
| really lucky, DSL, which was 0.384mbps, so 30 times slower, and
| also latency was a lot higher.
|
| So you were very limited in what you could do. Mostly it was
| just email and remote access ssh with tmux, where you sometimes
| had to deal with really high latency.
| anthk wrote:
| More like GNU screen and telnet over VPN's, SSH was from late
| 90's.
| jedberg wrote:
| We had ssh but you're right it was screen not tmux
| jonny_eh wrote:
| In this case, if you were an animator, you could still create
| models/animations, but you wouldn't be able to sync up with
| the rest of team on a daily basis. Perhaps a little
| frustrating, but with decent communication it would have been
| manageable.
| macintux wrote:
| My employer paid for ISDN in 1999 and I thought I'd reached
| nirvana. It wasn't dramatically faster than modem, but an
| always-on connection directly to the corporate network was
| quite handy when getting paged at 3am.
| jasonwatkinspdx wrote:
| I worked remote in 98. Depending on your location it wasn't
| quite that bad. I had SDSL at 1.54 mbit in those years and it
| was workable. You could get ADSL here then too for a bit more
| download at the cost of upload and overall connection
| stability. Granted, I was doing gamedev which had smaller
| assets than a film, but you could manage a bit more than just
| a terminal a lot of places.
| bityard wrote:
| Unless you were an independent entity that could do most of
| your work over the phone with only occasional in-person
| meetings (e.g. certain kinds of salespeople and some artists),
| it was very rare.
|
| It only happened in this instance because the person in
| question had a newborn baby and I guess maternal leave wasn't a
| thing at Pixar back then.
| jedberg wrote:
| > and I guess maternal leave wasn't a thing at Pixar back
| then.
|
| The opposite. After she finished mat leave she was allowed
| the special exemption to work from home so she could still
| spend time with her baby.
| adrianomartins wrote:
| Wow, this makes me think, do animation teams don't really use a
| version control system to have multiple people contributing to
| the same movie at the same time? Does anybody knows how that
| works?
| rkapsoro wrote:
| IIRC perforce is (was?) commonly used in teams with workflows
| involving large assets.
| cecilpl2 wrote:
| Perforce is still the defacto standard in the video game
| industry where it's not uncommon for a game's source assets
| to run into the tens of terabytes.
|
| That said, Toy Story 2 was developed in the late 90s, and
| while Perforce existed then I don't know how popular it was.
| KaiserPro wrote:
| Oh they 100% do use a version control system.
|
| Your standard "shot" will contain 10s or hundreds of versions
| of animation, lighting, texturing, modelling & lighting. Not to
| mentions compositing. They are often interdependent too.
|
| A movie like shrek will have literal billions of versions of
| $things.
|
| > Does anybody knows how that works?
|
| This changes by company, but everyone tends to have an asset
| database. This is a thing that (should) give a UUID to each
| thing an artist is working on, and a unique location where it
| lives (that was/is an NFS file share, although covid might have
| changed that with WFH)
|
| Where it differs from git et al is normally the folder tree is
| navigable by humans as well so the path for your "asset" will
| looks something like this:
|
| /share/showname/shot_number/lighting/v33/
|
| There are caveats, something like a model of a car, or anything
| else that gets re-used is put somewhere else in the "showname"
| folder.
|
| Now, this is all very well, but how do artists(yes real non
| linux savvy artists) navigate and use this?
|
| Thats where the "Pipeline" comes it. They will make use of the
| python API of the main programs that are used on the show. So
| when the coordinator assigns a shot to an artist, the artist
| presses a button saying "load shot" and the API will pull the
| correct paths, notify the coordination system (something like
| ftrack or shotgun) and open up the software they normally use
| (maya, zbrush, mari, nuke, etc etc) with the shot loaded.
|
| Once the artist is happy, they'll hit publish.
|
| The underlying system does the work of creating the new
| directory, copying the data and letting the rest of the artists
| know that there are new assets to be pulled into their scene as
| well.
|
| Then there are backups. Again this is company dependent. Some
| places rely on hardware to figure it out. As in, they have a
| huge single isilon cluster, and they hook up the nearline and
| tape system to it and say: every hour sync changes to the
| nearline. every night, stream those to tape.
|
| Others have wrapped /bin/rm to make sure that it just moves the
| directory, rather than actually deletes things.
|
| Some companies have a massive nearline system that does a
| moving window type sync, so you have a 12 hourlys, 7 dailies
| and 1 monthly online at once. The rest is on tape. The bigger
| the company, the more often the fuckup, the better the backups
| are tested.
| tbabb wrote:
| Pixar used RCS at the time. Problem is, when you run `rm -rf
| /`, that deletes the RCS directories as well.
| doublerabbit wrote:
| Not really. With multiple animators working on files
| simultaneously, its kind of hard.
|
| Sadly my animation knowledge has left me since I left the
| studio working gig three years ago. But we did create content
| for netflix and many animators came up sheepishly because they
| deleted a folder for it to be restored. It's not as uncommon as
| you think.
|
| FWIW, the live archive/backup server was called Dumbo. Which
| were 3x 4u Supermacho chassis with over 1.2PB in drives served
| over 10Gbit to each workstation connected to 1Gbit running
| CentOS 5. I dropped the new chassis while racking once and is
| partly the reason to why I lost my job :/
| doublerabbit wrote:
| Can't edit my post, but for clarity. Im wrong on the line
| "Not really. With multiple animators working on files
| simultaneously, its kind of hard." see comments above.
| blhack wrote:
| The data was saved because a mother was working from home with
| her baby, and was keeping a backup of the files there.
| fabiancook wrote:
| TLDR birth causing WFM, producing a verified backup chain.
| irrational wrote:
| 10% of the movie was lost forever? Well, crap. Now I'm sad for
| the day.
| philistine wrote:
| 100% of the movie they were working on ended up lost. The team
| working on the film was not Pixar, but another studio assigned
| the job while Pixar were holding up during negotiations.
|
| Once those negotiations were over, Pixar started work on a
| sequel, and this version was never released.
| irrational wrote:
| LOL Now I want to see that other version!
| tagoregrtst wrote:
| Dont be, 90% of the film was recovered.
| orenjacob wrote:
| Hi folks, I'm the Oren mentioned in the story(ies) above. This
| comes up on HN every year or two.
|
| Quick reference to lots more detail posted here ->
| https://www.quora.com/Did-Pixar-accidentally-delete-Toy-Stor...
| ldamerow wrote:
| You beat me to it. :D
___________________________________________________________________
(page generated 2022-02-14 23:01 UTC)