Walk Softly and Carry a Big Schtick

Yet another post about my restoration progress. I've hit a particular milestone: All of the photos recovered from the dead hard disk using the "Encase" professional-grade package have been uploaded into my temporary working directories on the server.

This is a major accomplishment. And not just terms of disk storage. (Right now my directories for the work in progress add up to 24 GB.)

With my gizmo that automatically renames the directory with the datestamp (at least those directories containing high-res images all from the same year), I can easily see which photos are from (for example) 2004.

( There are 173 directories that now have '2004' in the name... )

You can see that I've started annnotating the directory names to include a key word or two. I'm about halfway through with that. I'm going to go through the ones that just say "2004" and split them up into dates; then I'll start tackling specific dates. I'll probably start with January, unless anyone wants to express a preference for any particular photos from 2004 on my website that you'd like to see restored first....

I'm really excited that I can now target specific pages of the website to be worked on, instead of just taking them in order by where they were located on the old disk drive!

Naruto night (06-Jun-2007) photos are now online.

So there was a screening of the (now 5 years old) movie, Naruto the Movie: Ninja Clash in the Land of Snow. library_lynn and I decided to see the showing at The Block in Orange. ( They were a lively, enthusiastic audience. )

Many of us stayed for the featurette afterwards. They ended with a bunch of the cast and crew --including on the Japanese side -- thanking all the fans who watch the show "and sometimes dress like us." I hope that makes onto the DVD version!

We saw one or two fans dressed up in costumes. The photo shown here includes "Cigam" who is the lady now in charge of the Orange County Anime Meet-up, usually on the 2nd or 3rd Saturday of the month. (Picked not to conflict with the L.A. group, but ironically it means it's on the same day as either CFO or Cinema Anime's screenings at the LASFS clubhouse.)

I missed the meet-up this weekend -- did some chores at my folks' house, worked on Hazel's Picture Gallery, and went to the Hugger House potluck. But I'm hoping I can go next month, on the 21st.

This afternoon, Sunday (Father's Day), Lynn and I are going to a "Cosplay Ice Skating" event here in Anaheim. Mostly to support the event and take pictures. Next weekend is Stepfather's Day. Then the following weekend (yikes, it's coming up soon) we head out to Long Beach for Anime Expo.

I mentioned I've been working on Hazel's Picture Gallery. Specifically, I've been working on the restoration project. Matthew Randolph has been very helpful in finding thumbnails and whole images, and I've been matching up what he's recovered to files from the old disk. (There are thousands of JPEG images, so it takes time to go through all of them.) I am pleased to announce that we've been able to bump up the "Percentage Restored" figures. The thumbnail images are now 77% restored, and the large-size picture collection as a whole is 44% restored.

The damage, and the restoration, are not spread evenly over the photo collection. The section hardest hit consists of the photos from 2004. Those were too "new" to have been picked up by most of the archiving mechanisms on the web, but too "old" to still be sitting on my various computers. So I'm really pleased that a lot of the work that has been accomplished lately have been from 2004. (For example, I went to three weddings in 2004. Incidentally, if you invite me to your wedding, I'll probably go around taking pictures of everyone attending -- it gives me something to do, a purpose.) The thumbnails for 2004 are 49% restored so far.

As an example of how slow these things go, the jump from 43% to 44% represents about a week of hard work on my part, mostly hours late at night... I try to keep working on it -- and not let it go without progress for weeks or months at a time -- and the restoration percentage is one of the sub-goals I set myself. "I'll stay up until 2am again tonight, so maybe I can get it up another notch" is basically what's going through my mind if you happen to see I'm online late at night.

(I light up my AIM when I'm online -- my AIM handle is the same as my LJ handle -- and shut it down again when I'm done with my computer. So it's easy to see if I'm here at the computer, even if the AIM system times me out for inactivity and says I'm "away.")

( I made up ''I supported Hazel's Picture Gallery'' graphics. Here are, in no particular order, the faces of the first six people who contributed money to the restoration project... )

LiveJournal trivia question: Which five of these six have LJ handles?

Want to see more of them? (Warning: lots of images, takes some time to load.)

Incidentally, I'm not mentioning this because I'm passing the hat again. Donations are always a surprise when they come in. But throwing more money at the task will not make it go any faster -- it's almost all in my court, and it's mostly a matter of finding the free time. The donations that came in during 2005-2006 were enough to pay for the program used on the dead drive to recover the files and to pay for two years of professional hosting service. Donations received since then are going into the general fund, mainly for continued hosting charges and to pay to replace my digital camera when I lose it in a hospital room.

I'm putting out another call for people who want to try to recover JPEG files by judicious hex editing. Snipping bytes 0xC000-0xCFFF from them didn't work on these, but I'm hoping it'll be almost as simple a process. Who wants to help?

First directory: www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-bad/jpg-08-00/.

Cosplay Picnic at Laguna Niguel (08-Apr-2006), Party at Janis and Glen's House (08-Apr-2006), AniZona 2 (14/15/16-Apr-2006), Lux Barbecue and Orientation (23-Apr-2006), and Brandon's 21st Birthday Party (07-May-2006) photos are now online. Click on the picture of rema_suzuki (leader of the rum_group out in Riverside) to see the anizona pictures.

Okay, great progress; everything's stable enough for me to resume adding pictures. Slightly new feature -- I changed the Cross-Reference index page photo selection. Instead of picking the first person with that initial to show you, it picks the one I have the most pictures of. So it should not be any surprise that for B you'll see my lovely wife library_lynn. Nor is it a surprise that R's picture is the daughter of my heart. (Who graduates from UC Irvine in about an hour and a half -- I better disconnect and get over to the campus.)

Okay, I've got the cross-reference back online. You can click on the picture of Maria, Harlie the Dog, and library_lynn to see an example of a recently cross-referenced page. Nothing groundbreaking -- there are links to the names of the people cross-referenced, and everything's starting to look back to normal.

So I can get started on typing in the captions, because they'll get processed fully now..

[Update at midnight: I started typing in captions received, and came across a new name that needs to be added to the cross-reference engine. So that "add name" feature has to be rebuilt next for me to continue.]

Photos of Maria and Friends Part 21 (Jan/Feb-2006), LASFS and Aftermeetings part 251 (Jan/Mar-2006), and St. Jude Hospital (23-Jan-2006) are now online. Click on the picture of selinawoman if you'd like to see the LASFS pictures, including Grace Molloy's first trip to lasfs. (Baby's first club meeting.)

The cross-reference name index pages are all regenerated in the new style, except for the index indices. That is, you can find Naomi Fisher under "F," but the "F" page itself hasn't been regenerated, and the new cross-referencing engine isn't online. But there are a couple of new wrinkles in the name index pages themselves, you should take a look if you haven't already.

(Selina, when you see this, write me -- I'd like to assign you a different task if you still want to help with the website.)

Oh, and some time ago we passed the 10,000-page mark. We have over 17,000 photos complete on the website, and if you count thumbnails-only we have 20,000 photos visible; and over 3700 picture pages displaying them plus over 6400 different names cross-referenced, adding up to 10,232 individual HTML sub-pages. As of those three above going online. And all 10,232 of those files have been recreated recently from the database, with the new look. [Most of them have been regenerated a couple of times, in fact; I just thought of another variation for the picture pages, for large batches, such as our UK trip for Interaction last year.]

In other news: does anyone local have shorty ethernet patch cables? I need one. 12" is plenty.

Photos of some Chocolate Bears (20-Feb-2006) are now online.

Inoue-san gave me this lovely box of chocolates - a tin, actually, containing chocolates with a teddy bear theme.

Update on the website, inside and outside: I'm experimenting with a new look to Hazel's Picture Gallery; click on the tin of bears to see a sample. And on the inside, I'm almost done putting the tools back together...

More Portland Westercon 2001 pictures. Anyone recognize these people? Or can tell me which days they were taken?




In other news: I've found some of my notebooks full of printouts taken from Hazel's Picture Gallery, usually from when I'd just uploaded photos and I was trying to match up captions to them. So that will help with some of the recovery. I don't think I've found all of the notebooks, either, because hardly any of the 2001 photos that I've been working on recently are shown in the ones I've found... Oh, and our recovery percentage currently stands at 39%. Since I started tracking these figures back in March, I've fully recovered over two thousand photos; about half of them (997) from 2001.

Here's one from Kansas City -- Dee's funeral in 2004. I think this is a picture of Rick Lancaster and Lance Mallis, can anyone local confirm?

I've been going through the files recovered from the disk drive, plus the photos (thumbnails mostly) that I was able to get from the Wayback Machine. I've been correcting corrupted photos (when the corruption consists of garbage inserted at offsets 0xC000-0xCFFF, anyway), and trying to match up thumbnails -- and making educated guesses where I don't have the original thumbnail image at all. It's not that hard, when I can recognize the faces, or when I've written down their t-shirt slogan in the caption. But sometimes I need a little help.

How about this one on the left: is this Judith Moore?

How about this one on the right: is this Ashley Moore? It looks like there's a family resemblance...

The Hazel's Picture Gallery recovery story so far.

It occurs to me, especially after meeting Ashley, Chloe, Kate, Katherine, and Erin this afternoon, that not everyone who follows my LJ knows what this server recovery thing is all about. Allow me to explain. (The regulars, you know all this and can skip this post if you like.)

I bought my first digital camera several years ago, and started taking it with me to conventions. In fact I use my camera at most of the significant events in my life, especially activities that involve people or things I don't see every day. I upload my pictures to my website; I have a website for myself and library_lynn at www.boston-baden.com. (Bostonbaden.com, without the dash, is the domain I use when I give people my e-mail address. E-mail works either way, for my primary e-mail addresses -- hazel or chaz, and hazelchaz works too -- and you can get to the website with or without the dash too, it'll redirect you if it needs to.)

The pictures I upload are cut down into two sizes in addition to the high-resolution images; the high-res images aren't always put online, but a viewing-size (approx 640x480) and a thumbnail image (about 100 pixels high and wide) go with them. The web pages themselves are plain ordinary HTML pages; originally generated by a program and hand-edited, then later generated by a program using a crude DB_File database, and now generated by programs that use a MySQL database. (Dreamhost makes it easy to do MySQL databases. I love 'em.) I haven't finished rewriting the programs, though; so part of the unfolding story here has to do with my software work for the site.

Last summer, after several years of taking and uploading photos, I'd passed the 25,000 mark -- and then we had the great server crash on the machine in Minneapolis that had been hosting my domain. Yikes! He thought I had backups for everything; I had thought (for quite some time, actually) that a new mondo backup unit was just a short while away from coming online, and hadn't disk space to save my photos. (Ironically, it didn't occur to me to copy them onto a machine with cd-rom burning capability, even though I'd been burning cd-roms full of MP3s from all of my CDs for the home music project.)

So I signed up for a paid hosting account on Dreamhost and started rebuilding from what backups were available. By winter, the dead easy work had already been done: the stuff that was easily rescued off of the old computer, the files that I had copies of in one form or another (mostly from 2000-2002, plus a scattering of recent photos that hadn't been deleted off of my primary machine), and so forth.

The disk drive from the old server has been subjected to a professional-grade recovery process, by somebody that works in the data-recovery field and knows his stuff. (Thanks, dspisak!) Many people donated money against the possibility of our having to pay a thousand dollars to a data recovery company; the first one to explicitly say she'd put in $20 was Naomi Fisher, and when I put the tip jar on the website the money came rushing in. Some of that money went to the software Dan needed, and the rest is being applied against the hosting fees at Dreamhost. We have two years prepaid at this point, and I really appreciate the donations. (I need to start writing thank-you notes, in fact. But that's another story.)

The data recovered from the old server, however, is generally in the form on unidentified, unmarked, uncaptioned, unnamed and undated JPEG files. And a great deal of them seem to be corrupted to boot. At first I was worried that we were seeing corrupted data from the drive; now I believe that the recovery program may have not completely understood the filesystem on the server, and the data are all there, they just need some work in a hex editor.

A great deal of work has been done on that front, notably by testerscot, bovil, johno, and lately gvdub. (My apologies if I've left out any of the hex editor hacking crew, I'm typing this from memory.) So the result of that is hundreds and thousands of perfectly good, ready-to-reinstate photos, we just have to figure out where they go on the server.

Meanwhile. Several (at least half a dozen) people have brightly suggested that The Wayback Machine Internet Archive might magically solve all my problems. Well, sort of, a little, not as much as you might expect. First of all, while the Wayback Machine has been saving copies of every web page it can lay its spider paws on, that doesn't always include all of the images on the pages. Or the images that are linked from the pages. And, they don't get every page, although they do index and save a lot.

I knew using the Wayback Machine would basically get a number of my thumbnail images, and a smaller number of the larger images. I'd held off investigating what I could get back, because I knew I would often just get a thumbnail and not the matching large image -- and if I just restored the thumbnail to a page, you wouldn't be able to tell that the full size picture was still missing. This lack, however, is something I can deal with on the new MySQL platform. And, even just a thumbnail helps, because the other part of the recovery project is yielding up perfectly good large images. So now the task of dovetailing the results from the two approaches begins.

I think that's the gist of it. I go to about 12 conventions a year, science fiction and animé now that I've started our little animelosangeles convention; i take a lot of photos by sf fan standards, but I'm strictly small time compared to the people snapping photos at animé conventions. I am now about two or three conventions behind, plus a few parties and picnics, in my picture-uploading -- this is because I have other projects apart from the website software I mentioned, and the data recovery, there's a big convention (or a medium-sized one, if you come from an animé convention viewpoint) that I'm head webmaster for, and... well. You're up to date. Stay tuned here for further updates.

Any questions?