hazelchaz: (Default)

Yet another post about my restoration progress. I've hit a particular milestone: All of the photos recovered from the dead hard disk using the "Encase" professional-grade package have been uploaded into my temporary working directories on the server.

This is a major accomplishment. And not just terms of disk storage. (Right now my directories for the work in progress add up to 24 GB.)

With my gizmo that automatically renames the directory with the datestamp (at least those directories containing high-res images all from the same year), I can easily see which photos are from (for example) 2004.

There are 173 directories that now have '2004' in the name... )

You can see that I've started annnotating the directory names to include a key word or two. I'm about halfway through with that. I'm going to go through the ones that just say "2004" and split them up into dates; then I'll start tackling specific dates. I'll probably start with January, unless anyone wants to express a preference for any particular photos from 2004 on my website that you'd like to see restored first....

I'm really excited that I can now target specific pages of the website to be worked on, instead of just taking them in order by where they were located on the old disk drive!

hazelchaz: (Default)

I thought I might explain something about how the Hazel's Picture Gallery recovery process is coming along. I just spent a couple hours working on the website, and our "Photos Lost" statistic still sits at 44% restored, 16,221 photos of 29,108 still lost. Hasn't gone up at all tonight. So what have I been doing?

Well, I've been sorting out JPEGs uploaded from the dead drive recovery discs into directories, They get uploaded into directories of 100 each. The first thing I do is skim through them to sort them out by event.

Here's two sets of directories... )

The "Initial Upload" shows what a collection of directories (in this case, from batch 38) looks like before I start any analysis. The first program I run on the server sorts out the corrupted files and renames all the directories.

You'll see that the "Before" list just has "jpg-19-" followed by a number, and then either nothing else, "bad", or all or part of a datestamp such as 2003, 2003-01, or 2003-01-11. The "After" list shows what it looks like after I've gone through and split up some of them. (These are actual snapshots of some of my current directories -- I've gone through the 17's but not the 19's, and thought you might interested in the gory details of the work I'm doing.) You can see that I'm not done with this phase for batch 17. For example directory jpg-17-10 doesn't have any kind of identification. So if there are original images with timestamps, they're not all from the same year. Or there aren't any original high-resolution images at all. I have to go and look at them to see what I can see.

(Why does one batch go up to 17, one to 45, and one to 31? Because each batch represents about 300-400 MB, and on the original data recovery DVDs they were directories full of thousands of files. I split them up into 100-file directories, and sorted them out by file extension -- JPG, GIF, etc. -- before they're uploaded to the server.)

If you want to see more about the process, I've made a copy of the jpg-17-23 directory with more annotations. Details enough to make your eyes glaze over... And there's a picture of [livejournal.com profile] neo_serenity in there, by the way. But that photo wasn't lost (or has already been restored). Part of the challenge here is identifying which photos from the old hard drive don't need to be restored because they're already in place -- so I potentially have to go through all 29 thousand, not just the 16 thousand that are missing!

Anyhow, what I was saying the other day was that if I can restore fifteen photos in fifteen minutes, then I might be able to get 60 photos restored in an hour. And if I work on it for an hour each night, five nights a week, then each week I can get that percentage up by 1% or more, and in 54 short weeks I'll be done. That is, if the base "if" assumption holds...

(Where does the "15 photos" come from? I happened to be working on some Baycon 2004 photos, from one subdirectory. By the time I got done, I had reinstated fourteen photos to part 23 and part 24. I was working on those in the first place because right now I'm trying to give priority to the 2004 photos; that year's photos were the hardest hit in the crash/recovery.)

hazelchaz: (Default)

Naruto nightNaruto night (06-Jun-2007) photos are now online.

So there was a screening of the (now 5 years old) movie, Naruto the Movie: Ninja Clash in the Land of Snow. [livejournal.com profile] library_lynn and I decided to see the showing at The Block in Orange. They were a lively, enthusiastic audience. )

Many of us stayed for the featurette afterwards. They ended with a bunch of the cast and crew --including on the Japanese side -- thanking all the fans who watch the show "and sometimes dress like us." I hope that makes onto the DVD version!

We saw one or two fans dressed up in costumes. The photo shown here includes "Cigam" who is the lady now in charge of the Orange County Anime Meet-up, usually on the 2nd or 3rd Saturday of the month. (Picked not to conflict with the L.A. group, but ironically it means it's on the same day as either CFO or Cinema Anime's screenings at the LASFS clubhouse.)

I missed the meet-up this weekend -- did some chores at my folks' house, worked on Hazel's Picture Gallery, and went to the Hugger House potluck. But I'm hoping I can go next month, on the 21st.

This afternoon, Sunday (Father's Day), Lynn and I are going to a "Cosplay Ice Skating" event here in Anaheim. Mostly to support the event and take pictures. Next weekend is Stepfather's Day. Then the following weekend (yikes, it's coming up soon) we head out to Long Beach for Anime Expo.

I mentioned I've been working on Hazel's Picture Gallery. Specifically, I've been working on the restoration project. Matthew Randolph has been very helpful in finding thumbnails and whole images, and I've been matching up what he's recovered to files from the old disk. (There are thousands of JPEG images, so it takes time to go through all of them.) I am pleased to announce that we've been able to bump up the "Percentage Restored" figures. The thumbnail images are now 77% restored, and the large-size picture collection as a whole is 44% restored.

The damage, and the restoration, are not spread evenly over the photo collection. The section hardest hit consists of the photos from 2004. Those were too "new" to have been picked up by most of the archiving mechanisms on the web, but too "old" to still be sitting on my various computers. So I'm really pleased that a lot of the work that has been accomplished lately have been from 2004. (For example, I went to three weddings in 2004. Incidentally, if you invite me to your wedding, I'll probably go around taking pictures of everyone attending -- it gives me something to do, a purpose.) The thumbnails for 2004 are 49% restored so far.

As an example of how slow these things go, the jump from 43% to 44% represents about a week of hard work on my part, mostly hours late at night... I try to keep working on it -- and not let it go without progress for weeks or months at a time -- and the restoration percentage is one of the sub-goals I set myself. "I'll stay up until 2am again tonight, so maybe I can get it up another notch" is basically what's going through my mind if you happen to see I'm online late at night.

(I light up my AIM when I'm online -- my AIM handle is the same as my LJ handle -- and shut it down again when I'm done with my computer. So it's easy to see if I'm here at the computer, even if the AIM system times me out for inactivity and says I'm "away.")

I made up ''I supported Hazel's Picture Gallery'' graphics. Here are, in no particular order, the faces of the first six people who contributed money to the restoration project... )

LiveJournal trivia question: Which five of these six have LJ handles?

Want to see more of them? (Warning: lots of images, takes some time to load.)

Incidentally, I'm not mentioning this because I'm passing the hat again. Donations are always a surprise when they come in. But throwing more money at the task will not make it go any faster -- it's almost all in my court, and it's mostly a matter of finding the free time. The donations that came in during 2005-2006 were enough to pay for the program used on the dead drive to recover the files and to pay for two years of professional hosting service. Donations received since then are going into the general fund, mainly for continued hosting charges and to pay to replace my digital camera when I lose it in a hospital room.

hazelchaz: (Default)

Time for a Hazel's Picture Gallery Recovery update... We have 73% of the thumbnails restored, and 41% of the large viewing-size images.

I made a big sweep through the Wayback Machine, and Matthew Randolph has been scouring archive.org as well and other places with archived images and thumbnails. I've been uploading images from the data recovery discs from the dead disk drive and seeing if they're ready to use, or if the "$C000" fix makes them snap back into shape. So a lot of progress has been made.

If you've ever grabbed images (from 2005 or earlier) from my website, even if they're just the thumbnails or the medium-size images, I could really use getting copies of those back if they're not already back in place. And I could use a little help with some of the work from people with some specialized knowledge, please (see below). If you know the people who attend the Grilled Cheese Invitationals, or if you know how to edit raw JPEG data, here's your chance to help out a little.

A number of JPEG images appear to be corrupted, but might snap back into place if you can figure out what happened to the raw data. (On some of them, there was garbage inserted exactly $C000 bytes from the beginning of the file, and deleting that 16K byte chunk cured them. I tried that on these, and that didn't work them.) You can find these troublesome corrupted JPEG files at:
http://www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-need-help-fixing-corrupted-files/ (6 subdirectories), and a few more at

So that's what a technically-inclined person might be able to fiddle with. Now, for people who know the Gigsville/Theorylabs crowd:

These 18 images are all from the 2005 Grilled Cheese Invitational. If you can tell me who's in the photos, I've got a good chance of matching them up... )
hazelchaz: (Default)
I'm putting out another call for people who want to try to recover JPEG files by judicious hex editing. Snipping bytes 0xC000-0xCFFF from them didn't work on these, but I'm hoping it'll be almost as simple a process. Who wants to help?

First directory: www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-bad/jpg-08-00/.
hazelchaz: (Default)
I've revamped the "first page index" feature again. My example from a few days ago was the 'S' page and the index for people named Sam. Well, if you click on Sam this time, you'll see pictures of all of the Sams. (Almost all of them, actually; I haven't recovered any of my pictures of Sam Bird or Sam Stotts yet.)

My primary photo for a particular person is the earliest shortest caption I have; this means that a caption with just one person's name is favored over a caption with them and someone else. Some of the "Sam" pictures have more than one person in them: that's because I've never taken a picture of those particular Sams by themselves. An example would be Sam Myers; I've only met her once so far, and she was with her famous cookie-baking mom.

For common first names such as David, I don't show pictures of every person I have with that name -- it limits the output to 24 thumbnails, which will take enough time to load as it is. But go ahead and try it -- pick someone you think I might have a picture of, go to the cross-reference page, pick the first initial and then go to the bottom of the page and look for that first name. Let me know what you think, okay?

[EDIT: I've removed the 24-max limitation; it doesn't really make sense on the first-name index pages.]
hazelchaz: (Default)
Cosplay Picnic at Laguna Niguel (08-Apr-2006), Party at Janis and Glen's House (08-Apr-2006), Rema SuzukiAniZona 2 (14/15/16-Apr-2006), Lux Barbecue and Orientation (23-Apr-2006), and Brandon's 21st Birthday Party (07-May-2006) photos are now online. Click on the picture of [livejournal.com profile] rema_suzuki (leader of the [livejournal.com profile] rum_group out in Riverside) to see the [livejournal.com profile] anizona pictures.

Okay, great progress; everything's stable enough for me to resume adding pictures. Slightly new feature -- I changed the Cross-Reference index page photo selection. Instead of picking the first person with that initial to show you, it picks the one I have the most pictures of. So it should not be any surprise that for B you'll see my lovely wife [livejournal.com profile] library_lynn. Nor is it a surprise that R's picture is the daughter of my heart. (Who graduates from UC Irvine in about an hour and a half -- I better disconnect and get over to the campus.)
hazelchaz: (Default)
Whew. I rebuilt the "add new name" program, and started in on my e-mail backlog. As soon as I got to some Arisia photo captions, I started checking the pages from Arisia to make sure all of the names that we already had in the captions were in the system... nope... So the add-name gizmo has been getting a good workout. I ran into [livejournal.com profile] bleemoo's picture, so I have to make sure that the "add LJ link" system works again; I got him linked up, but there are others I met in Boston with LJ handles. I guess that's next!
hazelchaz: (Default)
Maria, Harlie, Lynn Okay, I've got the cross-reference back online. You can click on the picture of Maria, Harlie the Dog, and [livejournal.com profile] library_lynn to see an example of a recently cross-referenced page. Nothing groundbreaking -- there are links to the names of the people cross-referenced, and everything's starting to look back to normal.

So I can get started on typing in the captions, because they'll get processed fully now..

[Update at midnight: I started typing in captions received, and came across a new name that needs to be added to the cross-reference engine. So that "add name" feature has to be rebuilt next for me to continue.]
hazelchaz: (Default)
Y is for Yalow Okay, the cross-reference index pages A-Z have been regenerated. A couple of new features. If you click on the picture we have for Y, you'll see that every page for a letter has a sample photo with a caption that has a cross-referenced name that begins with that letter. (Generally the oldest one we have on file for the first person with that initial.)

Also, if you take a look under A you'll see that I've combined "Sally Abba" with "Sallie Abba." The see-also links for Greg Abba have been cleaned up to not refer to Sally anymore either. And, since there are a lot of names listed under A, we break the list into several 3-column sections; and in the top right corner are quick local links: if you're looking for Anderbear, you could find his section by scrolling down to "Bob Ander -- Aznaphrodite" or you could just click on "An."

Haven't got the actual cross-referencing engine rebuilt yet; also, the first-name index is temporarily unavailable. But it's coming along nicely.
hazelchaz: (Default)
Photos of Maria and Friends Part 21 (Jan/Feb-2006), SelinaLASFS and Aftermeetings part 251 (Jan/Mar-2006), and St. Jude Hospital (23-Jan-2006) are now online. Click on the picture of [livejournal.com profile] selinawoman if you'd like to see the LASFS pictures, including Grace Molloy's first trip to [livejournal.com profile] lasfs. (Baby's first club meeting.)

The cross-reference name index pages are all regenerated in the new style, except for the index indices. That is, you can find Naomi Fisher under "F," but the "F" page itself hasn't been regenerated, and the new cross-referencing engine isn't online. But there are a couple of new wrinkles in the name index pages themselves, you should take a look if you haven't already.

(Selina, when you see this, write me -- I'd like to assign you a different task if you still want to help with the website.)

Oh, and some time ago we passed the 10,000-page mark. We have over 17,000 photos complete on the website, and if you count thumbnails-only we have 20,000 photos visible; and over 3700 picture pages displaying them plus over 6400 different names cross-referenced, adding up to 10,232 individual HTML sub-pages. As of those three above going online. And all 10,232 of those files have been recreated recently from the database, with the new look. [Most of them have been regenerated a couple of times, in fact; I just thought of another variation for the picture pages, for large batches, such as our UK trip for Interaction last year.]

In other news: does anyone local have shorty ethernet patch cables? I need one. 12" is plenty.
hazelchaz: (Default)
Heather AlexanderI've got the new cross-reference page engine half done: it's generating index pages, and it takes into account "lost" photos so you won't see all those skulls on the index pages anymore. To see an example, click on Heather Alexander at right, where you'll find that my oldest photo of her (2001) is still lost; or take a look at the pages for Sally Abba or Sallie Abba. (Yes, I realize they're the same person: which is the correct spelling? When I get a chance I'll combine them. If nobody has better information, I'll use "Sally" because that's what's on the older pages.)

When you're looking at an index page, you're probably trying to find all the pictures of that person. Therefore I've tried to maximize how many pictures you get, up to a reasonable limit per page. I used to make it do one of three things: (1) show you every single photo, (2) show you one photo per page, or (3) show you one photo from one page per year. But this meant that the people I shot the most would have the fewest number of photos visible! So now for option (3) it shows you one photo each from an assortment of pages throughout the history of the website.

So far I've just regenerated the A's; I'll get to the rest of the alphabet in the next 24 hours. Then all I have to do is re-write the cross-referencer to use the new MySQL database too.
hazelchaz: (Default)
Chocolate BearsPhotos of some Chocolate Bears (20-Feb-2006) are now online.

Inoue-san gave me this lovely box of chocolates - a tin, actually, containing chocolates with a teddy bear theme.

Update on the website, inside and outside: I'm experimenting with a new look to Hazel's Picture Gallery; click on the tin of bears to see a sample. And on the inside, I'm almost done putting the tools back together...
hazelchaz: (Default)
Sandra gets sung toSandra gets sung toWell, I was going to upload my ConDor photos. But my database handling tools are still in rewrite... specifically, getallpix which rounds up all of the photos in a temporary directory isn't working. Arrgh!

But on the other hand, I've started work on the cross-reference index engine. Soon, I hope to have that up and working, where a cross-reference index page will get updated as lost pictures are recovered. For example, the cross-reference page for [livejournal.com profile] library_lynn still shows a bunch of "Not Found" skulls; you can click on the first link (for 1997) and see the 1997 photos, even though it doesn't show them on the index page. That's what I'm working on, and will be up soon; first, though, I'll fix getallpix and get those ConDor photos online...
hazelchaz: (Default)
More Portland Westercon 2001 pictures. Anyone recognize these people? Or can tell me which days they were taken?

two people in halllady in dresslady in hall

In other news: I've found some of my notebooks full of printouts taken from Hazel's Picture Gallery, usually from when I'd just uploaded photos and I was trying to match up captions to them. So that will help with some of the recovery. I don't think I've found all of the notebooks, either, because hardly any of the 2001 photos that I've been working on recently are shown in the ones I've found... Oh, and our recovery percentage currently stands at 39%. Since I started tracking these figures back in March, I've fully recovered over two thousand photos; about half of them (997) from 2001.
hazelchaz: (Default)
Okay, how about these? Who are these people, seen at Dee's wake in Kansas City in 2004? )
hazelchaz: (Default)
Someone and someone else Here's one from Kansas City -- Dee's funeral in 2004. I think this is a picture of Rick Lancaster and Lance Mallis, can anyone local confirm?

I've been going through the files recovered from the disk drive, plus the photos (thumbnails mostly) that I was able to get from the Wayback Machine. I've been correcting corrupted photos (when the corruption consists of garbage inserted at offsets 0xC000-0xCFFF, anyway), and trying to match up thumbnails -- and making educated guesses where I don't have the original thumbnail image at all. It's not that hard, when I can recognize the faces, or when I've written down their t-shirt slogan in the caption. But sometimes I need a little help.

Somebody How about this one on the left: is this Judith Moore?

Somebody How about this one on the right: is this Ashley Moore? It looks like there's a family resemblance...
hazelchaz: (Default)
Portland Westercon shot Anyone remember the Westercon in Portland back in 2001? I'm in the middle of restoring a whole bunch of photos from 2001, and most of them are fairly easy to place. But I can't remember if this one is from the Con Office or the Volunteers desk. Is there anyone out there who can place it, please? Or tell me who's in the picture? (Click on the picture to get a better look.)
hazelchaz: (Default)
The Hazel's Picture Gallery recovery story so far.

It occurs to me, especially after meeting Ashley, Chloe, Kate, Katherine, and Erin this afternoon, that not everyone who follows my LJ knows what this server recovery thing is all about. Allow me to explain. (The regulars, you know all this and can skip this post if you like.)

I bought my first digital camera several years ago, and started taking it with me to conventions. In fact I use my camera at most of the significant events in my life, especially activities that involve people or things I don't see every day. I upload my pictures to my website; I have a website for myself and [livejournal.com profile] library_lynn at www.boston-baden.com. (Bostonbaden.com, without the dash, is the domain I use when I give people my e-mail address. E-mail works either way, for my primary e-mail addresses -- hazel or chaz, and hazelchaz works too -- and you can get to the website with or without the dash too, it'll redirect you if it needs to.)

The pictures I upload are cut down into two sizes in addition to the high-resolution images; the high-res images aren't always put online, but a viewing-size (approx 640x480) and a thumbnail image (about 100 pixels high and wide) go with them. The web pages themselves are plain ordinary HTML pages; originally generated by a program and hand-edited, then later generated by a program using a crude DB_File database, and now generated by programs that use a MySQL database. (Dreamhost makes it easy to do MySQL databases. I love 'em.) I haven't finished rewriting the programs, though; so part of the unfolding story here has to do with my software work for the site.

Last summer, after several years of taking and uploading photos, I'd passed the 25,000 mark -- and then we had the great server crash on the machine in Minneapolis that had been hosting my domain. Yikes! He thought I had backups for everything; I had thought (for quite some time, actually) that a new mondo backup unit was just a short while away from coming online, and hadn't disk space to save my photos. (Ironically, it didn't occur to me to copy them onto a machine with cd-rom burning capability, even though I'd been burning cd-roms full of MP3s from all of my CDs for the home music project.)

So I signed up for a paid hosting account on Dreamhost and started rebuilding from what backups were available. By winter, the dead easy work had already been done: the stuff that was easily rescued off of the old computer, the files that I had copies of in one form or another (mostly from 2000-2002, plus a scattering of recent photos that hadn't been deleted off of my primary machine), and so forth.

The disk drive from the old server has been subjected to a professional-grade recovery process, by somebody that works in the data-recovery field and knows his stuff. (Thanks, [livejournal.com profile] dspisak!) Many people donated money against the possibility of our having to pay a thousand dollars to a data recovery company; the first one to explicitly say she'd put in $20 was Naomi Fisher, and when I put the tip jar on the website the money came rushing in. Some of that money went to the software Dan needed, and the rest is being applied against the hosting fees at Dreamhost. We have two years prepaid at this point, and I really appreciate the donations. (I need to start writing thank-you notes, in fact. But that's another story.)

The data recovered from the old server, however, is generally in the form on unidentified, unmarked, uncaptioned, unnamed and undated JPEG files. And a great deal of them seem to be corrupted to boot. At first I was worried that we were seeing corrupted data from the drive; now I believe that the recovery program may have not completely understood the filesystem on the server, and the data are all there, they just need some work in a hex editor.

A great deal of work has been done on that front, notably by [livejournal.com profile] testerscot, [livejournal.com profile] bovil, [livejournal.com profile] johno, and lately [livejournal.com profile] gvdub. (My apologies if I've left out any of the hex editor hacking crew, I'm typing this from memory.) So the result of that is hundreds and thousands of perfectly good, ready-to-reinstate photos, we just have to figure out where they go on the server.

Meanwhile. Several (at least half a dozen) people have brightly suggested that The Wayback Machine Internet Archive might magically solve all my problems. Well, sort of, a little, not as much as you might expect. First of all, while the Wayback Machine has been saving copies of every web page it can lay its spider paws on, that doesn't always include all of the images on the pages. Or the images that are linked from the pages. And, they don't get every page, although they do index and save a lot.

I knew using the Wayback Machine would basically get a number of my thumbnail images, and a smaller number of the larger images. I'd held off investigating what I could get back, because I knew I would often just get a thumbnail and not the matching large image -- and if I just restored the thumbnail to a page, you wouldn't be able to tell that the full size picture was still missing. This lack, however, is something I can deal with on the new MySQL platform. And, even just a thumbnail helps, because the other part of the recovery project is yielding up perfectly good large images. So now the task of dovetailing the results from the two approaches begins.

I think that's the gist of it. I go to about 12 conventions a year, science fiction and animé now that I've started our little [livejournal.com profile] animelosangeles convention; i take a lot of photos by sf fan standards, but I'm strictly small time compared to the people snapping photos at animé conventions. I am now about two or three conventions behind, plus a few parties and picnics, in my picture-uploading -- this is because I have other projects apart from the website software I mentioned, and the data recovery, there's a big convention (or a medium-sized one, if you come from an animé convention viewpoint) that I'm head webmaster for, and... well. You're up to date. Stay tuned here for further updates.

Any questions?
hazelchaz: (Default)
Okay, I'm ready to ask for a little help. [livejournal.com profile] gvdub, you want to help? You know many of the faces involved...

There are a bunch of directories under http://www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-ok/ which have [livejournal.com profile] lasfs in them. For example, http://www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-ok/jpg-08-11-lasfs/. Most of the photos were taken in and around LASFS in 2001.

Frank WallerFrank WallerI'd like help looking at the pictures in the "Recovery" area, and matching them up to what thumbnails I've been able to recover from the Wayback Machine so far. You can get to all of the LASFS in 2001 photos in one place -- for example, Part 102 (29-Mar-2001) clearly shows Frank Waller coming out of the back building in picture 0449-08 (at left), and that's the same photo we've recovered in http://www.boston-baden.com/hazel/Recovery/jpgfiles/export-08-ok/jpg-08-11-lasfs/, 8th photo from the beginning, lost-1118-fix.jpg, shown at right. (Note: The thumbnails won't always be exactly the same dimensions. But they'll be the same proportions, i.e. aspect ratio.)

So for this match-up, you'd tell me that lost-1118-fix.jpg from part 11 = picture 0449-08.

Anyhow. If anyone who'd like to help has the time to go through some of the collections of LASFS photos -- "LASFS part (n)" photos and the recovered ones; if you see page that's got corrupted images on it, don't worry about that, move onto another page. Send me a post telling me which "Recovery" directory you're looking at. Or volunteer here and I'll assign you one....

Thanks to everyone who's volunteered, I appreciate all the help!

P.S. recovery percentage is up to 37%. Slowly, slowly making headway...



hazelchaz: (Default)
Chaz Boston Baden

June 2017

     1 2 3
4 5678 9 10
11 12 1314 151617
18192021 222324


RSS Atom

Most Popular Tags

Page Summary

Style Credit

Expand Cut Tags

No cut tags
Page generated Sep. 25th, 2017 11:26 am
Powered by Dreamwidth Studios