« Child Trader | Main | Global Warming? »

May 17, 2008

File Numbering with the Canon EOS 40D

When you swap memory cards on the Canon EOS 40D (and the Canon EOS 20D also), it confuses the camera and the camera starts numbering the photos incorrectly. This has bugged me for some time, but now it's bothered me enough that I'm going to do something about it.

I first noticed this when I sent my Canon EOS 20D to Canon to be repaired. I set up my cameras number the files "Continuously", meaning - don't start renumbering from IMG_0001 every time I take the photos off of the Compact Flash memory card. The Canon EOS cameras generally do a fairly good job of this.

For my old Canon EOS 20D, it automagically created a new folder each time I shot 100 photos, so when I copied them off the CF card, they were numbered continuously, but parsed into folders with 100 photos per folder. For instance, if I shot 300 photos, they would be stored on the CF card like this:


\DCIM\424CANON\IMG\IMG_2401.JPG - IMG_2500.JPG
\DCIM\425CANON\IMG\IMG_2501.JPG - IMG_2600.JPG
\DCIM\426CANON\IMG\IMG_2601.JPG - IMG_2700.JPG

The Canon EOS 40D doesn't do this, and it took me a while to figure it out. Instead, when I set it up to continuously number the files, Canon now just throws them all into one single folder, no matter how many you shoot.
\DCIM\100CANON\IMG_2401.JPG - IMG_2700.JPG

So, now I just break them up into 100 photos per folder, to replicate what the EOS 20D used to do. (If you have 10,000 images in a single folder, it takes forever to browse the folder, refresh thumbnails, etc. So, the way the 40D works sucks, but this is how it works, as best as I can tell. If anyone knows differently, please advise.

In any event, this is all fine and dandy until you change CF cards. And when you change CF cards, it confuses the h3ll out of the camera. Why? I'm not clear. I called Canon up and asked them about this when they repaired my EOS 20D. When I got it back, the file numbering was advanced by about 10,000 frames, so I called them up and yelled at them for wearing out my shutter. But they explained to me that it picked up the file number from the CF card they used when they tested the camera. I wasn't sure if they were lying or not, but I couldn't prove otherwise, so I let it go.

It happened to me again when I sent the 20D to be repaired the 2nd time.

Then, it happened to me again yesterday when I swapped my two CF cards between the 20D and 40D cameras, as I was shooting both cameras at the Field Day yesterday.

This website erroneously states that "File numbering is maintained as you swap or erase memory cards." Uh, you're dead wrong on that assertion. Thanks for playing. Next.

So, renumbering the files really sucks, because it totally screws up my little system for keeping track of when the photos were shot. And now, it's pissed me off enough that I'm going to figure out what the deal is.

What I need to figure out is this - How does the Canon EOS 40D/20D know what to name the next file it shoots? I suspect that the answer to this question is not very simple. You would think that the computer had some sort of internal counter and bumped it up by one every time it took a photo. However, I suspect that this is not the case. If this were the case, then it wouldn't be renumbering the files on me when I switch CF cards. So, I can see that, in some cases, the camera does, for some reason, get file numbering information from the CF card. However, in other cases, it appears to not get file numbering information from the CF card. For instance, I can format the CF card with the camer, or in Windows, and it doesn't forget what the next file number will be. So, this indicates to me that it does store the file number internally as well.

I will try some experiments to see how the renumbering really works. First of all, I'll reformat the card with the camera and see if the file numbering resets or not (currently, the last image shot was IMG_7784.JPG in folder 100CANON).

After reformatting the CF card in the camera, it shot image IMG_7785.JPG in folder 100CANON. So, reformatting the card in the camera did not cause it to forget what the last file shot was. Now, I will repeat the experiment by reformatting the card in Windows. Hmmm. Before I do this, I'm trying to figure out what type of format to use (FAT16, FAT32, NTFS, etc.). This site says the 40D supports both FAT16 or FAT32.

So, I went into Windows XP Pro and right clicked on My Computer - Manage - Storage - Disk Management and it showed the following:


EOS_DIGITAL (E:)
1.91 GB FAT
Healthy (Active)

Hmmm. So, I have it formatted as FAT16, apparently. Here's a comparison of FAT16 vs. FAT32. I'm not really bright enough to figure this out tonight. So, I'll reformat it as FAT16 and see what happens.


So, I took out both camera batteries and removed the lens. Then, I reformatted the CF as FAT16 again (I'm not checking the "quick format" box, but it only takes about 10 seconds - how much quicker could it get?

Nope. Still no dice. I reassembled the camera, and it had forgotten what the date and time was, but it still knew what the next file number would be and it shot "100-7791". Argh.

Now, I'm not terribly excited about that fact that it shows up as 1.91 GB in XP. My thought is that it should show 3.9 GB, not 1.9 GB, being as how it's a 4 Gig memory card. When I put it in the camera, it shows me that I can take 438 photos. At roughly 4 MB per photo, the math says this will be 1.7 GB. So, now I'm wondering if I've had the card formatted incorrectly for the last 3 years. Hmmm.

Yesterday, at Field Day, I shot until the camera wouldn't shoot any more and said "CF CARD FULL" or something like that. So, let's see how much disk space I actually used. I shot images 3700-4168 yesterday. I shot these in high-res JPG format, and Windows XP shows the total file size to be....1.97 Gig, which means, in all probability, that I've been shooting using half of the space on my CF card since November of 2005. Fark me that makes me mad.

Oh yeah. This post makes it crystal clear that I've been using 1/2 of my 4 Gig card for 2 1/2 years and was too dumb to realize it.

"The original poster's camera does support a 32 Bit FAT, and he's using a 4GB card, which will require formatting with a 32 Bit FAT in order to use the entire card, unless you wanted to create a 2GB Partition formatted using a 16 Bit FAT (called FAT when formatting using Windows), leaving the rest of the card unused."

Although, now I wonder - if there truly is a 2 Gig file limit on FAT16, if I allow the camera to format my CF card, which I do fairly frequently, why in the h3ll doesn't it format it as FAT32? This web page claims that the EOS 20D "Automatically writes FAT16/FAT32 depending on capacity". Well, I'd beg to differ because I shot the 20D for years and I can promise you I never got more than 500 images on the CF card, but now my math tells me I should be able to get nearly 1,000 images on the CF card.

And the EOS 40D obviously isn't formatting it correctly either. So, possibly my problem is with the CF card partition. There's some advice on partitioning and formatting CF cards here.

"You'll find the disk admin tools under Control Center, Administrative Tools, Computer Management, Storage, Disk Management.

Under the partitions list at the bottom (it won't be in the table at the top) , you'll find the card reader somewhere. Right click on the space for the card and delete any existing partitions. Then, right click on it and tell it to create a new one. Make sure you're looking at your card reader versus your Windows partition. ;-)

Then, use the New Partition Wizard that comes up to create a new paritition using all the space on the drive and format as FAT32 with a 32KB cluster size (which seems to be more reliable with some cameras using larger media).

If for some reason you can't delete the partitions on it, there is another tool that I sometimes use for this purpose (Ranish Paritition Manager) to delete parititions and recreate the Master Boot Record on a drive. If Microsoft's Disk Admin doesn't work, I can post instructions on using Ranish (you'll need to download an ISO Image that includes it, burn a CD and boot to it to run it."

For some reason, when I right-clicked on the space for the card, "Delete Partition" was greyed out. Not sure why this is.

So, now I'm going to download the Ultimate Boot CD as an ISO file and burn the image onto a CD so that I can delete the partition on the CF card with some little trick utility called Ranish. (No, I'm not making this up. I'm sunburned from Field Day yesterday and from watching some Indian dance down at the Fort today and I'm tired and dusty and I'm switching from beer to coffee because I get the feeling we're in for a long night.)

Well, I created an "Ultimate Boot CD", and I managed to reboot the PC and hit F2 to get into the boot menu and tell it to boot from the CD, which it did, but then I couldn't figure it out. Probably, I'm not bright enough for this. Maybe I should take up carpentry or try selling an organ or something.

OK. So, I decided to go to SanDisk and see if there were any tips on how to delete the partition on this little bastardized piece of technology. I searched the SanDisk website, but they're about as big as Sony these days. Seems like they're making everything from USB powered toasters to Bullet Proof Baby Strollers.

So I decided to look at my CF card to see what product I actually had purchased. The front says it's a SanDisk 4 GB CompactFlash card. Then, on the back, I found a URL for a website for support. Score! The back of the card says:

For Support: www.sandisk.com/cf4gb

But, you go there and get a message that says:


"We’re sorry, but there is no Sandisk.com Web page that matches your entry. It is possible you typed the address incorrectly, or the page may no longer exist. You may wish to try another entry or choose from the links below, which we hope will help you find what you’re looking for."

Argh! Like, I can promise you that I bought this CF card in 2 1/2 years ago, so you'd think that someone at that farked up company would have the presence of mind to support the URL's if they were printed on their farking product less than three years ago. But no. Instead, I see this long list of products made by SanDisk, from Pachinko machines to ball bearings. Finally, I found my old-school 4GB SanDisk SD card, right between the boxes of SanDisk Waffle Mix to a set of SanDisk Metric Crescent Wrenches. Now, the website told me zero about the product. Zero. But then...then...as I stared at this freaking little enigma in my hand, I noticed something. Something that I'd not noticed before. There's a switch on the side of the card that has three positions:
1. - 4GB
2. - 2GB
3. - 2GB

So, apparently, it can be switched between two different 2 gig banks on the card, or it can be switched to option 1. for a total of 4 gigs. I switched it to position 1, and lo-and-behold, it now shows up in XP Pro as FAT32 and 3.81 GB FAT32. Now, how they accomplished this little miracle, I have no clue. But I just doubled the number of photos I can take. Better late than never, I suppose.

Just to see what was on the other 2 gig partition, I switched to position 3, but it showed up as FAT16, with no files on it. So, to the best of my knowlege, I've never used this switch. It's so small, I had to use the point of a 5mm mechanical pencil lead to switch it. Now, I've switched it back to 4GB (position 1).

So, I shoved it into the EOS 40D and now it says:


"CF card not formatted
Format card with this camera"

I put in the memory card and took a photo and it shot image "100-7790". So, it's still remembering the file number somehow. I'll try reformatting the card and power cycling the camera. It now shows I can take 878 photos. Great. I doubt I could feel much less intelligent at this point.

I snapped a shot and it took photo "100-7792", so I still haven't been able to trick it into forgetting what the next photo number is. I'm going to try another tack. I'm going to rename the last file in the camera to be IMG_8792 instead of IMG_7792. And then take a snapshot. The theory being that somehow it will take the greater of the two file numbers - the one magically stored in the camera, and the one stored on the CF card. I'm grabbing at straws, but what the h3ll? I'm already proven I'm a moron. I have no face to save at this point. I feel like Tonya Harding on a bender.

That did it. I tricked the camera by bumping up the image number on the CF card from IMG_7792.JPG to IMG_8792.JPG. Then, when I took a photo, it snapped "100-8793" (\100CANON\IMG_8793.JPG). This is something I had suspected for years, but just found the time to prove it conclusively one way or the other.

OK. Next test. I'm going to rename the filename to IMG_1000.JPG, and see if I can get it to go back to a lower number. (Every time I plug the CF card into my Dazzle 943-in-1 Memory Card Reader, that virus known as Adobe Photo Downloader rears its ugly head and wants to show me the files in some proprietary window and I thought I'd already slayed this Photo Downloader beast, but apparently I was mistaken.

OK. I tried to reset it to IMG_1001.JPG, but this didn't work. Insead, I just shot image "100-8794". So, it is behaving exactly like I have suspected it behaved for several years. I don't know why it does this. I don't have a clear picture just yet. But, if you swapping memory cards, and if the new memory card has higher image numbers than the image numbers you had been using, then the EOS 20D and 40D will both start numbering using the new (higher) image numbers.

My speculation is that the camera stores the "Last Image Number Used" inside the camera somewhere, and I don't mean on the CF card. Although, where the camera is storing this data is a deep mystery to me as removing both batteries and the lens causes it to forget the date, but not lose the last image number assigned.

I'm going to test this theory in a couple of ways, because it just seems so unlikely - so improbable - so counterintuitive.

Test 1: I will remove both batteries and the lens and then erase all data from my other (1 Gig) memory card using Active@ Kill Disk Hard Drive Eraser. Then, I will reassemble the camera and shoot a frame. If it still follows the old file numbering pattern of "100-8795", then I'll know for abso-posi-lutely that, somewhere, somehow, that EOS is, in fact, retaining inside the camera frame somwhere, somehow, the last image shot. How and why, I have no clue. But this will prove that the filenumber is, in fact, being stored, IMHO.

So, I ran the "Kill" process on my CF card, and it made a single pass, writing binary zeroes over all of the data on the card. Apparently, they also dropped the partition, so I recreated the partition using the Partition Wizard. It now shows the drive as "Healthy", but not active. I reassembled the camera and put the CF card in it and snapped a photo and it took photo "100-8795", so I've proven conclusively that the last image taken is stored somewhere inside the camera.

Now that I've proven this to my own satisfaction, I'm not thoroughly surprised by it, and here's why. The camera obviously has some sort of long-term memory, because it stores all sorts of things such as Custom Functions, and a myriad of other user-controlled camera settings. The only thing that really gets lost when both batteries are removed is the date time stamp. This is because, even if they somehow retained the date-time in the camera's other memory banks, it wouldn't be accurate, as they have no way of knowing how much time has passed since the batteries were removed.

OK. Now that we've proven conclusively how the camera determines what the next image number will be, I have to say I'm baffled. I have no idea why they did it this way. It sucks. It sucks big-time. It basically means that, in a nutshell, you can't swap CF cards willy-nilly like you'd like to be able to. Because, if the new CF card has images stored on it and if the file number on the new CF card is higher than the "next image filename" secretly stored in the Canon camera's long-term static memory banks, then the Canon will leap forward to the last image stored on the CF card plus 1.

How to prevent this from happening? If you have to swap CF cards, make sure there are no images on the card before you put it in the camera. Alternately, you could reformat the CF card in the camera before shooting any images.

There are a few solutions that will prevent the problem from occurring, but none of them are ideal. For instance, I was shooting yesterday at Field Day with two frames - the 20D for long shots and the 40D for close-up shots. When the 40D CF card filled up, what I wanted to do was take the half-full CF card out of the 20D and shove it in the 40D and have the 40D continue to shoot without adopting the 20D filenumbering format. Unfortunately, this is not possible if the 20D images are numbered higher than the 40D. If the 20D image numbers were lower, it would not be a problem at all, but that was not the case and it screwed me up in a big way.

The path out: Now that I thoroughly and completely understand the problem, the solution is fairly straight-forward:

1) Renumber all of the files I shot yesterday with some utility like IRFanView.
2) Reset the camera manually to restart numbering from IMG_0001.JPG. (Menu - File Numbering - Manual Reset).
2) Copy a picture onto the CF card with the PC and rename it to one less than whatever frame I want to restart the camera on. (I copied on IMG_4237.JPG. I was a zero byte dummy file to trick the camera.)
3) Shoot a frame and the camera will pick up the image name from the CF card and we'll be as good as gold.

OK. It worked. It's 2:41 a.m. I'm going to bed. Lord God, Canon sucks. Can someone please post if they find this useful? I feel like I'm the only one on Earth with this problem.

Posted by Rob Kiser on May 17, 2008 at 9:20 PM

Comments

There are several utilities * that rename the pictures files in the ISO 8601 date/time format: YYYY-MM-DD-HH-MM-SS (year, month, date, hour, minute, second). This ensures that (1) each file name is unique, and (2) easily sortable in chronological order.

Eg: A picture taken on May 18, 2008 at 12:51 am and 30 seconds would be called 2008-05-18-00-51-30.jpg.

Why cameras don't use that name format to save picture files, rather than something like IMG_001.jpg or PIC001.jpg, is beyond me.


* For example, Irfan View for Windows, and Exif Renamer for Mac. I've been using these programs for about 4 years now. At a glance, I can look into my photo directories and tell when a picture was taken without having to examine the properties of the file. Nor do I have to worry about a picture file overwriting another picture file with the same name if I combine directories.

(There is also Mihov Exif Renamer for Windows, but I have no experience with that program).

Posted by: Robert on May 18, 2008 at 12:54 AM

Well, it's obviously because IMG_0394.JPG is so much more descriptive. I can tell just from the filename that it's a photo of those witches in Peru.

Posted by: Rob Kiser on May 18, 2008 at 3:19 AM

Well, as much as I hate to admit it, now that I think about it, this would pretty much eliminate my problems with swapping CF cards and file naming/numbering conventions. And it isn't like the filename means anything to begin with. Basically, I'd be doing an end-run about Canon's buggy little procedure to keeping track of filename/foldernames.

Posted by: Rob Kiser on May 18, 2008 at 3:28 AM

So I finally tried Mihov EXIF Renamer on my Windows PC. Works as promised.

One quirk, though.

By default, it renames the file YEAR-MONTH-DAY-ORIGINAL FILE NAME, so you get something like

20060830-IMAGE.JPG

If you want it in YYYY-MM-DD-HH-MM-SS format, uncheck the box that says "Original File Name," and check the box es that say "Hour,"Minute," and "Second." Personal preference, but I really don't give a damn about the camera's file name.

It looks like you can also add a prefix or suffix if you want, such as "Peru" or "Field Day" if you want more descriptive file names (eg:"Peru-20071115-130000.jpg" or "20071115-130000-Peru.jpg").

For purposes of chronological sorting, I'd suggest adding a file name descriptor as a suffix instead of a prefix if you're going to add one, but that's just my personal preference.

Posted by: Robert on June 24, 2008 at 9:49 AM

Thanks for the post. I just got home from vacation and realized I had used a card I haven't used since I had my 10D. Lo and behold, all of my file names were screwed up. I seemed to remember having had this problem before when using the 10d exclusively (I think a friend used my cam with his card or something) and the fix involved just putting back in a card that had left off at the number at which I wanted to continue (if that makes sense?). In any event, that did not work. I too found that you could go to a higher number (pre-name?) but the camera ignored a manually-set lower file name number (post-name?).

I started playing with it, and after swapping out my card for the third time, the shutter button stopped working. It would focus, but not shoot, nor would any of the buttons on the back of the 40D work either, and the sensor cleaning wan't happening when I turned it off or on. I tried removing the main battery, lens, card, etc. but no love. I called a friend who spends a lot of time on the Interweb in photo forums to see if he had ever heard of this issue. He said he had not and recommended a call to Canon first thing Monday. Being that my vacation is just over, I wasn't heartbroken, since I got the shots I needed while gone, but I was still bummed and kept poking around. I realized there must be some kind of CMOS-like battery in the cam to remember the dates and stuff while changing batteries, but I figured it was buried and couldn't remember ever having seen it.

The short story - it's right inside the battery compartment(I know you know, you mention taking yours out above) - I found it while talking to him. I took it out and VOILA! - my camera is working again! Ingrate that I am I mentioned the file name thing to my friend who said I should count my blessings and live with it. He just copies his folders off the card and renames them (the folders) and leaves them where they sit. I figured it was worth a quick browse and here you are. I reset the count in the camera, took one shot, renamed it to the last file name I had in my photo folder, took another one, and I'm back on track!

I'm stoked, my OCD is appeased and I don't have a 1000-shot gap between what I've taken and what the camera thinks I've taken.

I do wish we had the option to break the folders into 100-shot groups or leave it like it is (I think the default size is 10,000 shots, then it creates a new folder).

Thanks for taking the time to share.

Jason

Posted by: Jason on August 3, 2008 at 3:20 PM

I found a way to restart the directory naming on the CF cards on my 30D.

I made the same mistake of putting the CF card from my 10D into my brand new 30D, which of course changed image file naming to pick up where the 10D had left, including the "DCIM\118CANON" directory.

I wanted it to go back to "DCIM\100CANON\IMG_0001.JPG" as it would have, had I put in a new CF card.

I renamed the "DCIM\118CANON" directory to "DCIM\999CANON" in a card reader, and put the CF card in the camera. The 30D complained that there was no room for pictures or something similar. I then manually reset the file numbering and formatted the CF card in the camera - and presto - back to basics.

I could then rename the latest image taken to IMG_0007.JPG and place in the "DCIM\100CANON" directory and load the CF card into the 30D.

That fixed the problem for me.

Posted by: Mogens Beltoft on October 29, 2008 at 2:22 PM

My Canon Powershot SD450 will separate files into sequentially numbered folders of 100, OR by month taken.
Cool!
I'm disappointed that my 40D doesn't have this capability.
Very.

Posted by: HT on June 14, 2009 at 3:41 PM

This was very interesting. I'm in the process of selling my Canon 5D, and I've been trying to build up a rough estimate of how many frames I shot during the time I have had it. I've simply been counting up all the .CR2 files I shot over the last year or so and tallying them with what I was doing at the time, but there are oddly huge gaps of thousands of frames here and there where the numbering leaps forwards. Given the information above, I must have swapped partially-shot cards (and in a couple of places I can see where I did that).

If only Canon had a simple "shot count" menu option. I have an ancient Kodak DCS 560 that tracks total shutter counts, and that camera was built in 1998.

Posted by: Ashley Pomeroy on January 1, 2010 at 12:19 PM

Post a comment




Remember Me?

(you may use HTML tags for style)


NOTICE: IT WILL TAKE APPROX 1-2 MINS FOR YOUR COMMENT TO POST SUCCESSFULLY. YOU WILL HAVE TO REFRESH YOUR BROWSER. PLEASE DO NOT DOUBLE POST COMMENTS OR I WILL KILL YOU.