November 21, 2002

Mac OS X Hell

I'm a computer consultant, specifically for Mac systems, though I'm familiar with PCs as well. I've been really into OSX this past year or so... it's pretty rock solid, and has not given me any real headaches... until just recently :( What follows is a pretty racy message by me, that I posted to a private list so I could vent after spending the better part of the day trying to get my laptop back in order. I cleaned it up some, esp. my mixing of tenses (hope I straightened all that out - how embarrassing), and added some further thoughts on what we thought went wrong.

Backups saved me from completely losing my mind, but some stuff was still permanently lost. Consider it a cautionary tale of hard drive space and OSX...

Sometime this afternoon I noticed that I couldn't save some small text files that I was working on. I had Mozilla, Mail, BBedit, and maybe a few other programs opened. I was trying to save a new document I had open in bbedit, but it said it couldn't because the drive was full. I checked my drive's stats and it plainly showed 80mb free. I had just installed this new copy of BBedit lite 6.1 so I figured maybe something was wrong with permissions or something. Anyway, to test things out (me, being this supposed techie) I tried to duplicate a file. I selected a file on my desktop, some small rinky-dinky little text clipping, and hit command-D for duplicate. Nothing. No file duplication, no warning or error message, nada. I thought maybe the desktop didn't refresh fast enough (has happened before) so I opened my home folder, and then the desktop folder. Still nothing. Now I thought maybe it's the keyboard shortcut I'm using. I mean, come on, I'm asking it to duplicate a 4k file, how hard can that be? Still no go. So now I'm wondering just what the fuck is going on and I option-dragged a document from one folder to another... forcing it to copy not move. This time I got the "can't do it, drive full" error.

!!

80MB free and the system says it's full??? So, I decided to play it safe and dump a few largish files to an external f/w drive. I freed up about 500mb, thinking that should be more than enough, and it is, because now I could duplicate with no problems.

No more than maybe 10 minutes have passed since this whole thing started, and I still had a bunch of programs open, if you recall. I noticed that Mail was acting kind of funny. No new mail was coming in, so I figured I should restart the machine and get it all refreshed. As I hit restart, I should've taken a good look at my desktop, with it's beautiful Halo 2 Desktop pics, the super-tiny icons that are tightly organized on the right hand side, the dock with all my preferred icons and such. I wasn't going to see those again for a few hours. :(

When the machine was done rebooting, my jaw hit the floor as I'm staring at the default OSX setup. My icons on the desktop are now all piled on top of each other, and my dock is generic! I started to get nervous now, so I launched Mail. What's the first thing I see? (I'm paraphrasing) "Welcome to Mail, you have no accounts setup, please create one" FUCK! I start to imagine the worst, all my recent emails are gone, and mayhaps even worse... thankfully I leave emails on the server for a week. Now I launch Mozilla. Cookies... gone, Bookmarks... gone, all prefs in general.... gone.

Now I had backups of most of this, and actually, after a few hours of somersaults and some backflips, the only thing I've permanently lost are my most recent set of bookmarks, which still sucks as I had collected quite a few dozen important (to me) urls over the last few weeks. I got back all my emails, I re-tweaked my finder prefs, and I even had a fairly recent backup of my cookies, but what really kicks my ass is how the damn OS didn't tell me a damn thing about the drive getting dangerously close to full, even at a setting as ridiculously high as 80mb.

With hindsight, it seems apparent to me that any program that was currently open and that had open files off the 'full' hard drive, tried to save the last state of those preferences at some point, couldn't write them back because the drive was on the fritz, and instead of warning me, just saved empty pref files. My Mail prefs, Mozilla prefs, cookies, bookmarks, finder prefs, etc... all got zeroed out. Why it didn't initially warn me when I was trying to duplicate just a simple little text clipping, I have no frikkin' clue.

Let this be a warning for those of you running OSX. Keep an eye on your OS drive's free space, and backup your library/plist folder alot more often than usual.

---

On further reflection, it was noted that 80mb is the size of an OSX swap disk file. It's possible that the VM engine in OSX got mangled somewhat, and did not want to take that last bit of space and leave the drive at 0% full. The strange thing is that I recall with absolute certainty that I've had the laptop as low as 20mb before and with no problems whatsoever.

What's infuriating is that even if the OS thought the drive was full, WHY didn't other programs or the OS itself warn me that I was close to losing data if I kept filling up the drive?

I checked the drive out with my trusty DiskWarrior, and Disk First Aid, etc., and nothing out of the ordinary was found. Whacky!

Posted by bs at November 21, 2002 11:29 PM
Comments

I had the same problem... and more than once. OSX loves to eat your disk space with vm. My only warning signal is itunes stops playing because it cant write to the library when it changes tracks... as soon as it stops i start frantically freeing up disk space and make sure not to quit any programs untill I have space... losing all your mail boxes and bookmarks sucks big time. Especially when i've done it twice now DOH!

My osx seems to have another problem aswell.. it loves vm so much it never lets go. I have 800mb of vm at the moment, and thats not unusual lately. Some of the vm files havnt been modified in 4 days now, i've run janitor and disk first aid, they just wont go away. Restarting seems to be the only way to get rid of them :(

If you ever think 'hey im sure I had more disk space' just type 'cmd shift g' in the finder and type in '/var/vm/' up comes your vm storage.

Posted by: Helios on November 26, 2002 9:34 PM

Need an explanation on how the SWAP file operates in OSX? a good place to start would be

http://www.peak.org/~luomat/articles/FAQs/Swapfile_Swapdisk/index.html

Also... read up on ADC on just how the OSX SWAP file behaives.

BTW... the swap file's size is dynamic... it can grow and shrink as needed. It resizes in chunks of about 80MB or so, and will only resize when it absolutly must. So if you open 20 apps that increases the swap file's size to 500MB, closing the apps will not shrink the swap file. It will only shrink when disk space becomes low.

Posted by: Dustin on November 29, 2002 12:59 AM

Thanks for the comments guys. I killed about a gig of games from the drive, and now I'm planning on giving myself a fairly early xmas treat and get a 30gb drive for the laptop. I also downloaded a neat utility app called ShadowDisk from StupidFish 23 that shows me my disk usage at all times.

I also made the var/vm folder a favorite, so I can quickly check it out whenever I'm curious.

Posted by: Miguel on November 29, 2002 2:54 PM

Crazy!!! I read this post of yours on Friday, and on Saturday I had exactly this problem. The powerbook (not actually mine) got shafted but I was able to rescue just about everything (thankyou, target disk mode!). The actual incident was a systems nightmare... Ran out of HD space while running Virtual PC, while transferring a 2 gig file to an external firewire drive, and as Virtual PC crashed out, the battery died and the external drive's connection locked up.
Crazy stuff... got some funky screenshots of dozens of Finders showing up at the same time.
Anyway, it was a good thing I'd read this, if only because I understood what had happened :)

Posted by: Inky on December 8, 2002 10:56 AM

Glad my tale of woe can help and educate others. :)

I spent a few hours this weekend cloning my 10gb laptop drive to a brand-new 40gb IBM Travelstar. It's nice to see the drive is '72% free' :)

Posted by: Miguel on December 9, 2002 2:22 PM

*ouch* ...why did i only look for "mozilla lost all bookmarks" AFTER it happened - and not before?! At least now i know :( !this hurt!

Posted by: t... on February 16, 2003 6:16 PM

very helpful posts here, but what i'd like to know is, am i just asking for it if i go in and trash the swap files? i used Cache Out to have my swap files on a disk other than my main hard drive, and they're currently eating up all the valuable space on that one. i've got about 7 swap files. is it possible to just delete them, and the OS will create more if it needs to? i feel like i'm playing with fire here. any help would be appreciated.

Posted by: vague blur on April 29, 2003 3:28 PM

Try the very neat and useful COCKTAIL:

http://www2.dicom.se/cocktail/index.html

It specifically has an option to delete your System CACHE files, including the VM files.

Posted by: Miguel on April 29, 2003 8:55 PM

this is kinda out of subject (it rather is a quota-subject; same faulty behaviour, though)

i ran into exactly the same problem today as one of my users complained once more about the dock always changing to the generic one and the desktop-background, too.

so i searched google (there's not much to find on this topic) and stumbled over this site. in my case the home-dirs are located on an x-serve with 10.2.4 and quota turned on. as we experienced all kind of problems (although i didn't know so far, that the default-desktop+dock was one of this kind, too), when the quota-limit gets reached, i always look at the consumed space whenever a user complains about strange behaviour.

but in this case, the user had just consumed about 600MB of 1GB quota. but last week i had the first time an incident, that a user complained about the cryptic "error -1425"(=quota full; thank you, apple!)-message without approaching even nearly the quota limit (Finder says 592MB; quota 1GB). but the "edquota"-command gives the following output:

/: 1K blocks in use: 1727736, limits (soft = 0, hard = 1048576)
inodes in use: 8958, limits (soft = 0, hard = 0)

after a reboot (OF THE SERVER!!!), edquota detects the right quota.

CONCLUSION:

- this happens with reached-quota-limit-user-accounts, too.

- whenever you experience such problems with a quota-on-fileserver serving home-dirs, have a look at edquota, wether the blockcount counts right.

Posted by: niko on May 6, 2003 1:00 PM

thnx bs, your my hero!

now everything works again.. I've lost 5h but what should you do in case you've lost ALL - damn..

so please apple: give me a sign when my space is full, pleeeeeeaaaaase.

Posted by: toby on June 12, 2003 3:51 PM

I have now experienced the exact same problem on my iBook about 3 times now in the last few months. It seems to affect all system versions up to now (10.2 - 10.2.6), so Apple is clearly sleeping...
Is it possible to recover lost mails and account info?

Posted by: Jan van Es on July 7, 2003 5:29 PM

At last, some info. I have been suffering from sudden VM failure. Lost preferences, went mad etc etc. I now use
http://www.mysterysoftware.com/DiskSpace/ and set it to
alert me when my scratch disk or VM swap disk has reached a certain threshold of mb remaining. It beeps and the menu reading flashes red. It set it to complain at about 400mb, which gives me about another hour before i think about restarting.

mad world. i really wish we could turn off VM sometimes.
I restart my computer 3 -4 times a week, just to resolve this
issue. I have a 30gb drive, with 10gb available for the boot partition.

yes, i too thought i was loosing my mind. Been using Macs for over a decade, and this VM thing is just a bit silly now.

Posted by: dirtymouse on December 1, 2003 10:53 AM

I am running OS X Server and i have two hard drives
One for the System
And other for storage(Files Receive Thru The FTP)

Anyway I have plenty of space in my storage Drive
But in my system drive It tellme that is full, i knew that was not possible because i don't put files on it so i check all the files and all the files in my system drive are less than two Gig so i check the information in my drive and it tellme that i have a hard drive of 37gigs and all 37 gigs are been used but as i said if i check the files the ones that i can see are less than 2 gigs , i know i have some invisible file using the other 35 gigs but how can i fix this with out deleting other invisible files that are needed for the system or how can i see invisible files and the sizes of them in the terminal
there is no way to install another program because my system is drive is full.
Thank you If some one can help
Gabriel

Posted by: Gabriel on December 4, 2003 1:55 PM

This is probably entry level stuff, but it's got me stumped. Had a folder on the Desktop for misc. files. Created a folder inside it for the PDF files which I grouped by sorting by type. But I could not drag them to the newly created PDF sub-folder which was too high up in the listing (Type=Folder and Type=PDF for the files). So I dragged the PDF folder out of the main folder file listing window onto the Desktop and then reselected the PDF files, 20 or so, in the window, dragged them to the folder on the Desktop and then dragged that folder back into the window. It showed up, entirely empty, files nowhere to be seen. Where'd the files go? I get the impression that this may have been an attempt to put files into an alias, don't know. Appreciate any help, feel free to preface it, "Hey, dummy!"

Posted by: Bob Hunt on December 17, 2003 2:30 PM

It's late march 2005 and the problem described persists. I have one Mail brainwashing every month or so. Running the latest system on 1 gig of ram. Not particularly happy.

Posted by: bo on March 31, 2005 10:39 AM

I wrote the initial past close to 3 years ago, and according to all the evidence I have seen, OSX does in fact warn you when you are getting dangerously low on disk space. The common wisdom nowadays is that you need plenty of RAM in order to give the system decent breathing room, and you also need to have a nice percentage of free disk space. 1GB is decent.

OSX has improved on this topic, but we've also improved in knowing what to expect from the system based on what you give it resource-wise.

Bo, if Mail is indeed getting hosed on your system on a regular basis I'm more keen to look elsewhere for the culprit, unless (you didn't mention this) you have a small amount of HD space left.

My typical diagnostic routine is running Diskwarrior, Disk Utility's repair disk, repair permissions, and then finally (if that doesn't help) simply dragging your mail folder out of the library, deleting all Mail .plist and pref files, and then recreating your account info, and then reimporting your mail using Mail's own import feature or, if you'd rather do it manually, accordingly placing the mbox files in your new Mail library folder.

This is tech advice but I give you no warranty on it.

Posted by: Miguel on March 31, 2005 11:04 AM

Just wanted to let you know that they guy that suggested mac cocktail was right on. It fixed my problem and I recovered all the gigs that were lost to Virtual memory.

And if you want to check out if that's your problem do as the other guy said

"If you ever think 'hey im sure I had more disk space' just type 'cmd shift g' in the finder and type in '/var/vm/' up comes your vm storage."

I had over 5Gigs stored there (do a get Info on the swapFiles)

Thank you Mac Cocktail, and only like $15.

by the way, I think that the problem was that I installed too many fonts, requiring too much Ram per program, because I have over 1 Gig Ram.

good luck all.

Posted by: Rivocrates on April 3, 2005 11:01 PM

every time i try to go to download any thing on my computer a dailog box would alway apear telling
me to close and it wuld shut the promable down saying not enought hard drive space not enought space as well not enought menory so can you help repair my system now

Posted by: Ronnie on June 22, 2005 8:50 PM

After buying a mac mini to try out OS X Tiger, I was impressed for 20 minutes by all the flashy effects, then got down to work. I started downloading my pictures from my camera and laptop to the mac mini via iPhoto. All was good for a few weeks, then all of a sudden, out of the blue my iTUnes library vanishes. No problem I thought, i'll just drag my iTunes folder over to iTunes. It worked. Yay, no more problems, but then the next day, the dock went awol like you said above, with loads of question marks over the dock. I knew what had happened, so I checked my iPhoto, and all my photos were gone. All of them. Over 10,000. Lesson of the day; back up your work, and use a reliable system like Linux for things like this.

Posted by: spud on July 24, 2005 6:12 AM

HELP, anyone. I got this message today "Word cannot save or create this file. The disk may be full or write protected." and then mail was acting weird then the computer FROZE. Is this a sign of what you are all discussing above or is this a microsoft thing. Does anyone know. Should I start freaking out? HOw can I check if somehting is wrong?

Posted by: MG on August 13, 2005 5:09 PM

I have something to contribute which may help some readers.
I found this discussion searching for an answer to a 842MB VM Storage file that suddenly "appeared" (I had to show hidden files). Before and after I saw this page I used Cocktail's "clean system swap files," but the VM Storage stubbornly refused to go away.
Then it hit me (1) The MB disappeared after I came back from booting up in 9.2.2 to delete Norton applications which are no longer usable in/with 10.4.2, and (2) the file had the C-based Cubist smiling Mac face (which is not used in any version of OS X that I have seen).
The fact that Cocktail apparently could not find it, the trip taken to 9.2.2, and the face led me to boot 9.2.2 again. Sure enough, even though I had not had it turned on before upgrading to 10.4, "Virtual Memory" in the Memory Control Panel was turned on and automatically set to no less than 833MB [I tried to lower it, but somebody's mention of a certain minimum percentage of free disk space being reserved -- maybe not on this page (I looked at a few pages about OS 9 and virtual memory, too, before going back to find the mysteriously re-set control panel). So I turned it off. It informed me it would take effect at the next restart.
I booted into 10.4.2 and found the file still there. Re-booted into 9.2.2 and checked internal drive space. I was back to 1.5GB instead of 690 or so MB.
When I came back into 10.4.2, the drive space was still correct at 1.5GB.
Moral: apparently upgrading to 10.4.2 turns on VM in 9.2.2. That VM won't be erased by Cocktail or anything else I tried in 10.4. It had to be done in 9.2.2 and 9.2.2 had to be re-booted. I guess it freed up the space when it read that control panel on startup.

As for the OS X side, compared to 10.2, 10.4 is 'way better than 10.2 (I didn't go through the 10.3 step). I am running a G3 Blue Tower with just a 6GB internal drive (what can I say? sounded like a hell of a lot in 2000). Once I reinstated a practice from the days of the LC III of putting as many applications as possible on another drive (and being more assiduous about not saving superfluous e-mail), I have had virtually no -- if any -- problem with OS X's swap files and caches unnecessarily taking up disk space. Safari 2 in particular seems to be much more discriminating: the cache folders stay a reasonable size (under 7 MB) as compared to Safari 1.x in 10.8, which, when I first ferreted out loss of disk space in 10.2 had cached something like 70 MB of stuff. Firefox and Mozilla, for some reason, duplicate Bookmarks, named incrementally. When I looked in Firefox's profile I found well over a hundred; at 625KB each, they were taking an unnecessary amount of disk space. I hadn't been using Mozilla as much, but it had nevertheless made something like 75-80 copies at 444K each.

Posted by: Bill on August 15, 2005 12:43 AM

For anyone reading this and having low disk problems.

The stock Tech answer is: Never get below 20% free space. As the available disk space goes past about the 80GB level you can reduce that percentage.

This is OS indepedent.

Windoze , OS X, Linux, any system that uses dynamic VM.( For those not running/not wanting to run dynamic VM, make sure that you create a VM file big enough to handle everything. How big? Turn dynamic on, open all of your apps, check size of file and there go. )

If you do not follow this simple rule, aside from the disk becoming full you will fragment your data so increbily bad it will usually toss all of the directory mappings out and make it appear as though your data is gone.

If you are close and do not want to fall into the abyss. Cleanup your old files and defrag.

How?
Windoze users can defragment using the system tool. This is by far not the best way, but it the free. There are TONS of windoze fixing apps (it breaks, a lot) out there. Symantec makes some decent ones.

OS X users leave your box on for more than 8 hours it will do it itself. Or Go pick up Diskwarrior or Tech Tool Pro ( My fav) and you can manually do it.

Linux users you know what to do, if you don't ...ummm buy a book?

Bottomline, VM uses and releases disk space a lot, this in and of itself will cause data to get jumbled. You need to give it room to play its disk space Mombo or you will pay the price.

Hope this helps.

Posted by: Ryn on November 22, 2005 6:56 PM

HEy...... I was reading all of the above comments and something similar happened to me..... I was installing Norton Utilities and the computer asked me to reboot.... when the computer was up again, instead of having 39 GB i had only 9 GB left.... so 30 GB went missing..... what can I do find the hidden folders and programs and data..... PLEASE HELP.... thank you in advanced

Posted by: Frank Buenrostro on December 3, 2005 1:57 AM

Why do people use up so much disk space anyway? If you really need it (like I sometimes do for my movies), just use an external hard drive!

Posted by: deranged_physicist on March 22, 2006 6:10 PM
Post a comment