Afsctool Hfs Compression For Mac

Disk Utility User Guide

This is a review of the new Mac OS X Snow Leopard program Squeeze 'Squeeze is a background file compressor, which uses the new HFS-compression technology in Snow Leopard to transparently compress the folders you configure it to process. Mac OS will read those files normally, they will just take less disk space.' Since HFS compression is completely transparent, there are no adverse effects to expect (other than browsing the content libraries being almost unnoticeably slower). To start, you’ll need a command-line tool called afsctool which can compress (and, amongst other features, decompress) folders using HFS compression.

Disk Utility on Mac supports several file system formats:

Apple File System (APFS): The file system used by macOS 10.13 or later.
Mac OS Extended: The file system used by macOS 10.12 or earlier.
MS-DOS (FAT) and ExFAT: File systems that are compatible with Windows.

Apple File System (APFS)

Apple File System (APFS), the default file system for Mac computers using macOS 10.13 or later, features strong encryption, space sharing, snapshots, fast directory sizing, and improved file system fundamentals. While APFS is optimized for the Flash/SSD storage used in recent Mac computers, it can also be used with older systems with traditional hard disk drives (HDD) and external, direct-attached storage. macOS 10.13 or later supports APFS for both bootable and data volumes.

APFS allocates disk space within a container (partition) on demand. When a single APFS container has multiple volumes, the container’s free space is shared and is automatically allocated to any of the individual volumes as needed. If desired, you can specify reserve and quota sizes for each volume. Each volume uses only part of the overall container, so the available space is the total size of the container, minus the size of all the volumes in the container.

Choose one of the following APFS formats for Mac computers using macOS 10.13 or later.

APFS: Uses the APFS format. Choose this option if you don’t need an encrypted or case-sensitive format.
APFS (Encrypted): Uses the APFS format and encrypts the volume.
Il minihost modular download sites. APFS (Case-sensitive): Uses the APFS format and is case-sensitive to file and folder names. For example, folders named “Homework” and “HOMEWORK” are two different folders.
APFS (Case-sensitive, Encrypted): Uses the APFS format, is case-sensitive to file and folder names, and encrypts the volume. For example, folders named “Homework” and “HOMEWORK” are two different folders.

You can easily add or delete volumes in APFS containers. Each volume within an APFS container can have its own APFS format—APFS, APFS (Encrypted), APFS (Case-sensitive), or APFS (Case-sensitive, Encrypted).

Mac OS Extended

Choose one of the following Mac OS Extended file system formats for compatibility with Mac computers using macOS 10.12 or earlier. La cimbali s39 barsystem manual.

Mac OS Extended (Journaled): Uses the Mac format (Journaled HFS Plus) to protect the integrity of the hierarchical file system. Choose this option if you don’t need an encrypted or case-sensitive format.
Mac OS Extended (Journaled, Encrypted): Uses the Mac format, requires a password, and encrypts the partition.
Mac OS Extended (Case-sensitive, Journaled): Uses the Mac format and is case-sensitive to folder names. For example, folders named “Homework” and “HOMEWORK” are two different folders.
Mac OS Extended (Case-sensitive, Journaled, Encrypted): Uses the Mac format, is case-sensitive to folder names, requires a password, and encrypts the partition.

Windows-compatible formats

Choose one of the following Windows-compatible file system formats if you are formatting a disk to use with Windows.

MS-DOS (FAT): Use for Windows volumes that are 32 GB or less.
ExFAT: Use for Windows volumes that are over 32 GB.

See alsoPartition schemes available in Disk Utility on MacAbout Disk Utility on Mac

10.6: Compress files with HFS+ compression | 34 comments | Create New Account

Click here to return to the '10.6: Compress files with HFS+ compression' hint

The following comments are owned by whoever posted them. This site is not responsible for what they say.

Maybe the compression criteria has to do with whether it will be faster or not. If you can only get minimal compression, you won't really save time on the disk read after decompressing the file. At least, that is the whole reason suggested for compression in the Ars Technica article.

Ugh, so it looks like the file size reported in the Finder is the uncompressed size. Aside from the whole KB vs. KiB change, this just creates confusion. When I see the file size in the Finder, I expect it to mean how much space it takes up on the disk (the compressed size).
When copying or reading a file, where it's uncompressed automatically to be presented as if it was the uncompressed data on the disk, sure I need to see the uncompressed size.

Since I submitted this hint I've been working on a solution for the lack of information about HFS+ compression files, by developing this command line utility:
afsctool
(source is included)
Here's the usage information for it: The -c flag applies in place HFS+ compression to a file or recursively to a folder, with the option to specify the compression level (default is 9), change the default maximum size (in bytes) of files to process (default is 20971520, which is 20 MiB), and specify the minimum savings for a file to be HFS+ compressed. If the -k flag is given then compressed files will be checked after compression against the uncompressed files, and if any corruption has occurred then the file will be reverted back to its uncompressed form. The -l flag lists HFS+ compressed files in a folder or if the -c flag is given, it lists the files that fail to compress. The -d flag decompresses HFS+ compressed files (similar to afscexpand, but sometimes more reliable). The -a flag flattens a HFS+ compressed file to the data fork of the destination file, decompressing the original compressed file afterwards if the -d flag is given. The -x flag does the reverse of -a, unflattening a file with HFS+ compressed data in the data fork to the destination file, then decompressed the destination file is the -d flag is given. The -v flag increases verbosity.
Here's some example output: A warning about using this utility; do NOT try to compress the System folder. Any files that were not compressed in the System folder were left uncompressed for a reason! If you do try to compress your System folder further, then you will very likely end up having to reinstall Mac OS X.

This looks really useful! I'd like to incorporate this into my Backup Bouncer test suite:
http://n8gray.org/code/backup-bouncer
Do you approve? Do you have this tool hosted somewhere so I can check for updates?
Thanks,
-n8

Sure, as long as Backup Bouncer remains open source I have no problem with you reusing the code in afsctool. Currently I'm keeping this link updated with the latest version:
http://files.me.com/brkirch/ijt4f7

Your tool is great; the source contains a lot of magic numbers buried directly. I guess it took a lot of times for you to reverse-engineer the format!
If you could write a blog post about the format of the HFS compressed file, it would be very helpful for the rest of us..
By the way, do you think there is a private system call which does the compression? Can it be found inside the source code of xnu?

The first thing that I tried to do was figure out the system call used for compression; ditto uses the private framework Bom to compress files. However, looking into it further I found that the Bom framework makes a call to another private framework: AppleFSCompression. Unfortunately the syntax for the functions in AppleFSCompression is far from obvious, especially since it is a private framework which means that there are no included headers for it (and it also means that it will probably remain closed source). It didn't in the end matter however, because I found that zlib is used for the actual compression and that library is well documented. So I decided to simply figure out how the HFS+ compressed files are constructed and added that to my program.
Here's how the HFS+ compression is applied:
1. Check the file to ensure it does not have a resource fork or com.apple.decmpfs extended attribute.
2. Construct the headers, calculate the number of 64 KiB blocks needed based on the source file size.
3. Compress the 64 KiB blocks using zlib (Apple uses compression level 5, but other compression levels also work); if there is only one block then append it to the com.apple.decmpfs extended attribute if the compressed data is 3786 bytes or less in size, otherwise the compressed data is put into the resource fork of the file. If the resource fork is used to store the compressed data, then no block is allowed to be larger after compression (if a block is larger after compression then compression for the entire file will fail). After the compressed blocks are created then their locations and sizes are written to the resource fork data header.
4. Add the com.apple.decmpfs extended attribute to the file, then the resource fork if one is needed.
5. Truncate the data fork length to zero and use chflags to set the HFS+ compression flag.
This produces compressed files that are identical to the ones produced by ditto, provided compression level 5 was used for the zlib functions.

The download link is broken->goes to MacOSXHints.
Download afsctool here: http://files.me.com/brkirch/ijt4f7

Thanks for posting a fixed link, when I previewed the post it looked like the redirect it added should work, but apparently not. There is also one other mistake in my post; the default compression level is 5 (the same as the compression level ditto uses).

Just a quick q - everything I try to compress via afsctool seems to give me an 'Unable to compress file,' which isn't particularly informative…Â Any clue as to what could be causing everything to fail out? And it's not the obvious - I am on 10.6, HFS+…

The latest version of afsctool will only fail to compress a file if one of the following is true:
-the file has a resource fork
-the file has a zero length data fork
-the file is not a regular file (if it is for example, a symbolic link)
-the file will become larger after compression is applied
-the file is greater than 2 GiB in size
-there are insufficient privileges to change the file
-user specified conditions aren't met

I have made a website for my software:
http://web.me.com/brkirch
Future updates to afsctool will be posted there.

> (in fact, contrary to what some posts on this site have suggested, command line utilities like strings will see the same file regardless if it is compressed or not)
Not entirely true. While 'cat' will uncompress the file first, 'more' and 'less' will show a raw binary dump of the compressed file. That's a quick and dirty way to tell if a file is compressed, at least until Apple fixes it. Your utility is far more useful obviously.

Yes I could have worded that better. After I submitted this hint I found that there are some obscure ways to tell if a file is compressed. For example, if stat -f %f [file] returns a number that has the 32 flag set (the UF_COMPRESSED flag) then the file is HFS+ compressed. I can't say I can see how more or less can be used to identify compressed files though, as the data being printed to the screen is the uncompressed data, not the compressed (just tried it, you may have somehow gotten a different result but I tried it on some compressed files and it is definitely showing the uncompressed data).

Less certainly works differently than cat, but Apple would have to specifically modify it to not use standard POSIX calls. Here's the source to cat; search for 'fdopen' to see where it's opening its input file.

Afsctool Hfs Compression For Mac Pro

Less is a surprisingly complex utility but it still uses a standard POSIX function to open a regular file in this source file, search for open(qopen_filename, OPEN_READ) within the file to find it.

man 3 fdopen 2 open will confirm that both of these system calls are standard POSIX calls, so they don't really know anything about HFS vs. any other file system.

(Also, 'more' is simply the compatibility mode of 'less.' They're the same file, hard linked.)

Nope, I just compressed an rtf file, then used 'more' and 'less', the file appeared normally, as if nothing was special about it, no raw compressed data but the original, readable text.

Wow, I just saved several gigabytes by compressing a few big apps (just using 'sudo ditto --hfsCompression'), I can't tell the difference except for the free space I gained!
For example Photoshop still takes the same amount of time to launch after a cold reboot (even slightly quicker, if anything), does not complain about damaged files, I was afraid it might give me some DRM-like nastiness with moved/changed files, but no, everything just works like before, only taking a few hundreds of MB less!
Next up is the Application support folder :-)

Only one strange thing: when compressing some application folders, I got messages like
ditto: /Applications/Utilities/Gecomprimeerd/ResDataBridge 1.0/.BC.n6FLVl: Permission denied
ditto: /Applications/Games/StellaOSX2.2/Roms/.BC.YRr5fp: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.j2H1FL: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.r0fvwu: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.kCj925: Permission denied
ditto: /Applications/Games/MacMAME/old/Documentation/.BC.vm8yx2: Permission denied
I just listed a few, but I got similar messages for the Photoshop folder and several others.
When I go look for these mysterious files (.BC.xxxxxx), they simply do not seem to exist. 'ls -a' on the original folder does not show them. The compressed applications work just fine afterwards.
What are these 'files'? Is it just a bug in ditto? Is it some internal file system stuff that is not supposed to show up in the higher abstraction levels? It this were Windows I would immediately suspect a virus, but on a Mac that's obviously impossible..

Now all we need is an Automator script or Applescript that will compress all files moved into a folder.

the script is easy: the 'with administrator privileges' part is unnecessary (and annoying) if you're compressing files you have ownership of. It authenticates you whether it needs to or not.

Afsctool hfs compression for mac computers

I'm pretty certain you always need to have sudo to run the ditto --hfsCompress command.

Universal Catalytic Converter with Stainless Clamps, Better .... Dec 17, 2019 This type of universal catalytic converter is not a great solution for small cars that have the catalytic converter up close to the exhaust manifold. Tools needed are a Sawzall or angle grinder with a metal blade and some wrenches for the clamps. Some universal catalytic converter install notes. The Magnaflow catalytic converter in the video. Catalytic Converter Quick Fix; It would involve cutting out the existing cat and welding the. I know a rookie will take half a day at least without a. Cost to weld in new catalytic converter. It had rusted right through between the catalytic converter and the silencer.' How to Fix Your Car Exhaust, Without Welding or Power Tools.

never use sudo when calling shell scripts from applescript - opens a security hole. that's what 'with administrator privileges' is for. see this tech note

That attitude probably creates more security holes than all technical errors put together.
Security is not an absolute matter, nor does it have the same requirements everywhere. Insisting on following rigid little rules like this creates annoyances and inconviences that people then respond to by circumventing the security mechanism entierly, e.g., telling anyone who sits down at the machine their password so they can run secure processes, chowing all the files to their own user, etc.
For instance on my laptop I add myself to the wheel group and let the wheel group use sudo without a password. Would I do this on a corporate machine or webserver? No! But does that make it a security risk on my laptop? No. The point of security on my laptop is to protect my private documents and files. If they already have access to my user account the game is over.

I nominate this for the best SL hint so far. I've been enjoying compression of many files. I've found it especially effective with PDFs and with our X-ray data files. I also squeezed by fink /sw directory down by a factor of two. By the way, 'du' provides a quick way to assess hfs compression, eg:

Afsctool Hfs Compression For Mac Osx

Awesome tip, thanks!
Now, all we need is some enterprising developer to make a nice user friendly app to compress and get info on compressed files.
I am a bit nervous using the terminal to modify/convert files in the event I cause a problem with a typo.. ;)

I just discovered one - and it's free as a special via MacHeist.com right now.
It's called Squeeze.

There's HDCleanUp, does HFS+ Compression and a bunch of other related tasks. Only $10.

MacHeist.com has just begun to give away an app for free that basically provides a nice GUI for this tip - I think the give-away is only for one week.
Go to MacHeist.com to download Squeeze, the author's website is http://www.latenitesoft.com/squeeze/

Has anyone found a way to make Finder accurately report the compressed file sizes?
How do I know how much space I actually have on my drive when Squeeze says I have saved 22.9 GB and Finder says I have only 2.9 GB free (the same amount I had before Squeeze squeezed)?
Path Finder also shows only 2.9 GB free.

This HFS+ compression thread is the first clue to a problem I have. After a boot failure the local apple store did an archive and install. The OS was set up for multiple users with one admin. The admin account data is fine but all the other users data is invisible with a .BC file prefix on all the user data files. The current OS is 10.5.8. Could they have used 10.6 initially and compressed the users files? All system files are OK. Is there a way to restore the 190 GB of now invisible and unreadable files? In terminal mode they all show up with a ls -l command but the filenames are coded. The only solution the apple store offers is a wipe and install.

Afsctool Hfs Compression For Mac Os

There's a menu bar add-on called Squeeze which does HFS compression on folders in the background. But Squeeze won't compress anything in the /Developer folder. To get around this limitation and save some space in /Developer paste the following shell script into a file called dev-compress.sh .. .. then prep and execute.. After a few months I have seen no adverse effects on using XCode 3 or XCode 4 with these folders compressed.

Related to HFS+ Compression is the behavior of Retrospect 6.1 backup.
Apparently running Retrospect on a Snow Leopard system causes all HFS+ Compressed files to become uncompressed (and manifests itself as a multitude of execution errors described in the the log as changes of the modification date/time to the current date/time).
See 'https://discussions.apple.com/thread/2674794' and 'http://www.techcare.com/blogs/jk-buzz/102-jk-buzz-172010.html'.

I've found a nice utility that performs this as a background process:
http://latenitesoft.com/clusters/
So far it has worked well for me, including compressing all Adobe products, saving approximately 50% space, which makes quite a difference now that I've moved to a considerably smaller SSD drive.

Afsctool Hfs Compression For Mac Computers

There is a new GUI tool for HFS+ compression in appstore called MoreSpace Folder Compression.
http://itunes.apple.com/app/morespace-folder-compression/id521635253?mt=12
Simply but usable.

Comments are closed.