The Problem

The /storage filesystem created by the Raspberry Pi build of the OpenELEC XBMC Media Center is limited to 755.9MB, no matter the size of the SD card upon which it is installed.

All of my content is stored on two ReadyNAS network storage appliances, so the space limitation was easy to overlook until I attempted to download several add-on skins that failed during installation and were not cleaned up properly. Since the /storage filesystem also caches thumbnails and other movie and episode information in a hidden folder (/storage/.xbmc/), the Media Center staunchly refused to add more content, even after I deleted and re-added all video sources.

The XBMC System Info / Storage screen shows that /storage is full, confirmed by running “df“.

# df
Filesystem           1K-blocks      Used Available Use% Mounted on
none                    189512     89556     99956  47% /dev
/dev/mmcblk0p1          127730     99214     28516  78% /flash
/dev/mmcblk0p2          774064    773968         0 100% /storage
/dev/loop0               89472     89472         0 100% /
none                    191008         0    191008   0% /dev/shm

The Solution

You will need to delete and re-create an existing partition on your SD card. While everything worked for me without any data loss, it is possible that this may forever delete all your add-ons, your cached data and thumbnails, and many of your settings. If you are comfortable and OK with doing so, then proceed onward. If not, stop and do nothing.

First, the caveat:

Author’s Note: This worked for me on my RPi running the OpenELEC 3.01 build with Linux 3.6.11 and XBMC 12.2 (“Frodo”). It might not work for you on any other version, or even on the same version. If it doesn’t work for you and you break your Raspberry Pi or your Media Center because you didn’t understand the computer science behind it (or, Heaven forbid, I made a mistake in the instructions) — well, that’s not my problem and I don’t want to hear any whinging. You assume all risk and accept any data loss. I am not liable for anything, and neither is my family, my friends, my cats, my co-workers, the good and talented folks at Raspberry Pi or the XBMC Foundation, my web hosts, my ISP, the Easter Bunny, Barack Obama, or Wil Wheaton.

1. SSH to your Raspberry Pi as root (the default password is “openelec”, without quotes)
2. Get out of the default /storage partition.

   cd /

3. Disable XBMC.

   touch /var/lock/xbmc.disabled

4. Stop the XBMC server. OpenELEC’s wiki recommends the following, but only “as a last resort”:

   killall -9 xbmc.bin

   I prefer to use a gentler approach which allows the process to close down properly.

   killall -SIGHUP xbmc.bin

5. (Optional) Verify by running “top” that the XBMC processes are no longer running.
6. (Optional) Verify by running “mount” that the /storage filesystem is mounted. Make a note of the name of the underlying filesystem upon which /storage is mounted, e.g. “/dev/mmcblk0p2”.
7. Unmount the /storage filesystem.

   umount /storage

8. (Optional) Verify by running “mount” that the /storage filesystem is no longer mounted.
9. Start up the partition editor (the partition specified here must match the output of the underlying partition name of the /storage mount from either the “df” or “mount” commands issued earlier, without the “p*” suffix; i.e. specify the “/dev/mmcblk0” device when editing the “/dev/mmcblk0p2” partition)

   parted /dev/mmcblk0

10. Change display mode from size to sectors. — unit s
11. Display the list of partitions. — p

    p
    Model: SD 00000 (sd/mmc)
    Disk /dev/mmcblk0: 7995392s
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:
    
    Number  Start    End       Size      Type     File system  Flags
     1      2048s    258047s   256000s   primary  fat16        boot, lba
     2      258048s  1830911s  1572864s  primary  ext4

    Take note of the starting sector number of the second partition. The starting sector number of the second partition on my 4GB SD card is “258048”.

12. Remove the second partition. — rm 2
13. Re-create the partition, using the same starting sector as the ending sector and “-1” (minus one; which means use all remaining available space). — mkpart primary 258048 -1
14. (Optional) Enter “unit GB” and then enter “p” again to verify the new partition settings.

    unit GB
    p
    Model: SD 00000 (sd/mmc)
    Disk /dev/mmcblk0: 4.09GB
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    Disk Flags:
    
    Number  Start   End     Size    Type     File system  Flags
     1      0.00GB  0.13GB  0.13GB  primary  fat16        boot, lba
     2      0.13GB  4.09GB  3.96GB  primary  ext4

    The new partition on my 4GB SD card is 3.96GB (prior to allocating space needed by the filesystem).

15. Exit the partition editor. — quit
16. Check the updated filesystem, then resize and remount it:

    e2fsck -f /dev/mmcblk0p2
    resize2fs /dev/mmcblk0p2
    mount /dev/mmcblk0p2 /storage

17. (Optional) Verify with “df -h” that the filesystem is correctly sized:

    # df -h
    Filesystem                Size      Used Available Use% Mounted on
    none                    185.1M     87.5M     97.6M  47% /dev
    /dev/mmcblk0p1          124.7M     96.9M     27.8M  78% /flash
    /dev/loop0               87.4M     87.4M         0 100% /
    none                    186.5M         0    186.5M   0% /dev/shm
    /dev/mmcblk0p2            3.6G    756.6M      2.7G  21% /storage

    My updated /storage filesystem is now 3.6GB. Perfect!

18. Re-enable XBMC and the Media Center will automatically restart.

    rm /var/lock/xbmc.disabled

While at first this may seem to be a lengthy, complicated how-to, the actual steps are actually rather simple. With just a few minutes’ work, your OpenELEC SD card will be expanded to its full capacity and the Media Center will be functional once again.

As to my original problem of failed add-on installations, in my meanderings through the operating system, I discovered that XBMC does not delete the original downloaded ZIP files when you delete an add-on; it deletes only the unpacked files from the /storage/.xbmc/addons/ folder. The orphaned ZIP files (which can be 100MB or more for complex skins) are located within the /storage/.xbmc/addons/packages/ folder. I recommend only deleting those packages for add-ons that you have deleted via the user interface.

The following tutorial will help you generate a 367MB Raspbian image (uncompressed).

The standard 2012-12-16 version of Raspbian “Wheezy” fills up 1.5GB (or 88%) of a 2GB SD card, which has a formatted capacity of only 1.8GB. While I have a larger capacity SDHC card for this particular RPi (I have five “pies”), one of the projects I’m considering is potentially quite data intensive, so I want to reserve as much disk space as possible.

My previous article began addressing the removal of unnecessary packages but its primary focus was on reducing I/O usage, however this article is focused on conserving disk space and reducing the operating system fingerprint. This optimizes the RPi for a server-type environment, rather than its use as a desktop workstation.

This minimalist image is different from others since it is significantly smaller than most images, yet it retains sound and touch-screen support.

Remove the GUI Modules

Let’s start by getting rid of over 325 optional packages (about 700MB) used by the graphical user interface:

sudo apt-get purge xserver.* x11.* xarchiver xauth xkb-data console-setup xinit lightdm lxde.* python-tk python3-tk scratch gtk.* libgtk.* openbox libxt.* lxpanel gnome.* libqt.* libxcb.* libxfont.* lxmenu.* gvfs.* xdg-.* desktop.* tcl.* shared-mime-info penguinspuzzle omxplayer gsfonts
sudo apt-get --yes autoremove
sudo apt-get upgrade

Remove Development Packages (Optional)

My personal minimalist RPi image retains most of the default installed development packages.

sudo apt-get purge gcc-4\.[0-5].*

However, you can optionally recover almost 150MB of space by removing the majority of compiler and development packages from your image.

sudo apt-get purge `sudo dpkg --get-selections | grep "\-dev" | sed s/install//` gcc-4\.[0-6].* python.*
sudo apt-get --yes autoremove

Disable Memory Swapping

Memory swapping allows part of the usable memory of your computer to be stored on disk, effectively allowing the system to have more RAM than it does physically. Disabling swap files helps free up valuable disk space.

sudo swapoff -a
sudo apt-get purge dphys-swapfile
sudo rm /var/swap

Delete Unnecessary Extras

A few stray files and folders slice up more than 125MB of the Raspberry Pi.

sudo rm -rf /usr/share/doc/* /opt/vc/src/hello_pi/hello_video/test.h264 /home/pi/python_games
find /usr/share/locale/* -maxdepth 0 -type d |grep -v en |xargs sudo rm -rf
find /usr/share/man/* -maxdepth 0 -type d |grep -Pv 'man\d' |xargs sudo rm -rf

Replace SSHd with Dropbear

Dropbear is a lightweight SSH v2 server which lacks backwards compatability support for SSH v1 (translation: smaller code base, uses less memory and fewer resources) and is secure enough for general use — ideal for the Raspberry Pi.

sudo apt-get install dropbear openssh-client
sudo /etc/init.d/ssh stop
sudo sed -i 's/NO_START=1/NO_START=0/' /etc/default/dropbear
sudo /etc/init.d/dropbear restart

After testing that you can still connect to your RPi via SSH, remove the default ssh server.

sudo apt-get purge openssh-server.* openssh-blacklist.*

Eliminate Rarely Used Caches

Prevent aptitude from caching package data by creating /etc/apt/apt.conf.d/02_nocache. This will slow down package updates, but will save about 35MB of disk space in the final image. You could remove the aptitude package altogether, but I find its presence more useful than not, especially when the bulky cache files are removed.

Dir::Cache {
   srcpkgcache "";
   pkgcache "";
}

Since we’re done using apt for package updating, after you reboot your Pi with sudo reboot, you can remove the orphaned cache files to recover another 35MB of disk space:

sudo rm -f /var/cache/apt/*cache.bin
sudo apt-get --yes autoclean
sudo apt-get --yes clean

Prepare for Final Imaging

Numerous files and logs are safely re-created or re-fetched as needed. Removing them will shrink the overall size of the finalized minimalist distro.

sudo find / -type f -name "*-old" |xargs sudo rm -rf
sudo rm -rf /var/backups/* /var/lib/apt/lists/* ~/.bash_history
find /var/log/ -type f |xargs sudo rm -rf
sudo cp /dev/null /etc/resolv.conf

Having also incorporated some drive-I/O-reduction techniques, running df -hlP on my Raspberry Pi shows disk usage of only 367MB, a significant reduction from the original 1.5GB. Your mileage may vary. Shutdown the Raspberry Pi with sudo halt, remove the SD card from the Raspberry Pi board and access it from a Mac or Linux computer to create the final image.

Author’s Note: If you’re just cleaning up your Raspbian installation and have no need to create a distributable image, you can stop now.

To determine which drive designation to use on a Mac, type diskutil list and locate the Identifier of the Linux disk type that ranges around 1.9 GB. On my MacBook Pro, mine is “disk1s2”.

sudo dd if=/dev/disk1s2 of=2012-12-16-wheezy-raspbian-rr-minimalist.img count=3788800

An archive image file is transferred to your Linux PC or Mac, ready for the next steps to generate a zero-byte-filled, compressed final distribution image. Since I’m using a Mac, for which the steps are not very straightforward because the Mac will not natively mount the RPi ext4 partition, I’ll cover that in a future article.

In the meantime, for native Linux users, you can generate a compressed image with the following generic steps.

mkdir mnt
sudo mount -o loop,offset=$((512*122880)) 2012-12-16-wheezy-raspbian-rr-minimalist.img mnt
sudo sfill -z -l -l -f mnt
sudo umount mnt
7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on 2012-12-16-wheezy-raspbian-rr-minimalist.img.7z 2012-12-16-wheezy-raspbian-rr-minimalist.img

If you’ve gotten this far, congratulations! The original Raspbian installation image uncompressed to 1.5GB, while the minimalist version is well under 600MB — or even sub-400MB, depending on the options chosen. Use your image any way you want in-house, however I wouldn’t start distributing your image quite yet; wait for a future article where we cover the final topics of security and key re-generation. And no one, not even the builders of Raspbian, seems to have properly addressed the licensing concerns of the included non-free hardware drivers.

Many of the modern single-board or System-on-Chip computers (such as Raspberry Pi or BeagleBone) use standard SD memory cards for drive volumes. Since frequent writes eventually damage flash memory systems, in order to extend the life of the SD card, it is important to reduce as many superfluous writes as practical. Shutting down the Pi improperly while it is writing to the card can also induce errors and corruption to open volumes, so reducing writes can have the added benefit of prolonged stability of the file systems.

Author’s Note: This tutorial assumes you have the 2012-12-16 Raspbian “wheezy” distribution installed on your Raspberry Pi, and you have not begun additional customization. While the techniques described below generally apply to other common Linux distros, the instructions, code, and commands may not apply or function in those other cases. You have been warned.

While I’ll be using one of my Pi’s as a built-in intranet web server installed into my powered structured media cabinet (mainly as a wiki and general file server), the goal of significantly fewer periodic writes should be obtainable because I’m hoping that much of the information on the wiki will be fairly static. Worst case, the Pi could buffer logs in memory and then dump them to a NAS or to an Amazon EC2 instance — which would be another article for another time.

Install Sysstat

In order to measure the success of your efforts, you will need a method of monitoring disk activity — one of the most widely used commands of which is iostat, part of the sysstat package.

sudo apt-get install sysstat

Running iostat with no arguments displays aggregate usage information gathered since the system was last booted. Adding interval and count arguments first displays usage since boot, then displays usage during each time period since the previous report.

In the following example, three iostat reports are displayed; the first report on usage since boot is displayed instantly, followed by a usage report gathered over 300 seconds (or five minutes), followed by another five-minute report. Because the focus of this examination is on device usage and not CPU usage, the “-d” (or “device”) option is added; the opposite would require a “-c” (or “cpu”) option.

Run the command before you go any further in order to test the impact of the optimizations below.

$ iostat -d 300 3
Linux 3.2.27+ (raspberrypi) 	2013/02/07 	_armv6l_	(1 CPU)

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.13         2.87         0.95     138189      45840

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.05         0.00         0.35          0        104

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.06         0.00         0.49          0        144

According to the above initial results, during the first five minutes of examination, 104 kilobytes of data was written to disk and 144 KB was written during the second five minutes, for a total of 248 KB over ten minutes. No reads were performed during the entire test.

Disable Swapping

Memory swapping allows part of the usable memory of your computer to be stored on disk, effectively allowing the system to have more RAM than it does physically. Even on high-power Windows or Mac computers, running applications with extensive memory usage beyond the hardware RAM will result in the computer slowing down while the hard drive light blinks incessantly, indicating that the OS is swapping in and out available virtual RAM.

Raspbian utilizes a swap file (managed by dphys-swapfile) in lieu of a more standard swap partition. Removing the functionality is a better long-term solution that simply turning it off.

sudo apt-get remove dphys-swapfile

Remove Unused Packages

In order to build a distribution that satisifies the needs of the majority, Raspbian includes many packages that many RPi users won’t use, especially those that solely intend their Pi to be headless. Removing packages needed for the X Window System graphical user interface will recover a significant amount of disk space, as well as remove always-on daemons that continually generate log messages and other disk I/O (even if you never use the desktop).

sudo apt-get --yes purge xserver.* x11-.* xarchiver xauth xkb-data console-setup xinit lightdm lxde.* python-tk python3-tk scratch gtk.* libgtk.* openbox libxt.* lxpanel gnome.* libqt.* gvfs.* xdg-.* desktop.*
sudo apt-get --yes autoremove
sudo apt-get --yes autoclean
sudo apt-get --yes clean

Author’s Note: Use the above purge command with caution if you have already begun customization of your Raspberry Pi. You may delete packages that are unintended.

Purely to recover more disk space (and not neceesarily in consideration of the SD card life expectancy), I used sudo ncdu from the root partition to remove unused locales from within /usr/share/locale/ as well as everything under /tmp/.

Also consider removing the following files:

• /var/swap (100 MB)
• /opt/vc/src/hello_pi/hello_video/test.h264 (30 MB)

Before I started purging, disk usage on my Pi was 1.5 GB (as determing by running df -hlP). After performing all of above suggestions, total disk usage was reduced to only 609 MB, recovering almost 60% of installed space. Your milege may vary.

Reduce Unnecessary Logging

Every couple of minutes cron jobs tend to add log entries to /var/log/auth.log similar to the following:

pam_unix(cron:session): session opened for user root by pi(uid=0)
pam_unix(cron:session): session closed for user root

Edit the /etc/pam.d/common-session-noninteractive configuration file and add the following line near the bottom of the file just before the pam_unix.so call to prevent cron jobs that complete successfully from adding the authorization logs.

session [success=1 default=ignore] pam_succeed_if.so service in cron quiet use_uid

Comment out the log settings line in the /etc/dpkg/dpkg.cfg configuration file. Set the DHCP lease time on your router to no less than one day.

Rotate Logs Less Frequently

The default logrotate configuration files tend to rotate logs daily or weekly, unnecessary on a low-usage server. Edit the /etc/logrotate.d/rsyslog configuration file and replace the “daily” and “weekly” directives with “monthly”, and update the various “rotate nn” options to “rotate 12”.

For your (and my) reference, here’s a copy of my refactored main syslog logrotate configuration file:

monthly
size 10m
rotate 12
missingok
notifempty
compress
delaycompress

/var/log/syslog {
	postrotate
		invoke-rc.d rsyslog rotate > /dev/null
	endscript
}

/var/log/mail.info /var/log/mail.warn /var/log/mail.err
/var/log/*.log /var/log/debug /var/log/messages {
	sharedscripts
	postrotate
		invoke-rc.d rsyslog rotate > /dev/null
	endscript
}

Use Temporary Filesystems

Older Raspbian distributions once seem to have enabled temporary filesystems for the /tmp/ folder, but the latest “Wheezy” has the feature disabled by default, only mounting /run/ and its subfolders as tmpfs filesystems. Reboot after re-enabling the capability.

sudo sed -i 's/\#\s*RAMTMP\s*=\s*no/RAMTMP=yes/i' /etc/default/tmpfs

Verify Improvements

Run the same iostat command as above again in order to compare before and after results. Hopefully you should see some improvement.

$ iostat -d 300 3
Linux 3.2.27+ (raspberrypi) 	2013/02/08 	_armv6l_	(1 CPU)

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.44         1.74         7.84     177045     795692

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.00         0.00         0.00          0          0

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
mmcblk0           0.03         0.00         0.24          0         72

A small sampling of data isn’t truly sufficient to gauge long-term effectiveness of these procedures. That said, according to the latest results above, only 72 kilobytes of data was written to disk during the entire ten-minute test, a reduction of approximately 70% as compared to the initial usage test results.

Given similar usage over an extended period of time, the longevity of the above-tested SD card could be extended by more than three times simply by eliminating memory swaps, uninstalling unneeded modules, and reducing a few unnecessary logs and their subsequent maintenance.