Categories
Linux

Setting up your own ebook library

This article is about setting up you own ebook library with support for the OPDS catalog format . The COPS ebook catalog can read a Calibre library and expose it as a website. It is developed on PHP.

Prerequisites

  • Prepare a Linux system. In this guide we are using Ubuntu 14.04.4.
  • Have a cops.example.com DNS A record pointing to the IP of your server.

Install necessary packages

$ sudo apt-get -y samba nginx php5-fpm php5-gd php5-sqlite php5-json php5-intl git

Setup a SAMBA share

Samba is needed for sharing the Calibre library directory with the library administrator. Alternatively you could use NFS or even OwnCloud.

Note

It is a security risk to expose Samba on a public server. Restrict it through firewall or use OwnCloud instead.

  • Tweak the Samba configuration file (/etc/samba/smb.conf):
    #  unix password sync = yes
       unix password sync = no
    
    #  pam password change = yes
       pam password change = no
    
  • Add a Samba share for calibre (/etc/samba/smb.conf):
    [calibre$]
    path = /srv/calibre
    valid users = calibre
    write list = calibre
    read only = no
    
  • Restart Samba services:
    $ sudo service smbd restart ; sudo service nmbd restart
    
  • Add the calibre user:
    $ useradd -m -d /srv/calibre -s /usr/sbin/nologin calibre
    $ smbpasswd -a calibre
    

Setup Calibre

  • Now you should setup Calibre on your PC. On Debian/Ubuntu:
    $ sudo apt-get -y install calibre
    

    Windows users can download it from here:

    http://calibre-ebook.com/download_windows

  • Configure Calibre to use the calibre share as its catalog:

    • You can "mount" the calibre share on Linux or "map" the share on Windows.
    • Create an ebook Directory under the calibre share.
    • Create a new Calibre library inside the ebook directory. From the Calibre menu:
      • Callibre Library –> Switch/create library -> Create an empty Library at the new location and fill the path in the "New Location" text field.
    • If you get an error about "Corrupted database" just click ‘Yes" to rebuilt it.
    • Start adding books

Setup COPS OPDS

  • Download the software:
    $ mkdir /var/www
    $ cd /var/www
    $ git clone https://github.com/seblucas/cops.git
    
  • Setup COPS:

    Under the /var/www/cops/ directory, copy the config_default.php.example to config_default.php:

    $ cp config_local.php.example config_local.php
    

    Make the following changes in config_default.php:

    < ?php
        if (!isset($config))
            $config = array();
    
        /*
         * The directory containing calibre's metadata.db file, with sub-directories
         * containing all the formats.
         * BEWARE : it has to end with a /
         */
        $config['calibre_directory'] = '/srv/calibre/ebooks/';
    
        /*
         * Catalog's title
         */
        $config['cops_title_default'] = 'My Ebook Portal';
    
        $config['calibre_internal_directory'] = '/ebooks/';
    
        $config['cops_full_url'] = 'cops.example.com';
    
        $config['cops_x_accel_redirect'] = 'X-Accel-Redirect';
    
        /*
         * use URL rewriting for downloading of ebook in HTML catalog
         * See README for more information
         *  1 : enable
         *  0 : disable
         */
        $config['cops_use_url_rewriting'] = '1';
    

Setup nginx

  • Create an /etc/nginx/sites-available/cops.example.com
    server {
    
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;
    
        server_name cops.example.com;
    
        access_log  /var/log/nginx/cops.access.log;
        error_log /var/log/nginx/cops.error.log;
        root   /var/www/cops;
        #index feed.php;
        index index.php;
    
        #Useful only for Kobo reader
        location /download/ {
              rewrite ^/download/(\d+)/(\d+)/.*\.(.*)$ /fetch.php?data=$1&db=$2&type=$3 last;
              rewrite ^/download/(\d+)/.*\.(.*)$ /fetch.php?data=$1&type=$2 last;
              break;
            }
    
            #Can break loading the images - if you don't see anything, comment
            location ~ ^/images.*\.(gif|png|ico|jpg)$ {
                    expires 31d;
            }
            #Can also break loading the images, comment if it happens
            location ~ .(js|css|eot|svg|woff|ttf)$ {
                    expires 31d;
            }
    
        #Not necessarily correct, it depends on distro.
        location ~ \.php$ {
           try_files $uri =404;
           include /etc/nginx/fastcgi_params;
           fastcgi_param   SCRIPT_FILENAME  $document_root$fastcgi_script_name;
           fastcgi_pass    unix:/run/php5-fpm.sock;
        }
    
        location /ebooks {
            root /srv/calibre;
            internal;
        }
    }
    

    Note:

    The feed.php setting was redirecting me to an XML site with this error: This XML file does not appear to have any style information associated with it. The document tree is shown below. So I am using index.php instead, as the index file.

  • Enable the cops.example.com site and disable the default:
    $ cd /etc/nginx/sites-enabled/
    $ sudo ln -s /etc/nginx/sites-available/cops.example.com
    $ sudo unlink default
    
  • Restart nginx and php5-fpm:
    $ sudo service php5-fpm restart ; sudo service nginx restart
    
  • Change permissions to let nginx write to the library:
    $ usermod -a -G calibre www-data
    $ chmod -R g+w /srv/calibre/ebooks/
    

Now you can navigate to http://cops.example.com and enjoy your newly created ebook library!

References

  • https://github.com/seblucas/cops
  • http://blog.slucas.fr/en/oss/calibre-opds-php-server
  • https://github.com/seblucas/cops/wiki/Full-example-with-Nginx
Categories
Linux

Setup a new Ubuntu or Debian system on the Utilite

This is a guide to setup a recent Ubuntu system on a Utilite ARM computer, to replace the official Ubuntu 12.04 OS.

Prerequisites

  • You need to prepare a [kernel with cgroups](/?p=1073 "Building a new Linux kernel for your Utilite Computer") support.
  • An ARM bootable microSD or USB. You can use the official Ubuntu 12.04 [Linux Utilite Image](http://www.compulab.co.il/utilite-computer/wiki/index.php/Utilite_Linux_Image "Linux Utilite Image") on the removable device.
  • Backup any data you have on the original system!

Preparations

  1. Boot from microSD or USB.
  2. Update the system and install debootstrap:

    $ sudo apt-get update
    $ sudo apt-get install debootstrap
    

  3. Mount the SATA root filesystem on /mnt:
    $ sudo mount /dev/sda2 /mnt
    
  4. Clear the filesystem:

    Note
    Make sure you save any data you need, before doing this!.

    $ sudo rm -fr /mnt/*
    

Installation of an Ubuntu or Debian base system

  1. Use debootstrap to install the Ubuntu base system:
    $ sudo debootstrap --foreign --arch=armhf trusty /mnt http://ports.ubuntu.com/
    

    You can use vivid instead of trusty if you are feeling adventurous.

    For a Debian system use this command instead:

    $ sudo debootstrap --foreign --arch armhf jessie /mnt http://ftp.uk.debian.org/debian
    

    You can use stretch or sid instead of jessie if you are feeling lucky.

  2. Chroot into the new system:

    $ sudo mount -o bind /dev /mnt/dev
    $ sudo mount -o bind /dev/pts /mnt/dev/pts
    $ sudo mount -t sysfs /sys /mnt/sys
    $ sudo mount -t proc /proc /mnt/proc
    $ sudo cp /proc/mounts /mnt/etc/mtab
    $ sudo chroot /mnt
    

  3. Second stage debootstrap:

    After you enter the chroot jail, we need to complete the second stage of the base setup:

    # /debootstrap/debootstrap --second-stage
    

Installing additional packages

  1. Setup the correct timezone:
    # dpkg-reconfigure tzdata
    
  2. Add repositories in /etc/apt/sources.list.
  • For Ubuntu:

    deb http://ports.ubuntu.com/ trusty main restricted universe multiverse
    deb http://ports.ubuntu.com/ trusty-security main restricted universe multiverse
    deb http://ports.ubuntu.com/ trusty-updates main restricted universe multiverse
    deb http://ports.ubuntu.com/ trusty-backports main restricted universe multiverse
    
    deb-src http://ports.ubuntu.com/ trusty main restricted universe multiverse
    deb-src http://ports.ubuntu.com/ trusty-security main restricted universe multiverse
    deb-src http://ports.ubuntu.com/ trusty-updates main restricted universe multiverse
    deb-src http://ports.ubuntu.com/ trusty-backports main restricted universe multiverse
    

  • For Debian:
    deb http://ftp.uk.debian.org/debian jessie main contrib non-free
    deb http://ftp.debian.org/debian/ jessie-updates main contrib non-free
    deb http://security.debian.org/ jessie/updates main contrib non-free
    
    deb-src http://ftp.uk.debian.org/debian jessie main contrib non-free
    deb-src http://ftp.debian.org/debian/ jessie-updates main contrib non-free
    deb-src http://security.debian.org/ jessie/updates main contrib non-free
    
  • Run update after adding the repositories:
    # apt-get update
    
  1. Install additional software.
  • We need to create a fake /sbin/initctl to prevent apt from breaking:
    # dpkg-divert --local --rename --add /sbin/initctl
    # ln -s /bin/true /sbin/initctl
    
  • Install the software you need:
    # apt-get -y install language-pack-en ssh isc-dhcp-client net-tools man lsof less
    
  • Remove the fake /sbin/initctl:
    # rm /sbin/initctl
    # dpkg-divert --local --rename --remove /sbin/initctl
    
  1. Setup users:
    # passwd root
    # useradd -m -s /bin/bash utilite
    # passwd utilite
    # usermod -a -G adm,cdrom,sudo,dip,plugdev utilite
    
  2. Setup a kernel:

    Now you need to deploy the [kernel we prepare earlier](/?p=1073#toc-deploy-the-tarball "Building a new Linux kernel for your Utilite Computer") and reboot.

Choose a role for your system

  1. After you reboot you may find that networking is not working. If you are in a dhcp enabled network run dhclient to get an IP address:

    # dhclient
    

  2. Upgrade the system to the latest upgrades:
    # apt-get update
    # apt-get -y dist-upgrade
    
  3. Check the available roles
  • On Ubuntu you may want to install tasksel (already installed on Debian):

    # apt-get install tasksel

  • See the available roles on Ubuntu:

    # tasksel --list-tasks
    i server Basic Ubuntu server
    i openssh-server OpenSSH server
    u dns-server DNS server
    u lamp-server LAMP server
    u mail-server Mail server
    u postgresql-server PostgreSQL database
    u print-server Print server
    u samba-server Samba file server
    u tomcat-server Tomcat Java server
    u cloud-image Ubuntu Cloud Image (instance)
    u virt-host Virtual Machine host
    u ubuntustudio-graphics 2D/3D creation and editing suite
    u ubuntustudio-audio Audio recording and editing suite
    u edubuntu-desktop-gnome Edubuntu desktop
    u kubuntu-active Kubuntu Active
    u kubuntu-desktop Kubuntu desktop
    u kubuntu-full Kubuntu full
    u ubuntustudio-font-meta Large selection of font packages
    u lubuntu-desktop Lubuntu Desktop
    u lubuntu-core Lubuntu minimal installation
    u mythbuntu-frontend Mythbuntu frontend
    u mythbuntu-backend-master Mythbuntu master backend
    u mythbuntu-backend-slave Mythbuntu slave backend
    u ubuntustudio-photography Photograph touchup and editing suite
    u ubuntustudio-publishing Publishing applications
    u ubuntu-gnome-desktop Ubuntu GNOME desktop
    u ubuntu-desktop Ubuntu desktop
    u ubuntu-usb Ubuntu desktop USB
    u ubuntustudio-video Video creation and editing suite
    u xubuntu-desktop Xubuntu desktop
    u edubuntu-dvd-live Edubuntu live DVD
    u kubuntu-active-live Kubuntu Active Remix live CD
    u kubuntu-live Kubuntu live CD
    u kubuntu-dvd-live Kubuntu live DVD
    u lubuntu-live Lubuntu live CD
    u ubuntu-gnome-live Ubuntu GNOME live CD
    u ubuntustudio-dvd-live Ubuntu Studio live DVD
    u ubuntu-live Ubuntu live CD
    u ubuntu-usb-live Ubuntu live USB
    u xubuntu-live Xubuntu live CD
    u manual Manual package selection

  • See the available roles on debian:

    $ tasksel --list-tasks
    i desktop   Debian desktop environment
    u gnome-desktop GNOME
    u xfce-desktop  Xfce
    u kde-desktop   KDE
    u cinnamon-desktop  Cinnamon
    u mate-desktop  MATE
    i lxde-desktop  LXDE
    u web-server    web server
    u print-server  print server
    u ssh-server    SSH server
    u laptop    laptop
    

  1. Choose a role.

    If you would like a Desktop system, I suggest you choose something light like lubuntu-desktop or lxde-desktop.

  • Installing Lubuntu (Ubuntu):

    # tasksel --task-packages lubuntu-desktop | xargs apt-get -y install
    

    I prefer this method rather than the [TUI](https://en.wikipedia.org/wiki/Text-based_user_interface "Text-based User Interface") of tasksel, so as to have better overview and control of the process. Keep an eye on it every now and then, because it will ask you about keyboard-configuration and other questions. Select the default settings if you are unsure.

  • Installing LXDE Desktop (Debian):

    # apt-get install task-lxde-desktop network-manager
    

    For Desktop systems it may be a good idea to install network-manager as well.

  • Add the vivante GPU module on startup:

    # echo vivante >> /etc/modules
    

After you finish you can reboot your system and start playing.

References

  • https://github.com/umiddelb/armhf/wiki/Installing-Ubuntu-14.04-on-the-utilite-computer-from-scratch
  • https://wiki.debian.org/InstallingDebianOn/CompuLab/PC-Utilite/wheezy
Categories
Linux

Shrink a Linux partition to create Swap space

I have mentioned in an earlier post that I am an owner of a Utilite ARM computer. One of the issues I experienced with the default setup, is that there is no swap partition. That causes Firefox some memory hungry applications to perform badly.

Check for yourself, if swap is activated on your machine:

$ free
            total       used       free     shared    buffers     cached
Mem:       2006476     263752    1742724      20100      21428     107388
-/+ buffers/cache:     134936    1871540
Swap:            0          0          0

As you can see in the output, no swap space is used. So we are going to see how we can resize a linux ext2/ext3/ext4 partition to make space for a swap partition. We are then going to create a new swap partition and activate it on boot.

Prerequisites

Checking the current setup

After you boot using a removable medium, you can follow this procedure to check the current setup:

$ sudo fdisk -l /dev/sda

Disk /dev/sda: 29.8 GiB, 32017047552 bytes, 62533296 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5161f73

Device     Boot  Start      End  Sectors  Size Id Type
/dev/sda1  *      2048   206847   204800  100M  c W95 FAT32 (LBA)
/dev/sda2       206848 62533295 62326448 29.7G 83 Linux
  • START_SECTOR = 206848
  • END_SECTOR = 62533295
  • TOTAL_SECTORS = 62326448

As you can see, the main root partition (sda2) has a size of 29.7GB. We will reduce it to 28GB and you the rest of the space (1.7GB) for swap. That should be more than enough.

Now in order to avoid tedious calculations, and make sure the filesystem on sda2 takes all available space on the partition we are going to use this trick:

  • Resize the sda2 filesystem to 27GB.
  • Resize the sda2 partition to 28GB.
  • Resize the sda2 filesystem to all available space of the sda2 partition (28GB).

Temporarily resize sda2

  1. First force check the filesystem for errors:
    $ sudo e2fsck -f /dev/sda2
    

    If it finishes without errors move to the next step

  2. Resize the filesystem to 27GB:

    $ sudo resize2fs /dev/sda2 27G
    

  3. Verify the result:
    $ sudo dumpe2fs  /dev/sda2 | grep Block | head -2
    dumpe2fs 1.42.12 (29-Aug-2014)
    Block count:              7077888
    Block size:               4096
    
    $ echo '(7077888 * 4096)/1024/1024/1024' | bc -l
    27.00000000000000000000
    

    Exactly 27GB.

Resize the sda2 partition

To resize the sda2 partition we will delete it temporarily and recreate it using the same START_SECTOR but a different END_SECTOR.

  1. To calculate the new end sector you can use this formula:

    NEW_END_SECTOR = START_SECTOR + TOTAL_SECTORS * 28 / 29.7

    $ echo "206848+62326448*28/29.7" | bc -l
    58965788.87542087542087542087
    

    We can round up the NEW_END_SECTOR to 58965789.

  • Use fdisk to resize the partition:

    • Delete the sda2 partition:

      $ sudo fdisk /dev/sda
      Changes will remain in memory only, until you decide to write them.
      Be careful before using the write command.
      
        Command (m for help): d
           Partition number (1,2, default 2): 2
      
           Partition 2 has been deleted.
      

      Verify that sda2 is deleted:

      Command (m for help): p
      Disk /dev/sda: 29,8 GiB, 32017047552 bytes, 62533296 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0xa5161f73
      
      Device     Boot Start    End Sectors  Size Id Type
      /dev/sda1  *     2048 206847  204800  100M  c W95 FAT32 (LBA)
      

      It is deleted, but not really until you issue the write command.

    • Create a smaller sda2 partition:

      Command (m for help): n
      Partition type
         p   primary (1 primary, 0 extended, 3 free)
         e   extended (container for logical partitions)
      Select (default p): p
      Partition number (2-4, default 2): 
      First sector (206848-62533295, default 206848): 
      Last sector, +sectors or +size{K,M,G,T,P} (206848-62533295, default 62533295): 58965789
      
      Created a new partition 2 of type 'Linux' and of size 28 GiB.      
      

      For the end sector we use the value of the NEW_END_SECTOR we have calculated earlier. Now let’s verify the new partition:

      Command (m for help): p
      Disk /dev/sda: 29,8 GiB, 32017047552 bytes, 62533296 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0xa5161f73
      
      Device     Boot  Start      End  Sectors  Size Id Type
      /dev/sda1  *      2048   206847   204800  100M  c W95 FAT32 (LBA)
      /dev/sda2       206848 58965789 58758942   28G 83 Linux
      

    • Create a new partition of type swap:
      Command (m for help): n
      Partition type
         p   primary (2 primary, 0 extended, 2 free)
         e   extended (container for logical partitions)
      Select (default p): p
      Partition number (3,4, default 3): 
      First sector (58965790-62533295, default 58966016): 
      Last sector, +sectors or +size{K,M,G,T,P} (58966016-62533295, default 62533295):
      
      Created a new partition 3 of type 'Linux' and of size 1,7 GiB.
      
      Command (m for help): t
      Partition number (1-3, default 3): 3
      Partition type (type L to list all types): 82
      
      Changed type of partition 'Linux' to 'Linux swap / Solaris'.
      

      Verify that the new partition is created:

      Command (m for help): p
      Disk /dev/sda: 29,8 GiB, 32017047552 bytes, 62533296 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0xa5161f73
      
      Device     Boot    Start      End  Sectors  Size Id Type
      /dev/sda1  *        2048   206847   204800  100M  c W95 FAT32 (LBA)
      /dev/sda2         206848 58965789 58758942   28G 83 Linux
      /dev/sda3       58966016 62533295  3567280  1,7G 82 Linux swap / Solaris
      

      Seems OK.

    • Write changes and exit:

      Command (m for help): w
      The partition table has been altered.
      Calling ioctl() to re-read partition table.
      

      Remember that nothing is changed unless you issue the write command. If you make a mistake, just hit Ctrl+c and no harm done.

    1. Resize the filesystem to the full size of the partition:
    • Force check the filesystem

      $ sudo e2fsck -f /dev/sda2
      

    • Resize the filesystem to occupy all the available space in the partition:
      $ sudo resize2fs /dev/sda2
      

      Running resize2fs without a size definition, extends the size of the filesystem to the size of the partition.

    • Verify the new filesystem size:

      $ sudo dumpe2fs  /dev/sda2 | grep Block | head -2
      dumpe2fs 1.42.13 (17-May-2015)
      Block count:              7344867
      Block size:               4096
      

      The size in bytes is BLOCK_COUNT * BLOCK_SIZE i.e. 7344867 * 4096 = 30084575232. The size in GB is:

      $ echo '(7344867 * 4096)/1024/1024/1024' | bc -l
      28.01844406127929687500
      

    Restart into your internal drive

    At this point we need to restart into the Operating System installed on the internal SATA device and check if everything works:

    $ sudo fdisk -l /dev/sda
    Disk /dev/sda: 29,8 GiB, 32017047552 bytes, 62533296 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: dos
    Disk identifier: 0xa5161f73
    
    Device     Boot    Start      End  Sectors  Size Id Type
    /dev/sda1  *        2048   206847   204800  100M  c W95 FAT32 (LBA)
    /dev/sda2         206848 58965789 58758942   28G 83 Linux
    /dev/sda3       58966016 62533295  3567280  1,7G 82 Linux swap / Solaris
    

    The partition table appears to be correct. But we haven’t formatted the swap partition yet:

    $ free
                total       used       free     shared    buffers     cached
    Mem:       2006476     264380    1742096      20100      21568     107800
    -/+ buffers/cache:     135012    1871464
    Swap:            0          0          0
    

    We need to format the swap partition before we can use it.

    Setup the swap partition

    1. Format the swap partition:
      $ sudo mkswap /dev/sda3
      mkswap: /dev/sda3: warning: wiping old swap signature.
      Setting up swapspace version 1, size = 1,7 GiB (1826443264 bytes)
      no label, UUID=fa1e99ff-a9ab-4fd7-ba81-e5020f4e604b
      

      Take a note of the UUID: fa1e99ff-a9ab-4fd7-ba81-e5020f4e604b

    2. Activate swap:

      $ sudo swapon /dev/sda3
      

    3. Verify swap (in human readable form this time):
      $ free -m
                  total       used       free     shared    buffers     cached
      Mem:          1959        259       1700         19         21        105
      -/+ buffers/cache:        132       1826
      Swap:         1741          0       1741
      

      So we have a total of 1741MB of swap space.

    4. Setup persistent swap.

      To have swap activated persistently across reboots, we need to set it up in /etc/fstab:

      # echo 'UUID="fa1e99ff-a9ab-4fd7-ba81-e5020f4e604b" none swap defaults 0 0' >> /etc/fstab
      

      You need to run the above command as root.

    Now after you reboot you can check with free to see if the swap space is activated.

    Categories
    Linux

    Creating a new Debian package using pbuilder: gnujump

    Yet another practical packaging session, this ine on how to build a Debian package from scratch. This one too is based on the packaging tutorial and practical sessions of Lucas Nussbaum.

    We are going to download the source code of gnujump from upstream, and use the pbuilder tool to package it. The advantages of pbuider is that it creates a minimal chroot jail that helps you track any dependencies you might forgot to specify in debian/control. Also it gives you the opportunity to build Debian packages on Ubuntu and vice-versa. You can also build packages for releases other than your own.

    Prerequisites

    • A recent Debian or Ubuntu system.

    Preparation of the pbuilder jail

    Install pbuilder and relevant packages:

    $ sudo apt-get -y install pbuilder debootstrap devscripts packaging-dev debian-keyring ubuntu-archive-keyring
    

    NOTE: use ubuntu-keyring instead of ubuntu-archive-keyring if your build station is Ubuntu

    Prepare the target environment. For example for Ubuntu trusty use can use this setup:

    $ sudo pbuilder create --debootstrapopts --variant=buildd --mirror http://cy.archive.ubuntu.com/ubuntu --distribution trusty --architecture amd64 --components main --debbuildopts -mJohn Doe <john.doe@example.net>
    

    Prepare the new package

    1. Download the source:
      $ wget https://ftp.gnu.org/gnu/gnujump/gnujump-1.0.8.tar.gz
      
    2. Extract the archive and change into the source tree:
      $ tar xvzf gnujump-1.0.8.tar.gz
      $ cd gnujump-1.0.8/
      

    Prepare for debian packaging

    1. We are going to use the dh_make utility to prepare the debian directory and all the necessary files:
      $ DEBFULLNAME="John Doe" DEBEMAIL="john.doe@example.net" dh_make -s -y --createorig
      

      Check if all necessary files are there:

      $ find debian/
      debian/
      debian/compat
       debian/manpage.1.ex
      debian/preinst.ex
      debian/rules
      debian/gnujump.default.ex
      debian/README.Debian
      debian/copyright
      debian/gnujump.cron.d.ex
      debian/gnujump.doc-base.EX
      debian/changelog
      debian/README.source
      debian/control
      debian/menu.ex
      debian/manpage.sgml.ex
      debian/docs
      debian/init.d.ex
      debian/source
      debian/source/format
      debian/watch.ex
      debian/postrm.ex
      debian/prerm.ex
      debian/manpage.xml.ex
      debian/postinst.ex
      

      Now Check the contents of the debian/changelog, debian/rules and debian/control files.

    2. We will need to make some changes in the debian/control file:

      Source: gnujump
      Section: games
      Priority: optional
      Maintainer: John Doe <john .doe@example.net>
      Build-Depends: debhelper (>= 9), autotools-dev, libsdl1.2-dev, libsdl-image1.2-dev, libsdl-mixer1.2-dev
      Standards-Version: 3.9.5
      Homepage: http://gnujump.es.gnu.org
      ...
      

      The libsdl1.2-dev, libsdl-image1.2-dev and libsdl-mixer1.2-dev have been discovered by repeatedly compiling and failing until you get it right. The you can use the apt-cache search and apt-file search commands to discover the packages corresponding to the missing dependencies.

    3. Edit the debian/rules file to look like this:

      DH_VERBOSE = 1
      
      DPKG_EXPORT_BUILDFLAGS = 1
      include /usr/share/dpkg/default.mk
      
      export DEB_BUILD_MAINT_OPTIONS = hardening=+all
      
      export DEB_CFLAGS_MAINT_APPEND  = -Wall -pedantic
      export DEB_LDFLAGS_MAINT_APPEND = -Wl,--as-needed
      
      %:
              dh $@  --with autotools-dev
      
      CFLAGS += -lm
      

    4. Edit the debian/changelog, with dch -e, to look like this:
      gnujump (1.0.8-1) unstable; urgency=low
      
        * Initial release
      
       -- John Doe</john><john .doe@example.net>  Mon, 22 Jun 2015 17:41:22 +0300
      

    Build the package

    Run this command from the source tree:

    $ pdebuild
    

    Give your password when asked. pdebuild will download and install all dependencies in the pbuilder jail and then build the package. Changes will be lost from the jail, the next time you use it but if all goes well, you should see these files under /var/cache/pbuilder/result:

    $ ls -la /var/cache/pbuilder/result/
    total 5776
    drwxr-xr-x 2 root      root         4096 Ιούν 22 17:52 .
    drwxr-xr-x 9 root      root         4096 Ιούν 22 11:26 ..
    -rw-r--r-- 1 john john    1449 Ιούν 22 17:52 gnujump_1.0.8-1_amd64.changes
    -rw-r--r-- 1 john john 1560574 Ιούν 22 17:52 <strong>gnujump_1.0.8-1_amd64.deb</strong>
    -rw-rw-r-- 1 john john    9228 Ιούν 22 17:52 gnujump_1.0.8-1.debian.tar.xz
    -rw-rw-r-- 1 john john     885 Ιούν 22 17:52 gnujump_1.0.8-1.dsc
    -rw-rw-r-- 1 john john 2508641 Ιούλ 24  2012 gnujump_1.0.8.orig.tar.gz
    -rw-rw-r-- 1 john john 1814056 Ιούν 22 17:41 gnujump_1.0.8.orig.tar.xz
    

    Checking the result

    1. View the information related to the package file:
      $ dpkg -I /var/cache/pbuilder/result/gnujump_1.0.8-1_amd64.deb
      new debian package, version 2.0.
      size 1560574 bytes: control archive=5550 bytes.
          419 bytes,    11 lines      control              
        18390 bytes,   233 lines      md5sums              
      Package: gnujump
      Version: 1.0.8-1
      Architecture: amd64
      Maintainer: John Doe </john><john .doe@example.net>
      Installed-Size: 2355
      Depends: libc6 (>= 2.14), libgl1-mesa-glx | libgl1, libsdl-image1.2 (>= 1.2.10), libsdl-mixer1.2, libsdl1.2debian (>= 1.2.11)
      Section: games
      Priority: optional
      Homepage: http://gnujump.es.gnu.org
      Description: <insert up to 60 chars description>
       </insert><insert long description, indented with spaces>
      

    As you can see our work is not finished. We need to add a description in debian/control and rebuild it.

    1. Check the contents of the package file:
      $ dpkg -I /var/cache/pbuilder/result/gnujump_1.0.8-1_amd64.deb
      

    Install the package

    Now we can install the package:

    $ sudo dpkg -i /var/cache/pbuilder/result/gnujump_1.0.8-1_amd64.deb
    

    Now run the gnujump software and check if it works.

    This is not the whole story of course. You have to check the package against Lintian to see if it complies with the Debian Policy. And if it’s not compliant, you may need to use Quilt to patch it. You will need to fill debian/watch so you can track updates on the upstream with uscan. The work of a Debian packager never ends.

    References

    • https://wiki.ubuntu.com/PbuilderHowto
    • https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf</insert></john>
    Categories
    Linux

    Modifying an existing Debian package: grep

    This guide is a practical session on how to modify an existing debian package, when the version is not included in the current release. It is based on the excellent packaging tutorial and practical sessions of Lucas Nussbaum.

    We are going to download a source deb package, build it without modifications, modify it, patch it, rebuild it and in the end, install it in our system.

    Prerequisites

    • A recent Debian or Ubuntu system.

    Installation of development packages

    $ sudo apt-get -y install build-essential debhelper devscripts packaging-dev debian-keyring
    

    Get the source from the debian repos

    $ dget http://cdn.debian.net/debian/pool/main/g/grep/grep_2.21-1.dsc
    

    After that you should see these files under the current directory:

    $ ls
    grep_2.21-1.debian.tar.bz2  grep_2.21-1.dsc  grep_2.21.orig.tar.xz
    
    • grep_2.21-1.dsc: Package’s debian source control file.
    • grep_2.21.orig.tar.xz: Original source.
    • grep_2.21-1.debian.tar.bz2: Debian files and patches.

    Unpack the source

    The dpkg-source file will unpack the original and debian tarballs and apply the debian patches, if available.

    $ dpkg-source -x grep_2.21-1.dsc
    

    Download build dependencies

    $ sudo apt-get -y build-dep grep
    

    Build the unmodified package

    Change into the source tree and run debuild. We are using the -us and -uc flags to suppress warnings about signing.

    $ cd grep-2.21/
    $ debuild -us -uc
    

    It doesn’t take long before this package is build. In the parent directory you will find three more files:

    • grep_2.21-1_amd64.deb: the newly created deb package file.
    • grep_2.21-1_amd64.build: the build log. You can use this to check for errors or warnings.
    • grep_2.21-1_amd64.changes: Debian applied changes.

    Modify the package

    I suggest you do not touch the source, unless you fancy spending your day debugging. We will simply make some modifications on the debian/changelog and debian/rules files.

    1. Add the --with-gnu-ld flag in the DEB_CONFIGURE_EXTRA_FLAGS line of the debian/rules file.

      The --with-gnu-ld is a trivial change and shouldn’t break anything during build. Find this line:

      DEB_CONFIGURE_EXTRA_FLAGS += --without-included-regex
      

      And change it to this:

      DEB_CONFIGURE_EXTRA_FLAGS += --without-included-regex --with-gnu-ld
      
    2. Update the changelog file.

      We are going to use the dch utility which is simply a wrapper around your default editor, with changelog syntax checking. Run the following command in the source tree:

      $ DEBFULLNAME="John Doe" DEBEMAIL="john.doe@example.net" dch -i
      

      Make the following changes (in bold) the debian/changelog file:

      grep (2.21-1ubuntu1) experimental; urgency=low
      
       * Support for GNU ld linker.
      
      -- John Doe <john .doe@example.net>  Fri, 19 Jun 2015 13:01:37 +0300
      

      Note how the name and email of the patcher have been automatically inserted in the changelog entry. That’s because we defined the DEBFULLNAME and DEBEMAIL variables in the shell that runs dch.

    Rebuild the modified package

    $ debuild -us -uc
    

    You will see the following error:

    dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/grep_2.21-1ubuntu1.diff.eYCcPk
    dpkg-source: info: you can integrate the local changes with dpkg-source --commit
    dpkg-buildpackage: error: dpkg-source -b grep-2.21 gave error exit status 2
    debuild: fatal error at line 1376:
    

    The above error happens because the --with-gnu-ld flag makes changes to the upstream source and this is against the Debian policy.

    Apply patches for Policy compliance

    $ cp /tmp/grep_2.21-1ubuntu1.diff.eYCcPk debian/patches/90-enable-gnu-ld.patch
    $ echo 90-enable-gnu-ld.patch >> debian/patches/series
    

    It is considered a good practice to edit the patch, add a description and a short summary and fill the headers related to the [Patch Tagging Guidelines](http://dep.debian.net/deps/dep3/ "Patch Tagging Guidelines").

    This method is only a workaround. The correct way to deal with patches is [Quilt](https://pkg-perl.alioth.debian.org/howto/quilt.html "Quilt Howto").

    Final rebuild of the modified package

    $ debuild -us -uc
    

    You will see five additional files in the parent directory:

    • grep_2.21-1ubuntu1_amd64.deb: the new deb package file.
    • grep_2.21-1ubuntu1.debian.tar.bz2: the new debian directory tarball.
    • grep_2.21-1ubuntu1.dsc: the new debian source control file.
    • grep_2.21-1ubuntu1_amd64.build: the build log.
    • grep_2.21-1ubuntu1_amd64.changes: the changes file.

    Checking the differences between original and new

    The following commands will find the differences in the debian source control and changes files:

    $ diff ../*.changes
    $ diff ../*dsc
    

    Install the new package

    $ sudo debi
    

    Check if grep is the correct version:

    $ grep --version
    grep (GNU grep) <strong>2.21</strong>
    Copyright (C) 2014 Free Software Foundation, Inc.
    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
    This is free software: you are free to change and redistribute it.
    There is NO WARRANTY, to the extent permitted by law.
    
    Written by Mike Haertel and others, see <http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
    

    All done. If you have reached so far congratulations! You have successfully rebuilt a Debian package.

    References:

    • http://www.lucas-nussbaum.net/
    • https://www.debian.org/doc/manuals/packaging-tutorial/packaging-tutorial.en.pdf
    • http://dep.debian.net/deps/dep3/</john>
    Categories
    Linux

    Adding USB boot support on the Utilite nettop

    More than a year or so, I have been the proud owner of a first generation Utilite ARM appliance. But the harsh Cyprus summer, hit it hard and the microSD circuit stop working. I could no longer boot alternative systems on microSD cards. All I was left with, was the stock system with an aging Ubuntu 12.04 ARM port.

    Enter the U-Boot

    Das U-Boot is a boot loader that targets mostly embedded systems. Many ARM appliances, including the Utilite, are using U-Boot to boot their OS.

    I entered the U-Boot environment to do some checks about the mmc issue. To do that just press the ‘any‘ key when you see the following message:

    Hit any key to stop autoboot:
    

    Then type the following command in your U-boot terminal:

    CM-FX6 # mmc info
    Card did not respond to voltage select!
    

    That message does not look very healthy! When I tried to insert a microSD card, I got this message repeatedly, in the U-Boot terminal:

    EHCI timed out on TD - token=0x80008c80
    

    The keyboard stopped responding and I had no other option than to force restart. I think now we have enough evidence to say that the microSD circuitry went FUBAR. At this point you should probably contact CompuLab for a replacement.

    But this is not what I did. The appliance has 3 USB ports available, so why not use them? This will have the added benefit of USB devices being more accessible, than the hard to access microSD slot.

    Well, the original U-Boot image that the Utilite was shipped with, did not had USB boot support. This has long changed and the nice folks at CompuLab have built a newer version that supports booting from USB storage devices.

    Upgrading U-Boot

    First download the most recent U-Boot updater on your Utilite appliance:

    $ wget http://utilite-computer.com/download/utilite/u-boot/utilite-updater.tar.bz2
    

    Check the MD5 sum:

    $ echo a79e492f3eb626c770c5185cda0edfec ; md5sum utilite-updater.tar.bz2 
    a79e492f3eb626c770c5185cda0edfec
    a79e492f3eb626c770c5185cda0edfec  utilite-updater.tar.bz2
    

    Extract the archive:

    $ tar xvjf utilite-updater.tar.bz2
    

    Run the updater:

    $ sudo ./utilite-bootloader-update.sh 
    
    CompuLab CM-FX6 (Utilite) boot loader update utility 2.2 (Feb 8 2015)
    
    >> Checking for utilities... 
    >> ...Done 
    >> Checking that board is CM-FX6 (Utilite)... 
    >> ...Done 
    Please input firmware file path (or press ENTER to use "cm-fx6-firmware"): 
    

    Just press ‘Enter’ after the above prompt. Then answer ‘Yes’ (y) to the following questions:

    >> Looking for boot loader image file: cm-fx6-firmware 
    >> ...Found 
    >> Looking for SPI flash: mtd0 
    >> ...Found 
    >> Current U-Boot version in SPI flash: U-Boot 2014.10-cm-fx6-2.1 (Jan 19 2015 - 11:28:10) 
    >> New U-Boot version in file:      U-Boot 2014.10-cm-fx6-2.1 (Jan 19 2015 - 11:28:10) (500K) 
    >> Proceed with the update? 
    1) Yes
    2) No
    #? <strong>y</strong>  
    ** Do not power off or reset your computer!!! 
    >> Erasing SPI flash... 
    Erasing 4 Kibyte @ bf000 -- 100 % complete 
    >> ...Done 
    >> Writing boot loader to the SPI flash... 
    .........................
    >> ...Done 
    >> Checking boot loader in the SPI flash... 
    .
    >> ...Done 
    >> Boot loader update succeeded!
    
    ** Resetting U-Boot environment will override any changes made to the environment! 
    >> Reset U-Boot environment (recommended)? 
    1) Yes
    2) No
    #? y
    >> U-boot environment will be reset on restart. 
    >> Done!
    

    Then reboot the appliance:

    $ sudo reboot
    

    It is a good idea to keep a backup of the original U-Boot configuration for future reference:

    $ sudo fw_printenv > utilite.uboot.orig
    

    Enhancing the USB boot setup

    If you study the above U-Boot setup you will find that the USB config works only if you have an option boot.scr script under the first partition of your USB. This excludes USB drives with only the uImage file. That means that you cannot even boot the Utilite SSD installer if you burn it on USB, since it does not have a boot.scr script.

    I’ve been playing around with the U-Boot environment and I came out to this configuration:

    setenv bootcmd run setupmmcboot;mmc dev ${storagedev};if mmc rescan; then run trybootsrz;fi;run setupusbboot;if usb start; then if run loadscript; then run bootscript;else run usbbootargs;if run loadkernel; then run doboot;else setenv bootargs;fi;fi;fi; run setupsataboot;if sata init; then run trybootsmz;fi;run setupnandboot;run nandboot;
    setenv usbroot /dev/sdb2
    setenv usbrootdelay=1
    setenv usbbootargs=setenv bootargs root=${usbroot} rootdelay=${usbrootdelay}
    

    If you don’t want to configure the above commands manually, you can download my custom U-boot environment updater and the patch I prepared:

    Download the U-Boot environment update shell script:

    $ wget https://raw.githubusercontent.com/theodotos/arena/master/u-boot/u-boot-update-env.sh
    $ chmox +x u-boot-update-env.sh
    

    Download the U-boot custom configuration:

    $ wget https://raw.githubusercontent.com/theodotos/arena/master/u-boot/setup-usb-boot-utilite.uboot
    

    Now load the custom configuration into the U-Boot environment:

    $ sudo ./u-boot-update-env.sh setup-usb-boot-utilite.uboot
    

    If something goes wrong you can restore your original configuration using the original configuration we saved earlier:

    $ sudo ./u-boot-update-env.sh utilite.uboot.orig
    

    The U-Boot environment updater will also create a backup file, with the current U-Boot environment configuration, just before the update:

    $ ls *.bak
    u-boot_env-3366.bak
    

    Test with the Utilite SSD installer

    Now it is time to test our new configuration.

    First download the SSD installer:

    $ wget http://utilite-computer.com/download/utilite/installer/cl-installer_utilite-2_kernel-6.3_2014-12-17.img.xz
    

    Check the MD5 sum:

    $ echo 82eeb54c4d5245c60fd82c3e983d10e9 ; md5sum cl-installer_utilite-2_kernel-6.3_2014-12-17.img.xz
    82eeb54c4d5245c60fd82c3e983d10e9
    82eeb54c4d5245c60fd82c3e983d10e9  cl-installer_utilite-2_kernel-6.3_2014-12-17.img.xz
    

    Extract it:

    $ unxz cl-installer_utilite-2_kernel-6.3_2014-12-17.img.xz
    

    This will extract a cl-installer_utilite-2_kernel-6.3_2014-12-17.img image file in the working directory.

    Now comes the tricky part. We will load the image in a USB device. Make sure you choose the correct device! The dd tool we are using below, is a heartless beast that will chew the data out of every device you select in its of= flag. If by mistake you give your internal disk instead of the USB, you will lose your partitions and data! You have been warned!

    So after you select a USB (one that does not have any data you need) insert it in your Linux workstation. Then try the following check to find out the device name of the USB drive:

    $ dmesg | tail -n20
    [61425.197050] mce: [Hardware Error]: Machine check events logged
    [71124.693498] compiz[1853]: segfault at 80000000 ip 0000000080000000 sp 00007ffc438c50a8 error 14
    [73897.971544] usb 1-2: new high-speed USB device number 10 using xhci_hcd
    [73898.105247] usb 1-2: New USB device found, idVendor=0951, idProduct=1665
    [73898.105254] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
    [73898.105257] usb 1-2: Product: DataTraveler 2.0
    [73898.105260] usb 1-2: Manufacturer: Kingston
    [73898.105262] usb 1-2: SerialNumber: 50E54951351BBE70A9455C5B
    [73898.106083] usb-storage 1-2:1.0: USB Mass Storage device detected
    [73898.106370] scsi host8: usb-storage 1-2:1.0
    [73899.187591] scsi 8:0:0:0: Direct-Access     Kingston DataTraveler 2.0 PMAP PQ: 0 ANSI: 6
    [73899.187848] sd 8:0:0:0: Attached scsi generic sg2 type 0
    [73900.521857] sd 8:0:0:0: [sdb] 15335424 512-byte logical blocks: (7.85 GB/7.31 GiB)
    [73900.522528] sd 8:0:0:0: [sdb] Write Protect is off
    [73900.522531] sd 8:0:0:0: [sdb] Mode Sense: 23 00 00 00
    [73900.523124] sd 8:0:0:0: [sdb] No Caching mode page found
    [73900.523126] sd 8:0:0:0: [sdb] Assuming drive cache: write through
    [73900.562834]  sdb: sdb1 sdb2
    [73900.565248] sd 8:0:0:0: [sdb] Attached SCSI removable disk
    [73903.203134] EXT4-fs (sdb2): mounted filesystem with ordered data mode. Opts: (null)
    

    So the device name is sdb and it has two partitions: sda1 and sda2. Check if these partitions have been mounted automatically:

    $ mount | grep sdb
    /dev/sdb2 on /media/theodotos/rootfs type ext4 (rw,nosuid,nodev,relatime,data=ordered,uhelper=udisks2)
    /dev/sdb1 on /media/theodotos/boot type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)
    

    They are mounted. We need to unmount them before dumping the image on the device:

    $ sudo umount /dev/sdb1 /dev/sdb2
    

    Now we are ready to run dd (device dump) against the USB drive:

    $ sudo dd if=~/Downloads/cl-installer_utilite-2_kernel-6.3_2014-12-17.img of=/dev/sdb bs=1M
    

    You are not going to see any progress bar during the device dump. Just be patient and let the utility take its time.

    After dd finishes and returns back to the shell, safely remove the USB drive and try it on Utilite. If you reboot the appliance you should see the LXDE desktop of the Utilite installer. Unplug it and you are back to your Ubuntu, on the internal drive.

    References

    • http://www.compulab.co.il/utilite-computer/web/utilite-overview
    • http://www.compulab.co.il/utilite-computer/wiki/index.php/Utilite_U-Boot
    Categories
    Linux

    How to Join An Ubuntu Desktop Into An Active Directory Domain

    As a Free Software activist I wish I didn’t had to use, touch, see, hear, smell or taste Microsoft technologies. But until that happens, I need to know how to join a Linux machine in an Active Directory Domain.

    This is an article I recently submitted to the excellent Unixmen.com website:

    How to Join An Ubuntu Desktop Into An Active Directory Domain

    Categories
    Linux

    Introduction to the KVM Hypervisor (for Ubuntu)

    The KVM hypervisor is a virtualization system included with the Linux kernel. Along with XEN is one of the most attractive virtualization platforms based on Linux.

    KVM offers several advantages over the more user-friendly, VirtualBox. Since it is integrated into the mainstream Linux kernel, it boasts significant performance benefits [1]. Furthermore it is better suited as a virtualization platform solution while VirtualBox is better suited for short-term tests and casual, user owned, VMs. KVM supports many guest operating systems so you can use Linux, Unix, Windows or something more exotic.

    Install KVM on your system

    1. Make sure your system supports KVM.

      KVM is only supported on systems with Hardware-Assisted Virtualization. If your system does not support HAV you can revert to QEMU, a system which KVM is based on.

    • First install CPU checker:

      $ sudo apt-get -y install cpu-checker
      

    • Check if KVM is supported:
      $ kvm-ok
      INFO: /dev/kvm exists
      KVM acceleration can be used
      

      Looks OK. Still you may need to check your BIOS/EFI whether this feature is enabled.

    1. Install KVM:

      $ sudo apt-get -y install qemu-kvm
      

    KVM Basic Usage

    You can use KVM directly. This method is suitable for testing or troubleshooting but not appropriate for production VMs.

    1. Create a disk image:
      $ qemu-img create -f qcow2 testvm.qcow2 20G
      Formatting 'testvm.qcow2', fmt=qcow2 size=21474836480 encryption=off cluster_size=65536 lazy_refcounts=off
      

      The qcow2 format grows dynamically so it does not really occupy 20GB:

      $ ls -lh testvm.qcow2 
      -rw-r--r-- 1 theo theo   193K Μάι  14 18:43 testvm.qcow2
      
    2. Start a VM instance, to setup your system:
      $ kvm -m 1024 -hda testvm.qcow2 -cdrom ~/Downloads/ubuntu-15.04-desktop-amd64.iso -boot d -smp 2
      

      The options are explained below:

      • -m: memory in MB
      • -hda: first disk image to use
      • -cdrom: you can use an .iso file (ubuntu-15.04-desktop-amd64.iso) or a physical CD-ROM (/dev/sr0).
      • -boot: choose where to boot from. A parameter of d tells KVM to use the cdrom for booting.
      • -smp: Stands for Symmetric Multiprocessing. 2 is the number of CPUs available to the VM.

      After you run the command above you will get a window with your VM running in it:

      KVM VM

      This window will capture your mouse and keyboard when you work in it. If you want to return to your host OS just press Ctrl-Alt together and they will both be released.

    3. Run your VM.

      After the installation is finished you can run your VM from the disk image.

      • First let’s check the size of your disk:
        $ ls -lh testvm.qcow2
        -rw-r--r-- 1 theodotos theodotos 5,9G Μάι  14 19:33 testvm.qcow2
        

      So after the installation of Ubuntu Desktop 15.04 (Vivid Vervet) the disk image has grown to 5.9GB.

      • Run the VM from the disk image:
        $ kvm -m 1024 -hda testvm.qcow2 -smp 2
        

      A new window will pop up with the freshly installed OS.

    Running KVM under libvirt

    The libvirt system, is a platform for running VMs under many different hypervisors using a common API and toolset. It supports KVM, XEN, QEMU, VirtualBox and many others. This is the preferred method of using KVM because the VMs are globally available to privileged (local and remote) users, it facilitates VM management and you can configure autostart and many other features.

    1. Setting up libvirt:
      $ sudo apt-get -y install libvirt-bin
      
    2. Give appropriate permissions to the users expected to manage your VMs:
      $ sudo usermod -a -G libvirtd theo
      

      The theo user will be added as a member in the libvirtd group. After that you will need to log-out, for the permission to be activated.

    Creating a libvirt ready VM

    There are many tools to create VMs for libvirt. In this section we are going to examine two of them: virt-install and uvtool.

    1. Using virt-install.

      The advantage of virt-install is being distro agnostic. That means you can use it to install Debian, Ubuntu, RHEL, CentOS, Fedora, SUSE and many other distros as well.

      • Install virt-install:
        $ sudo apt-get -y install virtinst
        
      • Create a machine:
        $ sudo virt-install -n testvm -r 512 --disk path=/var/lib/libvirt/images/testvm.img,bus=virtio,size=4 -c ~/Downloads/ubuntu-14.04.2-server-amd64.iso --network network=default,model=virtio --graphics vnc,listen=127.0.0.1 --noautoconsole -v
        
        Starting install...
        Allocating 'testvm.img'                     | 4.0 GB     00:00     
        Creating domain...                          |    0 B     00:01     
        Domain installation still in progress.  You can reconnect to 
        the console to complete the installation process.
        
        • -n: VM name
        • -r: RAM in MB
        • –disk: Path for the virtual disk.
        • -c: defive the .iso file or CDROM device to use for the OS installation.
        • –network: Select your preferred networking mode.
        • –graphics: Select the graphics protocol. We are using VNC here that allows connections only from localhost. You can use the 0.0.0.0 (any) instead if 127.0.0.1 IP to allow connections from elsewhere.
        • –noautoconsole: do not run the guest console.
    • Connect to the VM and setup the guest OS:
      $ xtightvncviewer 127.0.0.1
      

      The VNC client will connect to the default VNC port which is 5900. You can append ::<port> to the hostname or IP address if you want to use a different port, e.g. xtightvncviewer 127.0.0.1::5901

      NOTE: If xtightvncviewer is not installed you can install it with sudo apt-get install xtightvncviewer. You can also use a graphical VNC client like Remmina.

  • Verify that the machine is created:

     $ virsh list --all
      Id    Name                           State
       ----------------------------------------------------
       -     testvm                         shut off
    

    The machine will appear as shut off after the OS setup finishes.

  • Start the VM:

    $ virsh start testvm
      Domain testvm started
    

  • Verify that the VM is started:
    $ virsh list
    Id    Name                         State
    ----------------------------------------
    3     testvm                     running
    
    1. Install a VM using uvtool:

      The uvtool is a tool to create minimal VMs. Unlike virt-install you can create only Ubuntu VMs but the overall setup is taken care by uvtool.

      • Install uvtool:
        $ sudo apt-get -y install uvtool uvtool-libvirt
        
      • Create a local repository of ubuntu-cloud images:
        $ uvt-simplestreams-libvirt sync release=trusty arch=amd64
        

      This command will download the trusty (14.04) release locally.

      • Query for local repository
        $ uvt-simplestreams-libvirt query
        release=trusty arch=amd64 label=release (20150506)
        
      • Generate an ssh key pair (unless you already have one):
        $ ssh-keygen -b 4096
        
      • Create a uvt based VM:
        $ uvt-kvm create --cpu 2 --memory=1024 --disk=10 testuvt
        

      This will create a trusty VM with 2 CPUs, 1GB RAM and 10 GB disk.

    • Verify the machine creation:

      $ virsh list
       Id    Name                           State
        ----------------------------------------------------
         5     testuvt                        running
      

    • Connect to your VM:

      $ uvt-kvm ssh testuvt --insecure

      You can get root access, on the VM, with sudo -i.

    Managing libvirt using the graphical Virtual Machine Manager

    Virtual Machine Manager is a front-end to libvirt. It help system administrators managing their VMs using a convenient graphical interface.

    1. Installing Virtual Machine Manager:
      $ sudo apt-get -y install virt-manager
      
    2. Running Virtual Machine Manager:
    • You can find it in the application menu or run virt-manager from the command line.
      virt-manager-1.png

      As you can see the two VMs we created earlier, are already there.

    1. Creating a new machine.

      • Press the Create a new machine icon:
        virt-manager-2.png
    2. New VM options.
    • Select one of the following option to continue:
      virt-manager-3.png
      Each option provides different steps. You may need to read the documentation for all the details. The first option is the most straight forward.

    Managing your VMs with virsh

    1. Listing machines.
    • List only running machines:

      $ virsh list
       Id    Name                           State
        ----------------------------------------------------
         5     testuvt                        running
      

    • List all machines:
      $ virsh list --all
       Id    Name                           State
        ----------------------------------------------------
         5     testuvt                        running
         -     testvm                         shut off
      
    1. Starting machines:
      $ virsh start testvm
      Domain testvm started
      
    2. Shutdown machines:
      $ virsh shutdown testvm
      Domain testvm is being shutdown
      
    3. Restart machines:
      $ virsh reboot testuvt
      Domain testuvt is being rebooted
      
    4. Set machines to autostart:
      • Enable autostart:
        $ virsh autostart testuvt
        Domain testuvt marked as autostarted
        
      • Disable autostart
        $ virsh autostart --disable testuvt
        Domain testuvt unmarked as autostarted
        
    5. Other useful virsh commands:
      • console: get console access to a VM.
      • destroy: destroy (delete) a machine.
      • dominfo: get the machine details.
      • migrate: migrate a machine to another libvirt host.
      • save: save the machine state.
      • snapshot-create: create a snapshot of the machine.

      To see all the supported commands you can run virsh --help.

    Learning to use libvirt is of great value to a Linux sysadmin because the same commands apply for KVM, XEN, VirtualBox, even container systems like OpenVZ and LXC.

    References

    • [1] http://www.phoronix.com/scan.php?page=article&item=ubuntu_1404_kvmbox&num=5
    • [2] https://help.ubuntu.com/14.04/serverguide/virtualization.html
    • [3] https://help.ubuntu.com/community/KVM</port>
    Categories
    Linux

    Installing Cisco Packet Tracer on Ubuntu

    Cisco Packet Tracer is network simulation program for students in the Cisco Networking Academy. There is an Ubuntu version for it and we are going to show you how to install it and configure it.

    This is tested on Ubuntu 15.10 (Vivid Vervet) but it should work on earlier versions. A requirement for this guide is to have root access on the target machine or sudo admin rights.

    Download Cisco Packet Tracer

    If you have an account on the Cisco Networking Academy you can download Packet Tracer from Student Resources. Ask your instructor if you can’t find it.

    After you download it, it should be in your Downloads folder:

    $ ls -l ~/Downloads/
    total 183928
    -rw-r-----  1 theo theo 188328996 Jan  31 03:34 Cisco Packet Tracer 6.2 for Linux - Ubuntu installation - Instructor version.tar.gz
    

    Students can download the student version instead.

    Unpack the package

    1. Change into the Downloads directory:
      $ cd ~/Downloads/
      
    2. Unpack the downloaded file:
      $ tar xvzf Cisco\ Packet\ Tracer\ 6.2\ for\ Linux\ -\ Ubuntu\ installation\ -\ Instructor\ version.tar.gz
      
    3. Verify the unpacked files:
      $ ls -l
      total 183932
      -rw-r-----  1 theo theo 188328996 Ιαν  31 03:34 Cisco Packet Tracer 6.2 for Linux - Ubuntu installation - Instructor version.tar.gz
      drwxr-xr-x 12 theo theo      4096 Ιαν  30 01:00 <strong>PacketTracer62</strong>
      

      The installation files are under the PacketTracer62 directory.

    Install Packet Tracer

    1. Change into the PacketTracer62 directory:

      $ cd PacketTracer62
      

    2. Run the installer:
      $ sudo ./install
      
      • Press the ‘Enter‘key to accept the EULA.
      • Press ‘Space‘ repeatedly to reach 100%.
      • Type ‘y‘ (yes) to accept the EULA.
      • Type the location where you want the package installed. If you just press ‘Enter‘ it will choose the default (/opt/pt).
      • When asked to create a symbolic link "packettracer" in /usr/local/bin type y (yes).
    3. Setup the Packet Tracer Desktop Link:

      While you are in the PacketTraser62 directory, copy the PT Desktop link to your Desktop and assign executable permissions on it:

      $ cp bin/Cisco-PacketTracer.desktop ~/Desktop/
      $ chmod +x ~/Desktop/Cisco-PacketTracer.desktop
      

      Unfortunately the links to the program and its icon are wrong. Edit the Exec and Icon directives in the Cisco-PacketTracer.desktop file as follows:

      # Exec=/usr/local/PacketTracer6/packettracer
      # Icon=/usr/local/PacketTracer6/art/app.png
      Exec=/opt/pt/bin/PacketTracer6
      Icon=/opt/pt/art/app.png
      

      Alternatively you can just type the packetracer command from your terminal.

    Although Packet Tracer is based on several Free Software libraries, it is not Free Software itself. I hope its Cisco developers see the light and release this great learning tool as Free Software. Only good things can come from such decision.

    Categories
    Linux

    Adding a new disk in an existing Volume Group (LVM)

    This is a guide about adding a new disk to an existing [LVM](http://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29 "Logical Volume Manager") VG

    Check current setup

    1. First let’s check and document the current setup.
    • Checking the /proc filesystem:
      # cat /proc/partitions 
       major minor  #blocks  name
          8        0    7880544 sda
          8        1     248832 sda1
          8        2          1 sda2
          8        5    7628800 sda5
        252        0    6574080 dm-0
        252        1    1036288 dm-1
      
    • Using lsblk:
      # lsblk
      NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
      sda                      8:0    0  7,5G  0 disk 
      ├─sda1                   8:1    0  243M  0 part /boot
      ├─sda2                   8:2    0    1K  0 part 
      └─sda5                   8:5    0  7,3G  0 part 
        ├─ubuntu--vg-root   252:0    0  6,3G  0 lvm  /
        └─ubuntu--vg-swap_1 252:1    0 1012M  0 lvm  [SWAP]
      
    1. Check and document the LVM layout.
    • Volume Group info:
      # vgs
        VG         #PV #LV #SN Attr   VSize VFree 
        ubuntu-vg    1   2   0 wz--n- 7,27g 16,00m
      
    • Physical Volume info:
      # pvs
        PV         VG         Fmt  Attr PSize PFree 
        /dev/sda5  ubuntu-vg  lvm2 a--  7,27g 16,00m
      
    • Logical Volume info:
      # lvs
        LV     VG         Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
        root   ubuntu-vg  -wi-ao----    6,27g
        swap_1 ubuntu-vg  -wi-ao---- 1012,00m
      

    Add a new physical or virtual disk

    Now add the new disk on your server. On VMs it is possible to add a new disk without powering off. On physical servers this can be possible too if the server comes with hot-swap functionality. Check your server specs first!

    1. Check if the new disk is detected.
    • You can use this command if you want your system to detect the new disk without rebooting:
      # for SCSI_HOST in /sys/class/scsi_host/* ; do echo "- - -" > $SCSI_HOST/scan ; done
      

      The above command simply loops through the SCSI hosts under the /sys/class/scsi_host directory and sends the "– – –" string to them. This forces the SCSI hosts to detect the new disk that has been attached.

    • Using the /proc filesystem:

      # cat /proc/partitions
      major minor  #blocks  name<br />
         8        0    7880544 sda
         8        1     248832 sda1
         8        2          1 sda2
         8        5    7628800 sda5
         8       16   31522680 sdb
       252        0    6574080 dm-0
       252        1    1036288 dm-1
      

      The size of the disk in GB, is 30GB:

      # echo '31522680/1024/1024' | bc -l
      30.06237030029296875000
      

    • Using lsblk:
      # lsblk
      NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
      sda                      8:0    0  7,5G  0 disk 
      ├─sda1                   8:1    0  243M  0 part /boot
      ├─sda2                   8:2    0    1K  0 part 
      └─sda5                   8:5    0  7,3G  0 part 
         ├─ubuntu--vg-root   252:0    0  6,3G  0 lvm  /
         └─ubuntu--vg-swap_1 252:1    0 1012M  0 lvm  [SWAP]
      sdb                      8:16   0 30,1G  0 disk
      

    Add the new disk to LVM Volume Group

    1. Create a new partition on the new disk:
      # fdisk /dev/sdb
      
      Welcome to fdisk (util-linux 2.25.2).
      Changes will remain in memory only, until you decide to write them.
      Be careful before using the write command.
      
      Device does not contain a recognized partition table.
      Created a new DOS disklabel with disk identifier 0xaea3ab78.
      
      Command (m for help): n
      Partition type
        p   primary (0 primary, 0 extended, 4 free)
        e   extended (container for logical partitions)
      Select (default p):< Using default response p.
      Partition number (1-4, default 1): 
      First sector (2048-63045359, default 2048): 
      Last sector, +sectors or +size{K,M,G,T,P} (2048-63045359, default 63045359):
      
      Created a new partition 1 of type 'Linux' and of size 30,1 GiB.
      
      Command (m for help): t
      Selected partition 1
      Hex code (type L to list all codes): <strong>8e
      Changed type of partition 'Linux' to 'Linux LVM'.
      
      Command (m for help): <strong>w</strong>
      The partition table has been altered.
      Calling ioctl() to re-read partition table.
      Syncing disks.
      
    2. Verify the creation of a new partition:
      # fdisk -l /dev/sdb
      
      Disk /dev/sdb: 30,1 GiB, 32279224320 bytes, 63045360 sectors
      Units: sectors of 1 * 512 = 512 bytes
      Sector size (logical/physical): 512 bytes / 512 bytes
      I/O size (minimum/optimal): 512 bytes / 512 bytes
      Disklabel type: dos
      Disk identifier: 0xaea3ab78
      
      Device     Boot Start      End  Sectors  Size Id Type
      /dev/sdb1        2048 63045359 63043312 30,1G 8e Linux LVM
      

    Add the new partition to the Volume Group

    1. Extend the Volume Group by adding a new disk:
      # vgextend ubuntu-vg /dev/sdb1
       Physical volume "/dev/sdb1" successfully created
       Volume group "ubuntu-vg" successfully extended
      
    2. Check the current free space of the Volume Group:
      # vgs
       VG        #PV #LV #SN Attr   VSize  VFree 
       ubuntu-vg   2   2   0 wz--n- 37,33g 30,07g
      
    3. Verify the new Physical Volume:
      # pvs
       PV         VG        Fmt  Attr PSize  PFree 
       /dev/sda5  ubuntu-vg lvm2 a--   7,27g 16,00m
       /dev/sdb1  ubuntu-vg lvm2 a--  30,06g 30,06g
      
    4. Check the state of the Logical Volumes:
      # lvs
       LV     VG        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
       root   ubuntu-vg -wi-ao---- 6,27g     
       swap_1 ubuntu-vg -wi-ao---- 1012,00m
      

      Nothing changed yet, of course.

    Resize the logical volume

    1. Use the lvresize command to resize the root volume:

      # lvresize -L 30,07g /dev/ubuntu-vg/root
       Rounding size to boundary between physical extents: 30,07 GiB
       Size of logical volume ubuntu-vg/root changed from 6,27 GiB (1605 extents) to 30,07 GiB (7698 extents).
       Logical volume root successfully resized
      

    2. Verify the volume resize:
      # lvs
       LV     VG        Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
       root   ubuntu-vg -wi-ao----   30,07g
       swap_1 ubuntu-vg -wi-ao---- 1012,00m
      

      The root volume is now at 30,07GB. Good.

    Resize the filesystem

    1. Check the current filesystem size:

      # df -hT
      Filesystem                   Type      Size  Used Avail Use% Mounted on
      udev                         devtmpfs  485M     0  485M   0% /dev
      tmpfs                        tmpfs     100M  5,1M   95M   6% /run
      /dev/mapper/ubuntu--vg-root  ext4      <strong>6,1G</strong>  2,4G  3,4G  41% /
      tmpfs                        tmpfs     496M     0  496M   0% /dev/shm
      tmpfs                        tmpfs     5,0M  4,0K  5,0M   1% /run/lock
      tmpfs                        tmpfs     496M     0  496M   0% /sys/fs/cgroup
      /dev/sda1                    ext2      236M   40M  184M  18% /boot
      tmpfs                        tmpfs     100M   16K  100M   1% /run/user/1000
      

      Still using the old size, as expected.

    2. Resize the root filesystem:

      # resize2fs /dev/mapper/ubuntu--vg-root
      resize2fs 1.42.12 (29-Aug-2014)
      Filesystem at /dev/mapper/ubuntu--vg-root is mounted on /; on-line resizing required
      old_desc_blocks = 1, new_desc_blocks = 2
      The filesystem on /dev/mapper/ubuntu--vg-root is now 7882752 (4k) blocks long.
      

    3. Verify the new size of the root filesystem:
      # df -hT
      Filesystem                   Type      Size  Used Avail Use% Mounted on
      udev                         devtmpfs  485M     0  485M   0% /dev
      tmpfs                        tmpfs     100M  5,1M   95M   6% /run
      /dev/mapper/ubuntu--vg-root  ext4       30G  2,4G   26G   9% /
      tmpfs                        tmpfs     496M     0  496M   0% /dev/shm
      tmpfs                        tmpfs     5,0M  4,0K  5,0M   1% /run/lock
      tmpfs                        tmpfs     496M     0  496M   0% /sys/fs/cgroup
      /dev/sda1                    ext2      236M   40M  184M  18% /boot
      tmpfs                        tmpfs     100M   16K  100M   1% /run/user/1000
      

    So now we have a logical root volume that expands across multiple physical disks. Notice, however, that this is not a very solid setup, since the loss of one of the physical volumes can bring down the whole system along with your data. Thus make sure that you have a solid and tested backup procedure in place. The restore procedure should also be documented in every detail in your disaster recovery practices.

    References

    • http://tldp.org/HOWTO/LVM-HOWTO/index.html
    • http://wingloon.com/2013/05/07/how-to-detect-a-new-hard-disk-without-rebooting-vmware-linux-guest/