Friday, April 2, 2010

How to extend LVM on VMWare Guest running Linux

Recently I found my one of virtual Linux machine which running on VMWare ESX server have run out of disk space. And after I googled the Internet, I think it's quite simple as my virtual Linux machine is running on LVM.

And I also found the following solution will also apply to any VMWare virtual machine which running Linux OS with LVM.

Expand the disk

Turn off the virtual machine that you wish to extend disk space.


You can expand the virtual disk size by entering ESX server hidden console mode or using VMware Infrastructure Client to remote manage your ESX Server. I will introduce both way here.

  1. Method One:

    Since the GUI mode is easy. Click Your VMWare Guest Machine in your remote management console from VMware Infrastructure Client. Then click "Edit Virtual Machine Settings" from "Getting Started" which is in the Right Panel. Select Hardware -> Hard Disk 1 -> Capacity - New Size (at Right Side), and then expand the Capacity to whatever size that you need. In our case, I expanded from 100GB to 200GB. Then press "OK".


  2. Method Two:

    For ESXi Server, using the following command to enter ESXi hidden console mode after press "Alt"+"F1":

    unsupported



    Then press "Enter", now enter your ESXi server root password. You should be able to successfully log into the hidden ESXi Server console.

    For ESX Server, If you press Alt-F1, you can get access to the Linux login prompt of the ESX service console and login to a command prompt.


Now enter the following command to expand the VM Disk:

vmkfstools -X 200G "Your VMDK file here"


To extend an existing Virtual Disk to 200GB.

Note:

If you are running VMware workstation product, then you can expand the virtual disk to expand the virtual disk on your Windows machine by running this command:

vmware-vdiskmanager -x 200G "My harddisk.vmdk"


After above steps, you finished preparing the extra virtual disk space. Now we can start your VMware virtual machine and open a terminal session to continue expanding your LVM.

Issue the df -k command which shows us that the logical volume is at 100% usage. We need to create a partition on /dev/sda. Type ls -al /dev/sda* which lists the existing partitions. Our new partition will be /dev/sda3.

# ls -al /dev/sda*
brw-r----- 1 root disk 8, 0 Apr 2 19:26 /dev/sda
brw-r----- 1 root disk 8, 1 Apr 2 19:26 /dev/sda1
brw-r----- 1 root disk 8, 2 Apr 2 19:26 /dev/sda2


Type fdisk /dev/sda then type n for new partition. Enter p and 3 for the partition number (in this instance, obviously enter the partition number that matches your environment.) Also accept the default First and Last cylinders. Finally type w to write table to disk and exit.

If prompted to reboot then do so to ensure that the new partition table is written. After a restart, type ls -al /dev/sda* to verify whether the new partition was created successfully.

# ls -al /dev/sda*
brw-r----- 1 root disk 8, 0 Apr 2 19:26 /dev/sda
brw-r----- 1 root disk 8, 1 Apr 2 19:26 /dev/sda1
brw-r----- 1 root disk 8, 2 Apr 2 19:26 /dev/sda2
brw-r----- 1 root disk 8, 3 Apr 2 19:26 /dev/sda3


After verification of new partition, we need to create a physical volume and add it to the volume group.

Type pvcreate /dev/sda3

[root@rhel5 ~]# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created


Type vgextend VolGroup00 /dev/sda3

# vgextend VolGroup00 /dev/sda3
Volume group "VolGroup00" successfully extended


We need to extend the logical volume. Type vgdisplay:

# vgdisplay
--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 2
Act PV 2
VG Size 99.84 GB
PE Size 32.00 MB
Total PE 3345
Alloc PE / Size 2868 / 99.88 GB
Free PE / Size 2870 / 99.97 GB
VG UUID MkluWy-e5PA-QTGN-fF7k-ZxO3-6bC7-qxfCii


Which shows that there is 99.97GB free that we can add to the volume group. To extend the volume type lvextend -L +99.97G /dev/VolGroup00/LogVol00. If you got an error message that stated that there was "Insufficient free space: 2871 extends needed but only 2870 evailable". After a quick google: Insufficient Free Extents for a Logical Volume. It seems that we needed to select a lightly smaller size. So we changed to 99GB rather than 99.97GB which solved this problem.

# lvextend -L+99G /dev/VolGroup00/LogVol00
Rounding up size to full physical extent 99 GB
Extending logical volume LogVol00 to 198 GB
Logical volume LogVol00 successfully resized


Finally we need to resize the file system by typing resize2fs /dev/VolGroup00/LogVol00.

resize2fs /dev/VolGroup00/LogVol00
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/VolGroup00/LogVol00 is mounted on /; on-line resizing required
Performing an on-line resize of /dev/VolGroup00/LogVol00 to 20101888 (4k) blocks.
The filesystem on /dev/VolGroup00/LogVol00 is now 20101888 blocks long.


Type df -k to see if the new space is available to the logical volume.

df -k
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
1713916000 82428400 88250280 48% /
/dev/sda1 101086 24538 71329 26% /boot
tmpfs 1037744 0 1037744 0% /dev/shm


The logical volume has now been resized and now has used space of just 48%!

29 comments :

  1. At my job, after doing the same kind of procedure graph, we began to ask ourselves, why are using a LVM on a Linux VM guests?

    Since we're no longer living in the physical OS world, we didn't need to use the OS hacks(LVM) to overcome physical disk limitations anymore.
    We decided to Just let the hypervisor and virtual storage do that work for us.

    For example, in our production setup (3 tier commerce with VMs for database , webserver, and appserver), we're see a great improvement in managability and performance (>10%) by just dropping LVM, and most partitions.

    In your example, the resize process is 7 functional steps:
    1. Increase size of VMDK
    2. In VM OS, Create Partition (??)
    3. REBOOT (!!)
    4. PVCreate
    5. VGExtend
    6. LVExtend
    7. Resize2fs

    Going to a LVM/partition-less setup reduces expansion to 3 steps and we don't need to take the VM OS offline!
    1. Increase size of VMDK
    2- Inside the VM, OS, rescan the scsi drive with:'echo 1 >/sys/class/scsi_device//rescan; dmesg' (dmesg will check that you drive isize has grown)
    3- Resize2fs.

    Our current disk arrangement has 3 VM HD devices
    0 - small device (100M) with a single BOOT partition
    1 - entire device is /
    2 - entire device is SWAP

    Doing this has simplified resizing so much, I now let the junior admins and my manager expand drive space as needed.

    It's also let's us really be spartan on space since expansion is so quick. Instead of increasing systems in 30-50GB chunks, we can do 10-15GB and let our rmonitoring system warn us when space gets tight.

    ReplyDelete
  2. Good comment, I agree them all. But this is just an article to let people have one option to expand disk size, and there are plenty ways to do same thing in both VM and Linux world though.

    ReplyDelete
  3. I used the method of Edward ( 7 steps) and I successfully expanded the disk of Linux EL5 guest from 150G to 350G for oracle appsR12.
    I would like to try out with 3 step but I am truggling with resize2fs.
    Could you please explain more about the use of resize2fs.
    I have done:
    $ echo "1" > /sys/class/scsi_device/0:0:0:0/device/rescan
    $ dmesg
    Extract from output of dmesg "SCSI device sda: 734003200 512-byte hdwr sectors (375810 MB)
    "
    $ resize2fs /dev/VolGroup00/LogVol00
    resize2fs 1.39 (29-May-2006)
    The filesystem is already 38256640 blocks long. Nothing to do!


    The volume is not bigger, check with :
    $vgdisplay
    Extract from output
    " Alloc PE / Size 4796 / 149.88 GB
    Free PE / Size 0 / 0
    "

    Thanks in advance

    ReplyDelete
  4. Please ignore my question, after reading carefully your post I have seen my mistake :
    I am using LVM/partionned.

    ReplyDelete
  5. Dear Sir,
    Thanks very much for this very useful tutorial.

    It has saved my bacon, in my Organisation.

    In my VMware Course, I was told, that Thin provisioning would automatically increase the space allocated in and to the OS.

    Imagine my surprise when my company database server crashed due to space limits :P

    You are really a life saver.

    ReplyDelete
  6. Nice post, it was very usefull to me.

    But I found that instead of 3 steps, slightly more steps are involved (in a partition-less LVM setup):

    1. Increase size of VMDK
    2- Inside the VM, OS, rescan the scsi drive with:'echo 1 >/sys/class/scsi_device/<>/rescan; dmesg' (dmesg will check that you drive isize has grown)
    3- resize PV, so that LVM recognizes the bigger disk:
    pvresize /dev/sdb
    4.Extend LV
    5.resize FS

    Cheers

    ReplyDelete
  7. On the slight chance that anonymous #1 reads this again, I would like to know what Linux distribution you are using? We have been testing the ability to grow a ext3 file system with no partitions and no LVM. We are trying to install RedHat 5 on /dev/sd# (with no partitions). It does not allow you to do this. We receive the error, "Device is a loopback and will need to be formatted before the install". To do some testing, I installed RedHat within partitions on my disks. Once the install is completed, I create a new file system on a standalone disk with no partitions (mkfs.ext3 /dev/sdd to be exact). I am able to then grow this partition exactly as you detailed above. However, I really want to know if there is a way to install RedHat on a partitionless system?

    If want to reply to me, email: jemurray@wustl.edu

    ReplyDelete
  8. Edward, thanks for posting this guide, it's helped a lot!

    ReplyDelete
  9. Hello,
    I think the post from Anonymous #1 mentions LVM/partition-less, which means (once again I think) that they're using the whole disk to run LVM on top of it. So instead of having /dev/sda1 as a Linux LVM partition and running 'pvcreate /dev/sda1', they probably didn't create a partition on sda, and used the whole disk for LVM ('pvcreate /dev/sda'), which then gives you the opportunity to increase the underlying VMDK, rescan the disk ('echo "1" > /sys/class/scsi_device/0:0:0:0/device/rescan' where 0:0:0:0 is the address of /dev/sda, as per 'ls -l /dev/disk/by-path'), and then increase the PV ('pvresize /dev/sda') after which you can increase the size of the LV you want to increase.
    On the 'lvextend' front, if you want to use all the remaining space, instead of running 'lvextend -L +99GB', you can run 'lvextend -l +100%FREE' (and notice the different in case on the '-L' and '-l' options).
    I hope this helps.

    ReplyDelete
  10. Although I have not tried this out ( but possibly will do that this week) the Comment #1 meant to create 3 virtual partitions from the vmware:
    0 - boot
    1 - / and evrything beneeth
    2 - SWAP
    From the guest os perspsective this woul look like either /dev/sda, /dev/sdb, /dev/sdc or maybe /dev/sda1, /dev/sdb1, /dev/sdc1. Depending on that you create FS directly on to these "virtual" partitions without any LVM in-between !!!

    When I test it, hopefully will rmember to post the results.

    ReplyDelete
  11. What do you mean by 3 partitions from the vmware? If you mean 3 VMDKs, and not using LVM, then I think you'll have to create a partition on each VMDK (which will appear each as a sd disk (/dev/sda, /dev/sdb and /dev/sdc), as I'm not sure you can use the whole disk to install linux directly without partition nor LVM.
    So, from a VMware point of view, the disks (VMDKs) are "virtual" disk, but as far as the guest OS is concerned, they are physical, SCSI disks. So you could use /dev/sda1 as /boot, /dev/sdb1 as the root filesystem, and /dev/sdc1 as the swap space. But I think you *will* need to create a Linux partition (type 83, from memory), for the /boot and / filesystems and Linux swap partition type (once again from memory, type 82).

    ReplyDelete
  12. Running CentOS under ESXi 4.1. This is the most understandable and straightforward procedure I found. Thanks very much!

    One reboot was required, after running fdisk, in order to update the partition table in the kernel.

    ReplyDelete
  13. This was fantastic. Thanks so much!

    ReplyDelete
  14. Good Post.
    The 3-step procedure described by anonymous one sounds good, but I think that's (at least with vmware workstation) is a one way road: One can increase the size of drives, but shrinking is no option (afaik). It would be of interest if this is also as easily as the described extension is possible

    ReplyDelete
  15. @ Paul Stone - you should not even need a reboot to update the partition table in the kernel, just re-read the partition table:
    blockdev --rereadpt /dev/sdx
    where x is the disk you've just changed.

    @ Anonymous (September 7), with VMware, you can indeed grow disks, but not reduce them. The way around that, I found, is to create another (smaller) disk, add it to the VG you want to recude, and pvmove all the LVs to the new disk. At least that's how I did it.

    ReplyDelete
  16. * In VMware I add second disk to virtual machine
    * run pvscan or partprobe : see additional disk
    * fdisk -l : list the disk
    * fdisk /dev/sdb or sdb1 : to create new partition, take defaults (primary) and make 8e type (L)
    * pvcreate /dev/sdb1 : create a new physical volume on the new partition
    * vgdisplay to get LVM in this case was lvroot
    * vgextend lvroot /dev/sdb1 : extend existing volume group (make sure vg free size is = new partition size using vgdisplay command)
    * pvscan : to see verify new physical volume
    * lvextend -L +4G /dev/mapper/lvroot-rootvg : extend existing LVM by adding new paritition or
    * lvextend -l +100%FREE /dev/mapper/lvroot-rootvg : make the LV larger
    * lvdisplay : make sure logical volume is increased
    * resize2fs /dev/mapper/lvroot-rootvg : resize file system

    ReplyDelete
  17. @ Nick,

    Should the "blockdev --rereadpt" command work even on /dev/sda (the boot device containing mounted filesystems)? The 'rescan' option for the '0:0:0:0' device worked so "fdisk -l" sees the larger disk size, but after I deleted and recreated the last partition on the disk (used for LVM PV), the blockdev command returns "BLKRRPART: Device or resource busy"

    ReplyDelete
  18. Thanks lol...
    Below mentioned link is very easy to understand,
    http://www.redhatlinux.info/2010/11/lvm-logical-volume-manager.html

    ReplyDelete
  19. Thanks for this guide, I just extended one of our production servers.

    ReplyDelete
  20. Thank you very much for this post!
    I did have some trouble with extending the disk though. After some searching, I found out that I had to delete all my snapshots before I could extend my hdd.

    Thanks again!

    ReplyDelete
  21. greatly helped, thanks a lot

    --Ldom

    ReplyDelete
  22. Great post! Thanks =)

    ReplyDelete
  23. vmkfstools -X 200G "Your VMDK file here" working fine. I am able to see that space in fdisk -l /dev/sdX

    But when creating a partition I am not getting that additional space.

    1) -X 110G makes the vmdk size to 110G . am i correct? Say I get 10G additional space by saying -X 110G on a 100G vmdk file. fdisk shows that its 110G , but when creating a partition I am not getting 10G???

    2) From vsphere client I am not able to see the change in disk size.

    ReplyDelete
  24. Thanks for great HOWTO :)

    ReplyDelete
  25. Thank you so much, it is very helpful.

    ReplyDelete
  26. Alright, since many are wondering about the first comment (I have, a lot) I found the way of making that setup work, it basicaly involves making a partition less install of your linux. Just tested it, resizing of / on /dev/sdc worked perfectly on production server.
    Complete series of instructions found here: http://luke.bowbak.org/2013/01/24/partitions-thing-of-the-past-2.html , big fucking thanks to Luke Bakken.

    ReplyDelete
  27. Excellent steps ... very helpful! Thanks a million!

    ReplyDelete
  28. Thanks for the process! Worked perfectly. And much easier to follow than reading dozens of pages of not very useful official Linux documentation.

    ReplyDelete
  29. At the point of "Type pvcreate /dev/sda3 ", I get not found or ignored by filtering. And I verified it's there, using your methods and p in fdisk /dev/sda, so I'm not sure why it's ignoring it.

    I actually went LVM for this VM because I had issues extending it previously with resize2fs using the official instructions: http://kb.vmware.com/kb/1007907

    ReplyDelete

Comments System

Disqus Shortname