How to Manage and Use LVM (Logical Volume Management) in Ubuntu

In our previous article we told you what LVM is and what you may want to use it for, and today we are going to walk you through some of the key management tools of LVM so you will be confident when setting up or expanding your installation.

As stated before, LVM is a abstraction layer between your operating system and physical hard drives. What that means is your physical hard drives and partitions are no longer tied to the hard drives and partitions they reside on. Rather, the hard drives and partitions that your operating system sees can be any number of separate hard drives pooled together or in a software RAID.

To manage LVM there are GUI tools available but to really understand what is happening with your LVM configuration it is probably best to know what the command line tools are. This will be especially useful if you are managing LVM on a server or distribution that does not offer GUI tools.

Most of the commands in LVM are very similar to each other. Each valid command is preceded by one of the following:

• Physical Volume = pv
• Volume Group = vg
• Logical Volume = lv

The physical volume commands are for adding or removing hard drives in volume groups. Volume group commands are for changing what abstracted set of physical partitions are presented to your operating in logical volumes. Logical volume commands will present the volume groups as partitions so that your operating system can use the designated space.

Downloadable LVM Cheat Sheet

To help you understand what commands are available for each prefix we made a LVM cheat sheet. We will cover some of the commands in this article, but there is still a lot you can do that won’t be covered here.

All commands on this list will need to be run as root because you are changing system wide settings that will affect the entire machine.

(Click on the thumbnail for a full size image)

How to View Current LVM Information

The first thing you may need to do is check how your LVM is set up. The s and display commands work with physical volumes (pv), volume groups (vg), and logical volumes (lv) so it is a good place to start when trying to figure out the current settings.

The display command will format the information so it’s easier to understand than the s command. For each command you will see the name and path of the pv/vg and it should also give information about free and used space.

The most important information will be the PV name and VG name. With those two pieces of information we can continue working on the LVM setup.

Creating a Logical Volume

Logical volumes are the partitions that your operating system uses in LVM. To create a logical volume we first need to have a physical volume and volume group. Here are all of the steps necessary to create a new logical volume.

Create physical volume

We will start from scratch with a brand new hard drive with no partitions or information on it. Start by finding which disk you will be working with. (/dev/sda, sdb, etc.)

Note: Remember all of the commands will need to be run as root or by adding ‘sudo’ to the beginning of the command.

fdisk -l

If your hard drive has never been formatted or partitioned before you will probably see something like this in the fdisk output. This is completely fine because we are going to create the needed partitions in the next steps.

Our new disk is located at /dev/sdb so lets use fdisk to create a new partition on the drive.

There are a plethora of tools that can create a new partition with a GUI, including Gparted, but since we have the terminal open already, we will use fdisk to create the needed partition.

From a terminal type the following commands:

fdisk /dev/sdb

This will put you in a special fdisk prompt.

Enter the commands in the order given to create a new primary partition that uses 100% of the new hard drive and is ready for LVM. If you need to change the partition size or want multiple partions I suggest using GParted or reading about fdisk on your own.

Warning: The following steps will format your hard drive. Make sure you don’t have any information on this hard drive before following these steps.

• n = create new partition
• p = creates primary partition
• 1 = makes partition the first on the disk

Push enter twice to accept the default first cylinder and last cylinder.

To prepare the partition to be used by LVM use the following two commands.

• t = change partition type
• 8e = changes to LVM partition type

Verify and write the information to the hard drive.

• p = view partition setup so we can review before writing changes to disk
• w = write changes to disk

After those commands, the fdisk prompt should exit and you will be back to the bash prompt of your terminal.

Enter pvcreate /dev/sdb1 to create a LVM physical volume on the partition we just created.

You may be asking why we didn’t format the partition with a file system but don’t worry, that step comes later.

Create volume Group

Now that we have a partition designated and physical volume created we need to create the volume group. Luckily this only takes one command.

vgcreate vgpool /dev/sdb1

Vgpool is the name of the new volume group we created. You can name it whatever you’d like but it is recommended to put vg at the front of the label so if you reference it later you will know it is a volume group.

Create logical volume

To create the logical volume that LVM will use:

lvcreate -L 3G -n lvstuff vgpool

The -L command designates the size of the logical volume, in this case 3 GB, and the -n command names the volume. Vgpool is referenced so that the lvcreate command knows what volume to get the space from.

Format and Mount the Logical Volume

One final step is to format the new logical volume with a file system. If you want help choosing a Linux file system, read our how to that can help you choose the best file system for your needs.

mkfs -t ext3 /dev/vgpool/lvstuff

Create a mount point and then mount the volume somewhere you can use it.

mkdir /mnt/stuff
mount -t ext3 /dev/vgpool/lvstuff /mnt/stuff

Resizing a Logical Volume

One of the benefits of logical volumes is you can make your shares physically bigger or smaller without having to move everything to a bigger hard drive. Instead, you can add a new hard drive and extend your volume group on the fly. Or if you have a hard drive that isn’t used you can remove it from the volume group to shrink your logical volume.

There are three basic tools for making physical volumes, volume groups, and logical volumes bigger or smaller.

Note: Each of these commands will need to be preceded by pv, vg, or lv depending on what you are working with.

• resize – can shrink or expand physical volumes and logical volumes but not volume groups
• extend – can make volume groups and logical volumes bigger but not smaller
• reduce – can make volume groups and logical volumes smaller but not bigger

Let’s walk through an example of how to add a new hard drive to the logical volume “lvstuff” we just created.

Install and Format new Hard Drive

To install a new hard drive follow the steps above to create a new partition and add change it’s partition type to LVM (8e). Then use pvcreate to create a physical volume that LVM can recognize.

Add New Hard Drive to Volume Group

To add the new hard drive to a volume group you just need to know what your new partition is, /dev/sdc1 in our case, and the name of the volume group you want to add it to.

This will add the new physical volume to the existing volume group.

vgextend vgpool /dev/sdc1

Extend Logical Volume

To resize the logical volume we need to say how much we want to extend by size instead of by device. In our example we just added a 8 GB hard drive to our 3 GB vgpool. To make that space usable we can use lvextend or lvresize.

lvextend -L8G /dev/vgpool/lvstuff

While this command will work you will see that it will actually resize our logical volume to 8 GB instead of adding 8 GB to the existing volume like we wanted. To add the last 3 available gigabytes you need to use the following command.

lvextend -L+3G /dev/vgpool/lvstuff

Now our logical volume is 11 GB in size.

Extend File System

The logical volume is 11 GB but the file system on that volume is still only 3 GB. To make the file system use the entire 11 GB available you have to use the command resize2fs. Just point resize2fs to the 11 GB logical volume and it will do the magic for you.

resize2fs /dev/vgpool/lvstuff

Note: If you are using a different file system besides ext3/4 please see your file systems resize tools.

Shrink Logical Volume

If you wanted to remove a hard drive from a volume group you would need to follow the above steps in reverse order and use lvreduce and vgreduce instead.

1. resize file system (make sure to move files to a safe area of the hard drive before resizing)
2. reduce logical volume (instead of + to extend you can also use – to reduce by size)
3. remove hard drive from volume group with vgreduce

Backing up a Logical Volume

Snapshots is a feature that some newer advanced file systems come with but ext3/4 lacks the ability to do snapshots on the fly. One of the coolest things about LVM snapshots is your file system is never taken offline and you can have as many as you want without taking up extra hard drive space.

When LVM takes a snapshot, a picture is taken of exactly how the logical volume looks and that picture can be used to make a copy on a different hard drive. While a copy is being made, any new information that needs to be added to the logical volume is written to the disk just like normal, but changes are tracked so that the original picture never gets destroyed.

To create a snapshot we need to create a new logical volume with enough free space to hold any new information that will be written to the logical volume while we make a backup. If the drive is not actively being written to you can use a very small amount of storage. Once we are done with our backup we just remove the temporary logical volume and the original logical volume will continue on as normal.

Create New Snapshot

To create a snapshot of lvstuff use the lvcreate command like before but use the -s flag.

lvcreate -L512M -s -n lvstuffbackup /dev/vgpool/lvstuff

Here we created a logical volume with only 512 MB because the drive isn’t being actively used. The 512 MB will store any new writes while we make our backup.

Mount New Snapshot

Just like before we need to create a mount point and mount the new snapshot so we can copy files from it.

mkdir /mnt/lvstuffbackup
mount /dev/vgpool/lvstuffbackup /mnt/lvstuffbackup

Copy Snapshot and Delete Logical Volume

All you have left to do is copy all of the files from /mnt/lvstuffbackup/ to an external hard drive or tar it up so it is all in one file.

Note: tar -c will create an archive and -f will say the location and file name of the archive. For help with the tar command use man tar in the terminal.

tar -cf /home/rothgar/Backup/lvstuff-ss /mnt/lvstuffbackup/

Remember that while the backup is taking place all of the files that would be written to lvstuff are being tracked in the temporary logical volume we created earlier. Make sure you have enough free space while the backup is happening.

Once the backup finishes, unmount the volume and remove the temporary snapshot.

umount /mnt/lvstuffbackup
lvremove /dev/vgpool/lvstuffbackup/

Deleting a Logical Volume

To delete a logical volume you need to first make sure the volume is unmounted, and then you can use lvremove to delete it. You can also remove a volume group once the logical volumes have been deleted and a physical volume after the volume group is deleted.

Here are all the commands using the volumes and groups we’ve created.

umount /mnt/lvstuff
lvremove /dev/vgpool/lvstuff
vgremove vgpool
pvremove /dev/sdb1 /dev/sdc1

That should cover most of what you need to know to use LVM. If you’ve got some experience on the topic, be sure to share your wisdom in the comments.

Duplicating and Backing Up LVM Backed Xen DomU from a Remote Server

LVM’s snapshot feature enables us to duplicate an LVM backed Xen DomU in seconds inside of the same volume group. When we want to duplicate LVM backed Xen DomU from a remote server, we need to make use of LVM’s snapshot function and tools like scp and dd. Backing up the DomU is only part of the process of duplicating one DomU. We can finish the backing up work after copying DomU’s image to remote server. How to duplicate and back up LVM backed Xen DomU from remote server is introduced in this post.

Assumption and our goal:

There are Xen DomU virtual machines in LVM logical volume /dev/vg_xen/vm-10.0.0.123/ on server 10.0.0.10 (remote server).

Now we want to duplicate the virtual machine vm-10.0.0.123 to vm-10.0.0.124 which we want to stored in logical volume vm-10.0.0.124 in volume group vgxen on server 10.0.0.11 (local server). vm-10.0.0.124‘s IP will be 10.0.0.124. There are 6 steps to duplicate this virtual machine. If we just need to back up it, we can just use step 1 to setp 3.

1. Duplicate the LVM logical volume on remote server

We can duplicate the LVM volume group on remote server by making a snapshot of it. The snapshot is a clone of the logical volume. We can make a new logical volume vm-10.0.0.123-snapshot as the snapshot of logical volume vm-10.0.0.123:

# lvcreate -L20480 -s -n 'vm-10.0.0.123-snapshot' /dev/vgxen/vm-10.0.0.123

Here the size of the new logical volume is 20G as the same of the original one.

Here we should make sure that vm-10.0.0.123 is power off to avoid the situation there are “write” that is still in DomU’s cache.

2. Save the snapshot to image file using dd tool on remote server

We can use dd command to save a image of the new created snapshot logical volume of the DomU:

# dd if=/dev/vgxen/vm-10.0.0.123-snapshot of=/lhome/xen-image/vm-10.0.0.123-lv.img bs=1k

This process may take several minutes or more depending on the size of the logical volume and the hard disk’s speed.

3. Copy the DomU’s image to the local server from the remote one

We can transfer the image from remote server after finishing the dd command on remote server.

# scp -c arcfour root@10.0.0.10:/lhome/xen-image/vm-10.0.0.123-lv.img /lhome/xen-image/

We set “-c arcfour” to get higher transfer speed of scp.

4. Create logical volume for the new DomU on local server

We can do this simultaneously with step 3. We create a new logical volume vm-10.0.0.124 under volume group vgxen on local server:

# lvcreate -L20480 -n 'vm-10.0.0.124' vgxen

The size of the logical volume is the same as the one of the DomU in remote server.

5. Duplicate the data in the logical volume from the image

In this step, we use the dd command to copy data from the image file to the newly created logical volume:

# dd if=/lhome/xen-image/vm-10.0.0.123-lv.img of=/dev/vgxen/vm-10.0.0.124 bs=1k

This may also take some time to finish.

6. Change the profile for new DomU on local server

After making the clone of the VBD. We can create a profile for the new DomU. This is the content of /lhome/xen/vm-10.0.0.124/vm.run:

name="10.0.0.124"
cpus=2
memory=2048
disk=['<strong>phy:vgxen/vm-10.0.0.124</strong>,xvda,w' ]
vif=[ 'bridge=eth0' ]
bootloader = "/usr/bin/pygrub"
onreboot = 'restart'
on_crash = 'restart'

The name and disk entry are changed.

7. Start the new virtual machine and configure the new virtual machine

We can start the new DomU and enter it’s console:

# xm create /lhome/xen/vm-10.0.0.124/vm.run
# xm console 10.0.0.213

Or directly:

# xm create -c /lhome/xen/vm-10.0.0.124/vm.run

After logging in vm-10.0.0.124, we can edit the network configuration file:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0

Change the IPADDR to 10.0.0.124 from 10.0.0.123. Then restart eth0:

# ifdown eth0
# ifup eth0

Make sure this interface doesn’t have HWADDR by commenting out the line that specifyHWADDR if we use Xen bridge network.

Log out of vm-10.0.0.124 and then use “Ctrl + ]” to exit the xm console.

The new virtual machine vm-10.0.0.124 which is the clone of vm-10.0.0.123 except the IP and the Dom0 on top of which it is running is ready to use now. This process can be written into one script.

source: http://lintut.com/duplicating-and-backing-up-lvm-backed-xen-domu-from-a-remote-server/

R and Java

http://cran.r-project.org/web/packages/helloJavaWorld/vignettes/helloJavaWorld.pdf

Writing R library

You should start creating a separate directory for your package
(e.g. foo).  As a minimum you need three things:

1) your R functions (as .R files in directory foo/R)

2) documentation of your functions in .Rd format (look "writing R
   extensions" and function prompt(), they must reside in foo/man directory)

3) The DESCRIPTION file (in the foo directory).  Everything else is
   made by R.

Now run "R CMD build foo" in the directory above (where the subdir foo
is lying).

How To Deploy Maven Based War File To Tomcat

Here we show you how to use Maven-Tomcat plugin to package and deploy a WAR file to Tomcat.1. Tomcat Authentication

First, add an user with administrator access right for Tomcat. To add Tomcat user, edit this file – “%TOMCAT_PATH%/conf/tomcat-users.xml“.

Nghề chia sẻ

1. Thông điệp chính
2. Tóm tắt nội dung
3. Đánh giá
4. Ứng dụng

Software engineering skills

An old friend, also a Computer Science professor asked me about the impact of the global financial crisis to the software industry. He is worrying that his students may not be able to get jobs when they graduated. I told him that I think the current crisis will have significant impact to the software industry but to the better because it will eliminate inefficient companies and create many opportunities for well managed companies. Regarding employment, software engineering jobs is still rated on the top five in global demand so he should not have to worry. Assume that his students have the skills that match the demands of the industry.

He told me that he has been teaching Computer Science for many years based on the same curriculum and he did not know anything about software industry needs. I explained to him that the rate of change of software technology is extremely fast. New programming languages are created every year with new tools appear every few month. This fast rate of technology change means that software people are faced with a continuing need to learn new skills all the time.

He seemed surprised:” What are the new skills that my students need to know now?” I told him that there were number of topics that current education had not even mention such as software security, quality control, measurements and metrics, software architecture, software methods, and technology evaluation and transfer. These gaps need to be quickly filled if students want to work in global companies as professional software engineers.

I also told him that due to the economic recession; parents are very selective on the investments in education for their children. They want to make sure that their children will have jobs when they graduated so they carefully review what university has to offer and what programs would benefit their children. In the U.S, applications to admission in top universities had increased significantly in the past few years but many state schools encountered lower enrollment. As the financial crisis began to impact government, budgets were reduced. I have read in newspapers that many classes in state university were cancelled, and some professors were laid off. These things are happening as schools begin to cut costs due to state budget cuts. My friend took out his note and asked me about topics that software students need to learn so I shared with him about my research in global trends at Carnegie Mellon University.

Lat year in 2007, I conducted a survey to determine the needs in software industry. For this research, I sent out questionnaires to over 800 software companies in U.S, Europe, Australia, and Asia. I also visited 15 companies that have important roles in software industry such IBM, Microsoft, Oracle, SAP and interview their managers to determine their current and future needs. Following is the results of my research: 

Topics Software Engineers need to learn between 2009 to 2019:

1) Agile method (i.e. SCRUM, Extreme Programming etc.).

2) Automated Testing

3) Computer Security

4) E-business, especially B2B (business to business)

5) Software as a Service (Saas)

6) SOA (Service-Oriented Architecture)

7) EA (Enterprise Architecture)

8) (BPR )Business Process Reengineering

9) Data mining and Business Intelligence;

10) Data warehouses

11) SCM (Supply Chain Management)

12) ERP (Enterprise Resource Planning)

13) CRM (Customer Resource Management)

14) RUP (Rational Unified Process)

15) UML (unified modeling language)

16) Use Cases

17) OLAP (On-line analytical processing)

18) GUI (Graphical user interface)

19) QFD (Quality function deployment)

20) RAD (Rapid application development)

21) Virtualization

22) Cloud computing

23) Supply-chain integration

24) E-Government

25) Entrepreneurship & Innovation

My friend reviewed his note and asked: “Many topics are relatively new. How can university professors stay current with these latest technologies? How can they learn these new concepts well enough to be effective in their teachings?

I told him that technology knowledge is not something you learn only once but must continue to learn all your life or be a lifelong learner. Due to the fast changing in technology area, knowledge has become an essential requirement to all professionals. As professor at Carnegie Mellon University, I have to attend number of seminars, workshops every year to keep current with technology changes. I have to conduct research and published papers every year in certain topics and have knowledge on some technological fields. We have to stay current so our students can stay current. 

My friend agreed:” That is good for professors because that is their jobs but what’s about software engineers and managers, how can they keep their knowledge current? There are so many things to learn and most people have limited amount of time that can be spent? 

I told him that not every new topic is appropriate for every project. It is important for software engineers and managers to know enough about each topic so they can select the appropriate one for specific projects. In the fast changing time, they need to update their skills by attending short course training like special seminars, workshops so they can make the right decision. That is why I think every software professional must also be lifelong learners. In the highly competitive global market, as inefficient companies are being eliminated, people with obsolete skills will also find it difficult to maintain their position.

----------------------------------------

Prof. Vu

Carnegie Mellon University

source: http://www.segvn.org/forum/mvnforum/viewthread_thread,147