Friday, 8 November 2019

Bash file test operators

Bash file test operators 



-e  FILE      - True if file exists
-a  FILE      - True if file exists  (same as previous, but deprecated)
-r FILE       - True if file exists and is redable by you
-w  FILE     - True if file exists and is writable by you
-x  FILE      - True if file exists and is executable by you

-f   FILE      - True if file exists and file is a regular file (not a directory or device file)


-s   FILE      - True if file exists and not zero size
-d   FILE      - True if file is a directory  
-b   FILE      - True if file is a block device
-c   FILE      - True if file is a characrer device
-h   FILE      - True if file is a symbolic link
-L  FILE      -  True if file is a symbolic link
-S  FILE      -  True if file is a socket
-g  FILE      -  True if set-group-id (sgid) set
-u  FILE      -  True if set-user-id (suid) set
-k  FILE      -  True if sticky bit set
-O  FILE     -  True if you are owner of file
-G  FILE      -  True if group ID of file same as yours
-N  FILE      -  True if file modified since last read
f1 -nt f2        -  True if file f1 is newer than f2
f1 -ot f2        -   True if file f1 is older than f2
f1 -ef f2        -    True files f1 and f2 are hard links to the same file

Example:

device0="/dev/sda2" # / (root directory)
if [ -b "$device0" ] then
     echo "$device0 is a block device."
fi
# /dev/sda2 is a block device.



Wednesday, 30 October 2019

LVM - Data Migration

Use LVM for live data migration

I have to migrate some data to new hard drive. Lets use LVM to do that.

Source: volume group vg_mydata
Destination: /dev/sdd
[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g     0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g     0
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 32.00m

[root@rifle ~]# lvmdiskscan
  /dev/centos_rifle/root   [      <6.20 GiB]
  /dev/sda1                [       1.00 GiB]
  /dev/centos_rifle/swap   [     820.00 MiB]
  /dev/sda2                [      <7.00 GiB] LVM physical volume
  /dev/vg_mydata/lv_mydata [       1.50 GiB]
  /dev/sdb                 [      <1.04 GiB] LVM physical volume
  /dev/sdc                 [     519.34 MiB] LVM physical volume
  /dev/sdd                 [       2.00 GiB]
  4 disks
  1 partition
  2 LVM physical volume whole disks
  1 LVM physical volume

1. Adding /dev/sdd to logical volume manager

[root@rifle ~]# pvcreate /dev/sdd
  Physical volume "/dev/sdd" successfully created.

2. Adding new disk to the volume group

[root@rifle ~]# vgextend vg_mydata /dev/sdd
  Volume group "vg_mydata" successfully extended

[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g     0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g     0
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 32.00m
  /dev/sdd   vg_mydata    lvm2 a--   <2.00g <2.00g

In vg_mydata we have two disks /dev/sdb and /dev/sdc, which have 0 and 32Mb free space respectively.

3.Moving the data from the /dev/sdb to /dev/sdd

[root@rifle ~]# pvmove /dev/sdb /dev/sdd
  /dev/sdb: Moved: 1.14%
  /dev/sdb: Moved: 28.03%
  /dev/sdb: Moved: 44.32%
  /dev/sdb: Moved: 62.88%
  /dev/sdb: Moved: 79.17%
  /dev/sdb: Moved: 95.83%
  /dev/sdb: Moved: 100.00%

Lets check what do we have now:
[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g      0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g   1.03g
  /dev/sdc   vg_mydata    lvm2 a--  516.00m  32.00m
  /dev/sdd   vg_mydata    lvm2 a--   <2.00g 988.00m

[root@rifle ~]# pvdisplay /dev/sdb
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               vg_mydata
  PV Size               <1.04 GiB / not usable 4.70 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              264
  Free PE               264
  Allocated PE          0
  PV UUID               ycc9Uq-9yuu-fxUN-OFmK-ceKU-ZihU-bzUGWT

We have 1.03 Gb free on disk /dev/sdb and all 264 physical extents (PEs) on the disk are free.

4.Removing /dev/sdb from the volume group

[root@rifle ~]# vgreduce vg_mydata /dev/sdb
  Removed "/dev/sdb" from volume group "vg_mydata"

4.Moving data from /dev/sdc to /dev/sdd

[root@rifle ~]# pvmove /dev/sdc /dev/sdd
  /dev/sdc: Moved: 0.83%
  /dev/sdc: Moved: 36.36%
  /dev/sdc: Moved: 72.73%
  /dev/sdc: Moved: 100.00%

Just to check that all data are moved:
[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g      0
  /dev/sdb                lvm2 ---   <1.04g  <1.04g
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 516.00m
  /dev/sdd   vg_mydata    lvm2 a--   <2.00g 504.00m

5.Removing /dev/sdc from the volume group

[root@rifle ~]# vgreduce vg_mydata /dev/sdc
  Removed "/dev/sdc" from volume group "vg_mydata"

6.Removing /dev/sdb and /dev/sdc from logical volume manager

[root@rifle ~]# pvremove /dev/sdb /dev/sdc
  Labels on physical volume "/dev/sdb" successfully wiped.
  Labels on physical volume "/dev/sdc" successfully wiped.

And check what we have now:
[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize  PFree
  /dev/sda2  centos_rifle lvm2 a--  <7.00g      0
  /dev/sdd   vg_mydata    lvm2 a--  <2.00g 504.00m

[root@rifle ~]# df -h /mydata/
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_mydata-lv_mydata  1.5G  1.3G  153M  90% /mydata

All data migrated to the new disk without downtime.

Tuesday, 29 October 2019

Deleting Logical Volumes, Volume Groups and Physical Volumes

Click here to read how to create of the mirrored Logical Group.

Warning: ALL DATA ON LOGICAL VOLUMES, VOLUME GROUPS and PHYSICAL VOLUMES WILL BE LOST!

1.Unmount file system

[root@rifle ~]# umount /mirrored_data

2.Remove logical volume

[root@rifle ~]# umount /mirrored_data
[root@rifle ~]# lvremove /dev/vg_mirror/lv_mirror
Do you really want to remove active logical volume vg_mirror/lv_mirror? [y/n]: y
  Logical volume "lv_mirror" successfully removed

3.Remove physical volumes from the volume group and delete volume group

Check what we have:
[root@rifle ~]# vgs
  VG           #PV #LV #SN Attr   VSize  VFree
  centos_rifle   1   2   0 wz--n- <7.00g     0
  vg_mirror      2   0   0 wz--n-  3.98g  3.98g
  vg_mydata      2   1   0 wz--n- <1.54g 32.00m

Now for vg_mirror I have 2 PV and no one LV (remember step 2)

Remove /dev/sdd from vg_mirror:
[root@rifle ~]# vgreduce vg_mirror /dev/sdd
  Removed "/dev/sdd" from volume group "vg_mirror"

Note: the disk now can be used in another Volume Group.

Remove disk from the Logical Volume Manager:
[root@rifle ~]# pvremove /dev/sdd
  Labels on physical volume "/dev/sdd" successfully wiped.

Delete volume group vg_mirror:
[root@rifle ~]# vgremove vg_mirror
  Volume group "vg_mirror" successfully removed


[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g     0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g     0
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 32.00m
  /dev/sde                lvm2 ---    2.00g  2.00g

Now I have /dev/sde which does not belong to any Volume Group. Free it:

[root@rifle ~]# pvremove /dev/sde
  Labels on physical volume "/dev/sde" successfully wiped.

All Logical Volumes, Volume Groups and Physical Volumes now deleted

LVM - Create Mirrored Logical Volumes

1.Checking which disks to use for mirror

[root@rifle ~]# lvmdiskscan
  /dev/centos_rifle/root   [      <6.20 GiB]
  /dev/sda1                [       1.00 GiB]
  /dev/centos_rifle/swap   [     820.00 MiB]
  /dev/sda2                [      <7.00 GiB] LVM physical volume
  /dev/vg_mydata/lv_mydata [       1.50 GiB]
  /dev/sdb                 [      <1.04 GiB] LVM physical volume
  /dev/sdc                 [     519.34 MiB] LVM physical volume
  /dev/sdd                 [      <1.99 GiB]
  /dev/sde                 [       2.00 GiB]
  5 disks
  1 partition
  2 LVM physical volume whole disks
  1 LVM physical volume

In my case I have two almost identical disks: /dev/sdd and /dev/sde.

2. Creating Physical Volume

[root@rifle ~]# pvcreate /dev/sdd   /dev/sde
  Physical volume "/dev/sdd" successfully created.
  Physical volume "/dev/sde" successfully created.

Checking physical volumes
[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g     0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g     0
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 32.00m
  /dev/sdd                lvm2 ---   <1.99g <1.99g
  /dev/sde                lvm2 ---    2.00g  2.00g

3. Creating Volume Group named vg_mirror

[root@rifle ~]# vgcreate vg_mirror /dev/sdd /dev/sde
  Volume group "vg_mirror" successfully created

Check VG
[root@rifle ~]# vgs
  VG           #PV #LV #SN Attr   VSize  VFree
  centos_rifle   1   2   0 wz--n- <7.00g     0
  vg_mirror      2   0   0 wz--n-  3.98g  3.98g
  vg_mydata      2   1   0 wz--n- <1.54g 32.00m

I created new virtual group vg_mirror with two PVs and no LVs in it.

4.Creating a new mirrored Logical Volume 

[root@rifle ~]# lvcreate -m 1 -l100%FREE -n lv_mirror vg_mirror
  Logical volume "lv_mirror" created.

Where parameters are:
-m 1                - have one mirror copy of data
-l100%FREE  - use all free space on the volume group

Check status of the new logical volume:
[root@rifle ~]# lvs /dev/vg_mirror/lv_mirror
  LV        VG        Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  lv_mirror vg_mirror rwi-a-r--- 1.98g                                    100.00

100.00 - means that both copies are in sync

5.Creating file systems

[root@rifle ~]# mkfs -t ext4 /dev/vg_mirror/lv_mirror
mke2fs 1.42.9 (28-Dec-2013)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
129792 inodes, 519168 blocks
25958 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=532676608
16 block groups
32768 blocks per group, 32768 fragments per group
8112 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

6.Creating mount point and mount

[root@rifle ~]# mkdir /mirrored_data
[root@rifle ~]# mount /dev/vg_mirror/lv_mirror /mirrored_data/

Checking file system size:
[root@rifle ~]# df -h /mirrored_data/
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_mirror-lv_mirror  2.0G  6.0M  1.8G   1% /mirrored_data


LVM - extending volume groups and logical volumes

🔖  Note 1: How to create Physical Volumes, Volume Groups and Logical Volumes you can read here.

🔖 Note 2: In the example below I assume that we are using linear volumes.


Extend existing Logical Volume

Each of logical volumes divided on LEs (logical extents), and each volume groups have PEs (physical extents).

To extend Logical Volume we have to have enough free PE on the volume group.

1.1 Lets check what disks do we have:

[root@rifle ~]# lvmdiskscan
  /dev/centos_rifle/root   [      <6.20 GiB]
  /dev/sda1                [       1.00 GiB]
  /dev/centos_rifle/swap   [     820.00 MiB]
  /dev/sda2                [      <7.00 GiB] LVM physical volume
  /dev/vg_mydata/lv_mydata [       1.00 GiB]
  /dev/sdb                 [      <1.04 GiB] LVM physical volume
  /dev/sdc                 [     519.34 MiB]
  3 disks
  1 partition
  2 LVM physical volume whole disks
  1 LVM physical volume

So we have free disk /dev/sdc

1.2 Check how many free PEs are on the /dev/sdc:

[root@rifle ~]# pvdisplay /dev/sdc
  --- Physical volume ---
  PV Name               /dev/sdc
  VG Name               vg_mydata
  PV Size               519.34 MiB / not usable 3.34 MiB
  Allocatable           yes
  PE Size               4.00 MiB
  Total PE              129
  Free PE               129
  Allocated PE          0
  PV UUID               SQEy8B-b3df-rIF2-1dgu-67FV-dM3a-UuarFu

There are 129 free PEs.

1.3 Adding disk  /dev/sdc to the volume group vg_mydata

Creating Physical Volume from /dev/sdc:
[root@rifle ~]# pvcreate /dev/sdc
Physical volume "/dev/sdc" successfully created.

Adding Physical Volume /dev/sdc to the Volume Group vg_mydata:
[root@rifle ~]# vgextend vg_mydata /dev/sdc
  Physical volume "/dev/sdc" successfully created.
  Volume group "vg_mydata" successfully extended

Check how many free space do we have on the Volume Group now:
[root@rifle ~]# vgs
  VG           #PV #LV #SN Attr   VSize  VFree
  vg_mydata      2   1   0 wz--n- <1.54g 548.00m

[root@rifle ~]# pvs
  PV         VG           Fmt  Attr PSize   PFree
  /dev/sda2  centos_rifle lvm2 a--   <7.00g      0
  /dev/sdb   vg_mydata    lvm2 a--    1.03g  32.00m
  /dev/sdc   vg_mydata    lvm2 a--  516.00m 516.00m

Now we have 2 Physical Volumes and 516Mb (there are two figures: 548M and 516M      516 = 129 PE multiply 4K PE sizefree space on our Volume Group vg_mydata

File system space before expansion:
[root@rifle ~]# df -h /mydata/
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_mydata-lv_mydata  976M  2.6M  907M   1% /mydata

1.4 Extend Logical Volume and resize the file system

[root@rifle ~]# lvextend -L +516M  -r  /dev/vg_mydata/lv_mydata
  Size of logical volume vg_mydata/lv_mydata changed from 1.00 GiB (256 extents) to 1.50 GiB (385 extents).
  Logical volume vg_mydata/lv_mydata successfully resized.
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/mapper/vg_mydata-lv_mydata is mounted on /mydata; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/mapper/vg_mydata-lv_mydata is now 394240 blocks long.

🔖  Note 3: Parameter -r means also resize file system.

What size do we have now:
[root@rifle ~]# df -h /mydata/
Filesystem                       Size  Used Avail Use% Mounted on
/dev/mapper/vg_mydata-lv_mydata  1.5G  3.0M  1.4G   1% /mydata

Lets check the map of LV:
[root@rifle ~]# lvdisplay /dev/vg_mydata/lv_mydata -m
  --- Logical volume ---
  LV Path                /dev/vg_mydata/lv_mydata
  LV Name                lv_mydata
  VG Name                vg_mydata
  LV UUID                23DwLN-jLgw-cbbr-vaXw-Pgc2-VGQV-V43H9G
  LV Write Access        read/write
  LV Creation host, time rifle, 2019-10-28 13:01:27 +0300
  LV Status              available
  # open                 1
  LV Size                1.50 GiB
  Current LE             385
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

  --- Segments ---
  Logical extents 0 to 263:
    Type                linear
    Physical volume     /dev/sdb
    Physical extents    0 to 263

  Logical extents 264 to 384:
    Type                linear
    Physical volume     /dev/sdc
    Physical extents    0 to 120

Logical Extents from 0 to 263 are on disk /dev/sdb and LEs from 264 to 384 are on /dev/sdc   

Monday, 28 October 2019

The Logical Volume Manager - Creating Physical Volumes (PV),Volume Groups (VG), Logical Volumes (LV)

Benefits of using LVM

  • Flexible capacity - When using logical volumes, file systems can extend across multiple disks, since you can aggregate disks and partitions into a single logical volume.
  • Resizeable storage pools - You can extend logical volumes or reduce logical volumes in size with simple software commands, without reformatting and repartitioning the underlying disk devices.
  • Online data relocation - To deploy newer, faster, or more resilient storage subsystems, you can move data while your system is active. Data can be rearranged on disks while the disks are in use. For example, you can empty a hot-swappable disk before removing it.
  • Convenient device naming  - Logical storage volumes can be managed in user-defined groups, which you can name according to your convenience.
  • Disk striping  -You can create a logical volume that stripes data across two or more disks. This can dramatically increase throughput.
  • Mirroring volumes - Logical volumes provide a convenient way to configure a mirror for your data.
  • Volume Snapshots  - Using logical volumes, you can take device snapshots for consistent backups or to test the effect of changes without affecting the real data.

Levels of abstraction


    Logical Volumes (LV) -  a volume group is divided up into logical volumes.
    There are three types of LVM logical volumes: linear volumes, striped volumes, and mirrored volumes.
    Volume Group (VG) - Physical volumes are combined into volume groups.
    Physical Volume (PV) - is a block device such as a partition or
    whole disk

    Three types of volumes

    linear volume  - aggregates multiple physical volumes into one logical volume.

    striped volume  - enhances performance by writing data to a predetermined number of physical volumes in round-round fashion. With striping, I/O can be done in parallel. 

    mirrored volumes - maintain identical copies of data on different devices.

    Detailed information about different types of volumes here

    Commands

    lvmdiskscan - show all storage devices which have ability to be used with LVM.

    [root@rifle ~]# lvmdiskscan
      /dev/centos_rifle/root [      <6.20 GiB]
      /dev/sda1              [       1.00 GiB]
      /dev/centos_rifle/swap [     820.00 MiB]
      /dev/sda2              [      <7.00 GiB] LVM physical volume
      /dev/sdb               [      <1.04 GiB]
      /dev/sdc               [     519.34 MiB]
      4 disks
      1 partition
      0 LVM physical volume whole disks
      1 LVM physical volume


    lsblk               -  lists information about all or the specified block devices. It queries the /sys virtual file system to obtain the information that it displays.
    [root@rifle ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 8G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 7G 0 part ├─centos_rifle-root 253:0 0 6.2G 0 lvm / └─centos_rifle-swap 253:1 0 820M 0 lvm [SWAP] sdb 8:16 0 1G 0 disk sdc 8:32 0 519.3M 0 disk sr0 11:0 1 1024M 0 rom
    lsblk -p           - show all paths to the devices

    [root@rifle ~]# lsblk -p
    NAME                              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    /dev/sda                                8:0    0     8G  0 disk
    ├─/dev/sda1                         8:1    0     1G  0 part /boot
    └─/dev/sda2                         8:2    0     7G  0 part
      ├─/dev/mapper/centos_rifle-root 253:0    0   6.2G  0 lvm  /
      └─/dev/mapper/centos_rifle-swap 253:1    0   820M  0 lvm  [SWAP]
    /dev/sdb                            8:16   0     1G  0 disk
    /dev/sdc                            8:32   0 519.3M  0 disk
    /dev/sr0                           11:0    1  1024M  0 rom

    fdisk -l            - shows extended information about all disks
    df -h                - shows the storage which attached to your linux systemin human                          readable format

    1.Create one or more physical volumes. 

    pvcreate initializes the disk for use by LVM and lists what we created

    [root@rifle ~]# pvcreate /dev/sdb
    Physical volume "/dev/sdb" successfully created.

    [root@rifle ~]# pvs
      PV         VG           Fmt  Attr PSize  PFree
      /dev/sda2  centos_rifle lvm2 a--  <7.00g     0
      /dev/sdb                lvm2 ---  <1.04g <1.04g

    2.Create a volume group from those one or more physical volumes and view volume groups.

    [root@rifle ~]# vgcreate vg_mydata /dev/sdb
      Volume group "vg_mydata" successfully created

    [root@rifle ~]# vgs
      VG           #PV #LV #SN Attr   VSize  VFree
      centos_rifle   1   2   0 wz--n- <7.00g    0
      vg_mydata      1   0   0 wz--n-  1.03g 1.03g

    3.Create one or more logical volumes from the volume group. 

    [root@rifle ~]# lvcreate -L 1G -n lv_mydata vg_mydata
    Logical volume "lv_mydata" created.
    [root@rifle ~]# lvs     # you can also use lvdisplay command
    [root@rifle ~]# lvs
      LV        VG           Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      root      centos_rifle -wi-ao----  <6.20g
      swap      centos_rifle -wi-ao---- 820.00m
      lv_mydata vg_mydata    -wi-ao----   1.00g

    We also can let the command use percentage of the free space:
    lvcreate -l 100%FREE -n lv_mydata vg_mydata

    Now we can threat the LV above as usual disk partition

    4.Create a file system on the new partition

    [root@rifle ~]# mkfs -t ext4 /dev/vg_mydata/lv_mydata
    mke2fs 1.42.9 (28-Dec-2013)
    Filesystem label=
    OS type: Linux
    Block size=4096 (log=2)
    Fragment size=4096 (log=2)
    Stride=0 blocks, Stripe width=0 blocks
    65536 inodes, 262144 blocks
    13107 blocks (5.00%) reserved for the super user
    First data block=0
    Maximum filesystem blocks=268435456
    8 block groups
    32768 blocks per group, 32768 fragments per group
    8192 inodes per group
    Superblock backups stored on blocks:
            32768, 98304, 163840, 229376

    Allocating group tables: done
    Writing inode tables: done
    Creating journal (8192 blocks): done
    Writing superblocks and filesystem accounting information: done

    5.Mount the new file system

    [root@rifle ~]# mkdir /mydata #create new directory [root@rifle ~]# mount /dev/vg_mydata/lv_mydata /mydata/ [root@rifle ~]# df -h /mydata/ #show free space Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_mydata-lv_mydata 976M 2.6M 907M 1% /mydata

    6.Mount the new partition in the boot time

    Edit /etc/fstab and add the following line at the end:
    /dev/vg_mydata/lv_mydata   /mydata   ext4 defaults  0 0

    You can test mount by dismounting /mydata and mount it againg by using the following commands:
    [root@rifle ~]# umount /mydata
    [root@rifle ~]# mount -a    #mount filesystems using /etc/fstab configuration file
    [root@rifle ~]# mount |grep "/mydata"  #check if /mydata is mounted
    /dev/mapper/vg_mydata-lv_mydata on /mydata type ext4  (rw,relatime,seclabel,data=ordered)

    7. Deleting Logical Volumes, Volume Groups and Physical Volumes





    Wednesday, 4 September 2019

    Linux system logging


    • Linux uses syslog standard for message logging.
    • Uses facilities and priorities to categorize the messages.

    Facility code Keyword Description

    0 kern          Kernel messages
    1 user          User-level messages
    2 mail          Mail system
    3 daemon  System daemons
    4 auth          Security/authentication messages
    5 syslog  Messages generated internally by syslogd
    6 lpr          Line printer subsystem
    7 news  Network news subsystem
    8 uucp  UUCP subsystem
    9 cron          Clock daemon
    10 authpriv  Security/authentication messages
    11 ftp           FTP daemon
    12 ntp           NTP subsystem
    13 security   Log audit
    14 console   Log alert
    15 solaris-cron Scheduling daemon
    16–23 local0 – local7 Locally used facilities


    The priorities (in order of severity) are:

    Code    Keyword
    0            emerg (or panic)
    1            alertdebug
    2            crit
    3            err (or error)
    4            warning (or warn)
    5            notice
    6            info
    7            debug

    Syslog servers

    syslogd
    rsyslog
    syslog-ng

    rsyslog

     /etc/rsyslog.conf:
    include /etc/rsyslog.d/*.conf

    Logging rules

    Consist of two fields: selector field and action field

    Example:
    #Log anything 'warn' or higher. 
    #Exclude authpriv, cron, mail, and news. These are logged elsewhere. 
    #Don't log private authentication messages! 
    *.warn;\ authpriv.none;cron.none;mail.none;news.none    -/var/log/syslog 

    #The authpriv file has restricted 
    access. authpriv.* /var/log/secure 

    #Log all the mail messages in one place. Use caching mode to improve I/O performance
    #- before /var/log/maillog means use caching mode. you may loose some messages if system crash
    mail.* -/var/log/maillog     

    #Log cron stuff 
    cron.* /var/log/cron 

    #Everybody gets emergency messages 
    *.emerg * 

    #Save news errors of level crit and higher in a special file. 
    uucp,news.crit /var/log/spooler

    Logger command

    logger - a shell command interface to the syslog(3) system log module

    logger [options] message

    Where important options are:
    -p  FACILITY.PRIORITY
    -t   tag
     
    Valid facility names are:
    auth
    authpriv   for security information of a sensitive nature
    cron
    daemon
    ftp
    kern       cannot be generated from userspace process, automatically converted to user
    lpr
    mail
    news
    syslog
    user
    uucp
    local0 to local7
    security   deprecated synonym for auth

    Valid priority names are:

    emerg
    alert
    crit
    err
    warning
    notice
    info
    debug
    panic     deprecated synonym for emerg
    error     deprecated synonym for err
    warn      deprecated synonym for warning

    Logger examples


    logger System rebooted
    logger -p local0.notice -t HOSTIDM -f /dev/idmc
    logger -n loghost.example.com System rebooted
    logger -p mail.info -t test-message "Test message here."

    Tuesday, 3 September 2019

    How To Use Systemctl to Manage Systemd Services and Units

    Starting and Stopping Services

    sudo systemctl start application.service                   or
    sudo systemctl start application

    Restarting and Reloading

    sudo systemctl restart application.service

    if application has ability to reload its configuration files:

    sudo systemctl reload application.service

    universal command (if you don't know is there in application ability to reload):

    sudo systemctl reload-or-restart application.service

    Enabling and Disabling Services

    systemctl enable application.service

    systemctl disable application.service

    Change the “runlevels”/targets after booting

    systemctl isolate multi-user.target

    Change the boot “runlevels”/targets

    Runlevels - Targets

    0          -  poweroff.target
    1          -  rescue.target
    2, 3, 4  -  multi-user.target
    5          -  graphical.target 
    6          -  reboot.target  

    Make “multi-user.target” the default “runlevel”


    systemctl enable multi-user.target
    systemctl set-default multi-user.target

    Checking the Status of Services

    systemctl status application.service

    systemctl is-active application.service

    systemctl is-enabled application.service

    systemctl is-failed application.service

    System State Overview

    Listing Current Units
    systemctl list-units

    systemctl

    systemctl list-units --all

    Use the --state= flag to indicate the LOAD, ACTIVE, or SUB states

    systemctl list-units --all --state=inactive

    systemctl list-units --type=service

    Listing All Unit Files

    systemctl list-unit-files

    Displaying Dependencies

    systemctl list-dependencies sshd.service

    Checking Unit Properties


    systemctl show sshd.service

    Shortcuts

    systemctl rescue                      - put system into single-user mode

    systemctl halt                          - will halt the system - meaning the system will       shutdown and at the end stop at a screen with the last message beeing something like "System halted".

    systemctl poweroff                 -  will power off the system - meaning the system  will shutdown and at the end power off (sends an ACPI command to the board, then to the PSU, to cut the power)

    systemctl reboot                      - reboot the system


    Monday, 2 September 2019

    Shell history in Linux

    Shell history stored in memory and on disk. 

    In files:

    • ~/.bash_history
    • ~/.histfile
    • ~/.history

    history Command

    history                              - displays the  shell history
    history 5                            - print last 5 commands
    history |grep sysctl       
    history -d line_number     - delete line line_number from the history
    history -c                            - clear all history
    history -w                            - write to history file (Usually the history file is                                                         written to upon logout)
    Ctrl-r                                    - reverse history search (Ctrl-r again - next search)
    Enter                                    - execute command
    Arrows                                - change the command
    Gtrl-g                                  - cancel the search

    echo "secret command";history -d $(history 1)     - execute single command                                                                                            without being logged to                                                                                             history file

    Run all command without logging:
    [user1@rifle ~]$ unset HISTFILE
    [user1@rifle ~]$ echo $HISTFILE

    [user1@rifle ~]$

    History control

    !To make changes permanent you have to put the export commands below to your ~/bash_rc file. See more here...

    Controls how many commands store in memory

    esport HISTSIZE=1000

    Ignore duplicate commands in history

    export HISTCONTROL=ignoredups

    Filter some commands from history

    export HISTIGNORE='ls -l:pwd:date:' 

    List all commands with date and time

    export HISTTIMEFORMAT='%F %T  '

    Using the history

    Show last commands:


    !N                             - repeat command line number N
    !239                           - in above example execute command pwd 
    !!                               - repeat previous command
    !systemctl:p              - Show last command that begins from systemctl
    !string  or !? string   - repeat the most recent command starting with "string"
    !groupi                     - in example above run command yum groupinstall                                                  "Development Tools"
    vi !239:1                  - edit file one.txt (open vi editor with first parameter of                                              history line 239)
    !N^                             - first argument of the command number N. For the                                                  previous example will be            vi !239^
    !$                                - last argument
    ls  !239$                     - resulting command will be           ls examples/




    Wednesday, 28 August 2019

    Linux su and sudo commands - what is different?

    Difference


       There are two Linux command to switch to other users account or execute command as another user: su and sudo.

        The key difference is that to use su commands you have to know another user password, but to use sudo your don't to need know the password of the other user (when using sudo you are prompted for your password).

        The sudo configuration controlled by the system administrators (file /etc/sudoers).

    Examples:

    su username                      - run command as username
    su - username                   - login to username account with username's                                                              environment
    su - user1 -c ls                -  execute ls command as user1 in user1's home                                                           directory


    sudo -l                                - list all available commands
    sudo command                   - run command as root
    sudo -u user command       - run command as user
    sudo su                               - switch to superuser account
    sudo su  -                            - switch to superuser account with root's environment

    Sudo configuration


    visudo command - starts vi to edit /etc/sudoers file (root privileges required)

    sudoers format:
    user (host)=(users)[NOPASSWD:]commands

    examples:
    admin ALL=(ALL)[NOPASSWD:]ALL  - allow user admin from all hosts to                                                                            execute all command as root without
                                                                      password
    operator webserver1=/usr/sbin/poweroff  - allow user operator to power off                                                                               server webserver1


    Monday, 26 August 2019

    Processes and job control in Linux

    Listing process and information

    ps command - display process status

    - e                 - display all processes
    -f                   - full format listing
    -u username  - owned by user username
    -p PID           - information for process with process ID

    Some examples:
    ps -A or ps -e                - display every active process
    ps -ef or ps -eF              - display a full format listing
    ps -x                              - display all processes owned by you
    ps -fU user1                  - display all processes owned by user1
    ps -fG apache                - list all processes owned by apache group
    ps -eH                            - display a process tree
    ps -e --forest                  - display a process tree

    pstree - display processes in a tree format
    top     - interactive process viewer
    htop   - interactive process viewer (have to be installed from EPEL repository)

    Background and foreground processes

    Start background process
    command &

    Use jobs command to see list of background processes.
    [root@rifle ~]# vi &
    [1] 3669
    [root@rifle ~]# vi &
    [2] 3670
    [1]+  Stopped                 vi
    [root@rifle ~]# vi &
    [3] 3671
    [2]+  Stopped                 vi
    [root@rifle ~]# jobs
    [1]   Stopped                 vi
    [2]-  Stopped                 vi
    [3]+  Stopped                 vi


    Plus indicates current job.
    Current job also can be referred by  %% or %+
    Previous job %-
    [root@rifle ~]# jobs %%
    [3]+  Stopped                 vi


    Ctrl-z - suspend foreground process
    Ctrl-c - kill the foreground process
    bg [%number] - background a suspended process, without job number backgroung last stopped or suspended job.
    fg [%number] - foreground a background process, without number foreground current job.
    kill                  - kill process by job number or PID
    kill [sig] pid   - send a signal to a process with PID
                             by default sig is TERM (15)
    kill -9 pid       - if process does not terminated send a KILL signal to process
    kill -l             - show all signals

    Environment variables in Linux

    Environment variables

    Environment variable is a storage location where is name and value. They often affect the way programs behave.

     printenv command – Print all or part of environment.
    env command – Print all exported environment or run a program in a modified environment.
    set command – Print the name and value of each shell variable.
    printenv without arguments print all variables.
    printenv name_of_variable - print only this variable
    example:
     [root@rifle ~]# printenv HISTSIZE
    1000
      The same result you can get using the echo command. The variable name has to be used with $ .
    [root@rifle ~]# echo $HISTSIZE
    1000
    !Some information about used variables can be found in man pages of the command.
    ! Environment variables names are case sensitive.

    Creating/change variables

    export VAR='value'
    export EDITOR='vi'

    Removing variables

    unset  var

    Preserve variables from session to session

    Save them to your personal ~/.bash_profile file

    Customizing the prompt with PS1

    Bash, ksh and sh use enviroment variable $PS1.
    Control command for $PS1 are:
    d - the date in "Weekday Month Date" format (e.g., "Tue May 26")
    e - an ASCII escape character (033)
    h - the hostname up to the first .
    H - the full hostname j - the number of jobs currently run in background
    l - the basename of the shells terminal device name
    n - newline
    r - carriage return
    s - the name of the shell, the basename of $0 (the portion following the final slash)
    t - the current time in 24-hour HH:MM:SS format
    T - the current time in 12-hour HH:MM:SS format
    @ - the current time in 12-hour am/pm format
    A - the current time in 24-hour HH:MM format
    u - the username of the current user
    v - the version of bash (e.g., 4.00)
    V - the release of bash, version + patch level (e.g., 4.00.0)
    w - Complete path of current working directory
    W - the basename of the current working directory
    ! - the history number of this command
    # - the command number of this command
    $ - if the effective UID is 0, a #, otherwise a $
    nnn - the character corresponding to the octal number nnn
    \ - a backslash
    [ - begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
    ] - end a sequence of non-printing characters

    Standard $PS1
    [root@rifle ~]# echo $PS1

    [\u@\h \W]\$

    Adding date and time to the prompt:
    [root@rifle ~]#PS1='[\d \t \u@\h\W\]\$'
    [Mon Aug 26 10:30:01 root@rifle~#

    Set color to red if use root account:
    PS1='${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '

    To make changes in $PS1 permanent you have to add the PS1 variable to apropriate ~/.bash_profile file. 

    Add the following line to the end of /root/.bash_profile file:

    export PS1="${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ "


    To simplify $PS1 construction you can use the following link:







    Saturday, 24 August 2019

    Comparing the contents of files

    diff file1 file2 - compare two files


    sdiff file1 file2 - side by side comparison


    vimdiff file1 file2 - highlight differences in vim

    In vim use the following:
    CTRL-w w go to next window
    :q  - close current window
    :qa - close both files
    :q!  - quit vim 

    Linux redirection

    > - redirects standard output to file, overwrite a file

    >> - appends standard output to file

    < - redirects input from a file

    Linux Input/Output

    0 - stdin standard input

    1 - stdout standard output

    2 - stderr standard error

    Combine standard error and standard output

    2>&1

    Redirect standard output and standard output to file

    $command &>file

    Linux wildcards and patterns

    Wildcards and patterns are used in any of the file manipulation commands (cp, mv and so on)

    Windcards

    * - matches zero or more characters an file names
    ? - matches exactly one character in file names

    Patterns

    [pattern] - Matches exactly one character in the pattern.

    Examples

    ca[ls]
    Any of those file names will match:
    cal
    cas

    ca[ls]*
    Any of those file names will match:
    cal
    cas
    calls.txt
    casts

    Ranges

    [g-f] - matches any files that starts with g,e,f
    [2-5] - matches any files that starts with 2,3,4,5

    Named classes

    [[:alpha:]] - matches alphabetic letters in any case
    [[:alphanum:]] - matches alphanumeric characters
    [[:digit:]] - matches any digits from 0 to 9
    [[:lower::]] - lowercase
    [:upper:]] - uppercase
    [[:space:]] - spaces, tabs and new line characters


    Tuesday, 20 August 2019

    How to get back to previous directory

    Very often I need to "jump" back to previous directory. There is a form of cd command to allow this: 
    #cd -

    [root@rifle ~]# pwd
    /root
    [root@rifle ~]# cd /etc/postfix/
    [root@rifle postfix]# pwd
    /etc/postfix
    [root@rifle postfix]# cd -
    /root

    Tuesday, 13 August 2019

    Search help for the linux commands

    Search help for the linux commands

    Sometimes you need to search what certain linux command does or what arguments the command needs.

    There are several ways:

    1. Ask coomand itself for help

    Usually by adding --h or --help to command.
    [root@rifle ~]# ls --help
    Usage: ls [OPTION]... [FILE]...
    List information about the FILEs (the current directory by default).
    Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.

    Mandatory arguments to long options are mandatory for short options too.
      -a, --all                  do not ignore entries starting with .
    .
    .
    .

    Some commands needs -help parameter.
    [root@rifle ~]# zip -h
    Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
    Zip 3.0 (July 5th 2008). Usage:
    zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
    .
    .
    .

      -A, --show-all           equivalent to -vET
      -b, --number-nonblank    number nonempty output lines, overrides -n

    2.Using man command

    [root@rifle ~]# man ls

    3.Using man command --k parameter

    [root@rifle ~]# man -k sysctl
    _sysctl (2)          - read/write system parameters
    sysctl (2)           - read/write system parameters
    sysctl (8)           - configure kernel parameters at runtime
    sysctl.conf (5)      - sysctl preload/configuration file
    sysctl.d (5)         - Configure kernel parameters at boot
    systemd-sysctl (8)   - Configure kernel parameters at boot
    systemd-sysctl.service (8) - Configure kernel parameters at boot

    4. Using apropos or whatis commands

    [root@rifle ~]# whatis ls
    ls (1)               - list directory contents
    ls (1p)              - list directory contents

    5.Search the web






    cat and tac commands

    cat and tac commands

    cat -  cat - concatenate files and print on the standard output
    tac - concatenate and print files in reverse

    [root@rifle ~]# cat one.txt
    There are just some text lines.
    And again...
    And again...
    [root@rifle ~]# tac one.txt
    And again...
    And again...
    There are just some text lines.