• Follow us

Internet

How to Monitor the Progress of Linux Commands (With pv and progress)

Linux terminal prompt on a laptopFatmawati Achmad Zaenuri/Shutterstock.com

Instead of flying blind, use the Linux pv and progress commands to track a command’s progress. These utilities will give you progress bars for commands that don’t normally have any. You’ll see an estimated time until completion, too.

If you are on a long haul flight on an aircraft without video screens in the seatbacks, it isn’t easy to know how far through your journey you are. You know when you took off. You know how long the flight is expected to take. But how do you know if you are on track, on time, or way behind schedule? If you don’t want to watch the in-flight movie you can usually switch your video screen to show a map with the position of your aircraft on it. You also get some statistics, such as an expected time of arrival (ETA), which is great.

Starting a command from the terminal window can sometimes feel like a long-haul flight without a video screen. You have nothing to give any indication if all is well or if the process has hung, nor how close to completion it is. A flashing cursor isn’t very informative.

The pv and progress commands give you some statistics and a little visual feedback. You can see how close the process is to complete. That means you get an ETA for your running processes. Compared with staring at a cursor, that wins hands down.

Installing pv

You must install pv.

To install pv on Ubuntu use this command:

sudo apt-get install pv

sudo apt-get install pv in a terminal window

To install pv on Fedora use this command:

sudo dnf install pv

sudo dnf install pv in a terminal window

To install pv on Manjaro use this command:

sudo pacman -Syu pv

sudo pacman -Syu pv in a terminal window

Using pv

pv stands for pipe viewer. Piping has to be involved in the command somewhere. Here’s an example where we’re piping an ISO image through zip to make a compressed zip file of the ISO.

To slow the commands down enough that a screenshot could be taken, some of the files in the examples used for this article were stored on an old, slow, external USB called SILVERXHD.

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso | zip > gparted.zip

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso | zip > gparted.zip in a terminal window

The information pv gives us can be seen in the bottom line of the display.

pv output for creating a zip file in a terminal window

From left to right, the information that is displayed is:

The data transferred so far. The time elapsed fo far. The data transfer rate (throughput). A progress bar and a percentage completed figure. The estimated time left before completion (ETA). Copying a File with pv

To copy a file with output from pv, use this command:

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso > gparted.iso

pv /media/dave/SILVERXHD/gparted-live-1.0.0-1-amd64.iso | gparted.iso in a terminal window

We get a progress report as the file is copied.

copying a file with pv in a te terminal window

Copying Multiple Files with pv

To copy multiple files and folders with pv we need to use a little trick. We use tar to move the files for us.

tar -c help-files/ | pv | tar -x -C Documents/

tar -c help-files/ | pv | tar -x -C Documents/ in a terminal window

The tar -c help-files/ portion of the command instructs tar to create (-c) an archive of the files in the help-files folder.  This is piped through pv so that we get a display of the progress. It is then piped back into tar for the last portion of the command. The archive is extracted (-x) and the directory is changed (-C) to Documents before the extraction.

So, the files and folders that are in help-files are copied to the Documents folder, with a progress display.

copying files with pv and tar in a terminal window

The output is slightly different this time.

We do not get an ETA. The progress bar now displays a moving indicator. It shows that the process is active, but it doesn’t grow from left to right like a traditional progress bar. pv is limited to displaying the information it can extract from the process that is being piped.

Using pv and tar to Create an Archive

Copying files with pv and tar does not leave us with an archive file. A sort of “virtual” archive is created by tar, which is fed straight back into tar to extract the files. If our objective is to copy files, that is accomplished. But what if we want to create an archive file?

We can still use tar to create an archive file and get a progress report from pv. The options used with tar are -c (create archive), -z (compress with gzip) and -f (filename of the archive).

Note that we’re using - as the filename, which causes tar to use stdout, and to write its output to the terminal window. We don’t see that output because it is piped through pv.

The actual name of the archive is going to be the filename that we pipe the output from pv into. In this case, it is “help-files.tgz”.

tar -czf - ./help-files/ | pv > help-files.tgz

tar -czf - ./help-files/ | pv > help-files.tgz in a terminal window

We get the same progress indicators as before, and the archive file is created for us.

output of tar and pv in a terminal window

RELATED: How to Compress and Extract Files Using the tar Command on Linux

The pv Display Options

There are a number of options you can use with pv to change the details of its report.

If you use any of these options, all of the other options are turned off. So if you want to have three of the display options in use, then you need to specify those three options.

Using pv without any options is the same as using the -pterb options.

-p: display the percentage complete. This is the progress bar and the percentage completed figure. -t: display the elapsed time. -e: display the ETA. -r: display the rate of data transfer. -b: display the byte count (data transferred so far). -n: display the percentage as an integer. This prints the percentage completed as an integer figure, with each new update on a new line.

Let’s repeat the last command and pass the -p (percentage completed) option to pv.

tar -czf - ./help-files/ | pv - p > help-files.tgz

tar -czf - ./help-files/ | pv - p > help-files.tgz in a terminal window

This turns off all of the other display options. pv provides the percentage completed element only.

Because pv doesn’t get a percentage completed figure from tar, the progress bar is replaced with a moving indicator. There is no percentage figure.

output from pv and tar in a terminal window

Using pv With wc

We can use pv to pipe a  text file (or files) into wc. wc will then count the carriage returns, characters, and words and pv will give us a progress report.

Here we are piping all of the “.page” files in the help-files directory into wc.

pv ./help-files/*.page | wc in a terminal window

When wc completes we can see our count of carriage returns (lines), characters and words from all of the “.page” files in the help-files folder.

output from pv and wc in a terminal window

Installing the progress Command

The progress command gives the same sort of useful information as pv, but it works with a specific set of Linux commands.

To install progress in Ubuntu, use this command:

sudo apt-get install progress

sudo apt-get install progress in a terminal window

To install progress in Fedora, use this command:

sudo dnf install progress

sudo dnf install progress in a terminal widow

To install progress in Manjaro, use this command:

sudo pacman -Syu progress

sudo pacman -Syu progress in a terminal window

The Commands progress Works With

Typing progress in a terminal window and pressing Enter will give you a list of the commands that progress works with.

progress

output of progress commmand in a terminal window

Using Progress With Pipes

There are two techniques we can use to monitor commands with progress. The first is to use pipes.

The tar command is in the list of supported commands that progress can monitor, so let’s use tar.

The options we’ll use are the standard -c (create archive), -z (compress with gzip) and -f (filename) options. We’re going to create a compressed archive of everything in the help-files folder, and the archive will be named “help.tgz”.

We’re piping that into progress and using the -m (monitor) option so progress keeps reporting on the process until it has completed.

tar -czf help.tgz ./help-files/ | progress -m

tar -czf help.tgz ./help-files/ | progress -m in a terminal window

The terminal window will show the progress of the tar command as it creates the archive.

progress command monitoring tar in a terminal window

As each file is processed, it is listed, with the following information:

The process ID. The process name. Percentage completed. Data processed and total size of the file. Data rate (throughput). Estimated time remaining (ETA).

You might be surprised to see a second data set appear. This first data set is for tar. The second one is for gzip. tar calls gzip to perform the compression. Because gzip is in the list of supported commands, progress reports on it.

Using Progress in Continual Monitor Mode

You can use progress in a real-time continual monitor mode by using the -M (monitor) option.

Type the following command in a terminal window:

progress -M

progress -M in a terminal window

progress will report that there are no commands running for it to monitor. But you are not returned to the command line. progress waits until a command that it can monitor starts. It will then automatically start reporting on it.

progress waiting for a command to start in a terminal window

In another terminal window, type a command that is in the list of commands that progress can monitor.

We’re going to use cat. Commands that are over too quickly won’t register with progress, so we’ll list the contents of a very long text file.

cat words.page

cat words.page in a terminal window

In the terminal window with progress in it, you’ll see statistics for the cat command as it executes and works towards completion.

progress reporting on cat in a terminal window

When cat finishes listing the file progress returns to its waiting state.

Each time one of the commands it can report on performs a sizeable task, progress will automatically monitor it and report on it.

That’s pretty neat.

100% Completed

Take the guesswork out of wondering how a long-running command is doing, and take a break from contemplating your cursor with pv and progress .

Read More



Leave A Comment

More News

TechRadar: Internet news

Fake VPN website delivers malware New! 2019-08-20 14:00:42Cloned websites used to infect user's devices with banking Trojans.

Get hands on with ASUS’ ROG Phone II New! 2019-08-20 13:53:25Mobile gaming may be most famous for free-to-play, hyper-casual apps, but the success of mobile ports of blockbuster games like PUBG and Fortnite prov

The Apple Card is here in the US, New! 2019-08-20 13:50:53Apple says that the Apple Card is now open to anyone in the US through the Wallet iPhone app.

The best Windows tablets 2019: all of the New! 2019-08-20 13:35:37Brace yourself for the best Windows tablets to take on the go for 2019.

Best Acer laptops 2019 New! 2019-08-20 13:34:41Here are our top picks for the best Acer laptops this 2019. Whether you're looking for a super light Chromebook or a powerful gaming machine, Acer ha

Ransomware mutations double in 2019 New! 2019-08-20 13:22:27Cybercriminals are modifying existing ransomware and creating new strains.

Connecting the Arctic: Q&A with Inmarsat New! 2019-08-20 13:00:53Inmarsat’s new satellite payloads will deliver broadband connectivity to the Arctic.

Hurry! Three's 'half price for six months' unlimited New! 2019-08-20 12:59:02Thursday will be your last chance to score a bargain on half price SIM only deals and mobile phone deals with Three.

Best touchscreen laptops 2019: the best touchsceen laptops New! 2019-08-20 12:26:25Phablets might be the biggest craze, but if you want to do more with your touchscreen, then a laptop is the smart buy and we've rounded up the best.

Huawei ban: the global fallout explained New! 2019-08-20 12:19:30Huawei's US ban has been extended to even more affiliates, leading to a bleaker outlook for the company.

OnePlus TV: what do we want to see? New! 2019-08-20 11:57:54The smartphone manufacturer is expanding into television sets – but what can OnePlus bring to a saturated market?

This leaked PS5 patent gives us our best New! 2019-08-20 11:54:31This possible PS5 patent shows off a V-shaped ventilation system, possibly designed around the Roman numeral 'V', meaning '5'.

TechCrunch » Enterprise

IBM is moving OpenPower Foundation to The Linux New! 2019-08-20 13:12:35IBM makes the Power Series chips, and as part of that has open sourced some of the underlying technologies to encourage wider use of these chips. The

H2O.ai announces $72.5M Series D led by Goldman New! 2019-08-20 10:50:17H2O.ai‘s mission is to democratize AI by providing a set of tools that frees companies from relying on teams of data scientists. Today it got a

Reputation.com nabs $30M more to help enterprises manage New! 2019-08-20 07:01:54In these days where endorsements from influential personalities online can make or break a product, a startup that’s built a business to help co

The five technical challenges Cerebras overcame in building 2019-08-19 18:38:24Superlatives abound at Cerebras, the until-today stealthy next-generation silicon chip company looking to make training a deep learning model as quick

Join The New Stack for Pancake & Podcast 2019-08-19 16:00:26Popular enterprise news and research site The New Stack is coming to TechCrunch Sessions: Enterprise on September 5 for a special Pancake & Podcas

Ally raises $8M Series A for its OKR 2019-08-19 13:40:04OKRs, or Objectives and Key Results, are a popular planning method in Silicon Valley. Like most of those methods that make you fill in some form once

The five great reasons to attend TechCrunch’s Enterprise 2019-08-19 11:36:07The vast enterprise tech category is Silicon Valley’s richest, and today it’s poised to change faster than ever before. That’s proba

Microsoft acquires jClarity, a Java performance tuning tool 2019-08-19 11:26:00Microsoft announced this morning that it was acquiring jClarity, a service designed to tune the performance of Java applications. It will be doing tha

Simon Data hauls in $30M Series C to 2019-08-19 10:25:26As businesses use an increasing variety of marketing software solutions, the goal around collecting all of that data is to improve customer experience

Microsoft Azure CTO Mark Russinovich will join us 2019-08-15 15:49:24Being the CTO for one of the three major hypercloud providers may seem like enough of a job for most people, but Mark Russinovich, the CTO of Microsof

How Facebook does IT 2019-08-15 12:24:55If you have ever worked at any sizable company, the word “IT” probably doesn’t conjure up many warm feelings. If you’re workin

Alibaba cloud biz is on a run rate 2019-08-15 12:00:36Alibaba announced its earnings today, and the Chinese e-commerce giant got a nice lift from its cloud business, which grew 66% to more than $1.1 billi

Digital Trends

Sea Hero Quest is a game that may New! 2019-08-20 04:00:39After three years, the mobile game Sea Hero Quest has generated millions of hours worth of data that could help researchers find the next step in how

Where do AirPods go when they die? The New! 2019-08-20 04:00:30True wireless headphones are incredibly small and convenient, but they’re also some of the most ephemeral and disposable tech products available

FogCam is the world’s longest-running webcam but it’s New! 2019-08-20 03:30:09For the last 25 years, FogCam has been showing San Francisco’s famous fog — or not, depending on the weather conditions. It's become the

Visually stunning HP Omen X 27 HDR TN New! 2019-08-20 03:01:52The HP Omen X 27 could prove to be the sweet spot for gamers everywhere. At a 1440p resolution, it's blazingly fast, with a 240Hz refresh rate and a

Stuck in ELO hell? HP’s A.I. coach wants New! 2019-08-20 03:01:39HP has an exciting new update for its Omen Command Center software: A.I. coaching driven by machine learning. It uses the power of A.I. and analytics

AMD Ryzen and Nvidia GTX join forces in New! 2019-08-20 03:01:13The HP Pavilion range has two new entries that are perfect for anyone looking for their first dedicated gaming system but don't have a fortune to spe

You could soon see electric scooters driving themselves New! 2019-08-20 02:45:33It might not be too long before electric scooters for companies like Bird and Lime begin driving themselves to charging stations rather than having a

Razer newest Huntsman TE keyboard is even faster New! 2019-08-20 02:00:03The Razer Huntsman Tournament Edition is a professional esports keyboard with all-new Linear Optical Switches that are faster than the Huntsman Elite.

Ford looking to recruit gamers for its first-ever New! 2019-08-20 01:45:37Ford is getting into esports in a big way with the launch of its own racing team, called Fordzilla. The automaker is planning to start recruiting for

The best 4K TV deals for August 2019: 2019-08-19 21:06:24A good 4K smart TV is the only way to enjoy all your favorite shows, movies, and games in glorious Ultra HD. If you're looking to take your home ente

This Amazon-renewed 2012 13-inch Apple MacBook Pro is 2019-08-19 20:45:39If you are in the hunt for MacBook deals, check out this Amazon-renewed 2012 Apple MacBook Pro. You can get this late-generation Apple notebook for on

5 major Lord of the Rings characters who 2019-08-19 20:36:18Amazon's billion-dollar The Lord of the Rings TV show is set during Middle-earth's Second Age, thousands of years before The War of the Ring, but th


Disclaimer and Notice:WorldProNews.com is not responsible of these news or any information published on this website.