Useful Packages

There are a lot of crazy cool packages out there for linux and *nix ecosystems. Here are some of the packages that I find interesting and useful to interact with.

youtube-dl

youtube-dl is a command line utility to download video and audio from YouTube. Some other streaming websites are also supported like Facebook and Vimeo.

# list all available formats
youtube-dl -F videourl

# merge both YouTube DASH formats on-the-fly (requires ffmpeg)
youtube-dl -f 135+140 videourl

# download from a list, merge both DASH format, using name template
youtube-dl -f 135+140 -a urlist.txt -o "%(uploader)s - %(title)s.%(ext)s"

Interestingly, there are couples of web-UI frontend for youtube-dl.

mkvtoolnix

mkvtoolnix is a set of tools to manipulate *.mkv video files.

# -S option tells mkvmerge to remove all subtitle
mkvmerge -o output.mkv -S input.mkv

# merge subtitle file into mkv, softcoded
mkvmerge -o output.mkv --default-track 0 --language 0:eng subtitle.srt input.mkv

# extract the selected ID
mkvextract tracks input.mkv <trackid>:<output.ext>

# check tracks
mkvmerge -i input.mkv

# removing unwanted tracks. replace n,m,o with the IDs of the tracks
mkvmerge -o "new.mkv" -a n,m,o,... original.mkv

# extract the selected tracks from an MKV file
mkvextract tracks MovieFile.mkv 1:track1 2:track2 3:track3

ncdu

ncdu a handy utility for checking the disk space, based on ncurses. Since it is based on ncurses, it can interact with user in real time.

ncdu

There are actions we can carry out within ncdu, namely:

There are a lot of crazy cool packages out there for linux and *nix ecosystems. Here are some of the packages that I find interesting and useful to interact with.

ffmpeg

ffmpeg is a command line utility to encode various formats for video and audio files.

# my favorite options for satisfaction and space-wise
ffmpeg -i input.<ext> -c:v libx264 -c:a libfdk_aac -s 1024x576 output.mp4

# disable audio
ffmpeg -i input.<ext> -an -c:v libx264 output.mp4

# disable video
ffmpeg -i input.<ext> -vn -c:a libfdk_aac -b:a 126k output.m4a

# combine audio track into a video file, assume vid file has no audio track
ffmpeg -i input.mp4 -i audio.aac -c:v copy -c:a copy OUTPUT.mp4

# using FFmpeg to encode WebM/VPx video
ffmpeg -i input_file.avi -c:v libvpx -quality good -cpu-used 0 -b:v 500k -qmin 10 -qmax 42 -maxrate 500k -bufsize 1000k -threads 4 -vf scale=-1:480 -c:a libvorbis output.webm

# convert all files in folder with find command
find . -name "*.flac" -exec ffmpeg -i {} -c:a libmp3lame -ab 256 {}.mp3 \;
rename 's/\.flac//' *.mp3
mkdir FLAC && find . -name "*.flac" -exec mv {} FLAC \;

# rotate video recorded by using Instagram's boomerang feature
ffmpeg -i input.MOV -c copy -metadata:s:v:0 rotate=270 output.mp4

# convert lecture audio (from a voice recorder) to *.ogg libopus
ffmpeg -i input.wav -c:a libopus -b:a 32k output.ogg

To use ffmpeg for resizing a video, the -s argument can be supplied with hd1080, hd720, and hd480.

When I am using ffmpeg to convert & compress lecture recordings (either from my iPhone or a dedicated voice recorder), using libopus codec would be the most storage-efficient. At 32k bitrate, the audio quality is still good.

When it comes to converting video to animated GIF, this can be done with a combination of 3 tools. First, I cut video with Shotcut. I like Shotcut because it is quite fast and gives me precision where to cut. Then, I export frames by using ffmpeg. Then, I loop the frames with Imagemagick's convert.

An already processed video file with the size of 1.96 MB (MP4, H.264) was processed this way.

# Pre-processing: remove audio & subtitle
ffmpeg -i original.mkv -c copy -an -sn out.mkv

# Trim video with Shotcut (or whatever suits your preference)

# Export frames
mkdir frames
ffmpeg -i cut.mp4 -r 5 -vf scale=640:360 'frames/frames-%03d.jpg'

# Loop
cd frames
convert -delay 20 -loop 0 *.jpg myimage.gif

The animated GIF myimage.gif is about 4.5 MB in size.

Here is an example of bash function that I have in my ~/.zshrc.

function recopus { ffmpeg -i "$1" -c:a libopus -b:a 32k "$2" }

What this function does is that I can invoke it with recopus, it takes the first value $1 as the input and output it based on the second value $2. This function encodes an audio with the libopus audio codec at 32k bitrate. For example:

recopus recording.m4a third-lecture.ogg

IPTraf, Nethogs, and vnStat

I just learned today (11 February 2018) on how to use 3 useful Linux packages to monitor traffic. I just came to realize that maybe I need to monitor traffic on my DO instance via CLI because I am running VPN servers there (IPsec L2TP and OpenVPN). It would be great if I could see information regarding the traffic in and out.

Let's start with vnstat. Install with apt on Ubuntu. To use:

# to install
sudo apt install vnstat

# to check the interface eth0
vnstat -i eth0

# to check the interace eth0 hourly
vnstat -i eth0 -h

# force update database
sudo vnstat -u

# systemctl
sudo systemctl enable vnstat; sudo systemctl start vnstat

As for iptraf, install this package with apt. To use:

# to install
sudo apt install iptraf

# to use (ncurses UI)
sudo iptraf-ng

As for nethogs, install this package with apt. To use:

# to install
sudo apt install nethogs

# to use
sudo nethogs

pngquant and jpeg-recompress

I have been using TinyPNG web service to compress my images before uploading them to my blog. Just today (17 Feb '18), I bumped into a compiled binary known as jpeg-recompress that can be used to compress JPG image files. I tried it and compared it against TinyPNG, it looks like jpeg-recompress can be crowned as the winner here. Simply download the binaries from its GitHub page and place it in $PATH (figure out where your $PATH with the command echo $PATH). To use it:

# Single use.
jpeg-recompress image-original.jpg image-compressed.jpg

# Sequentially. Caution! This command overwrites the existing file. 
find * -type f -print -exec jpeg-recompress {} {} \;

Since I have quite a lot of JPGs (being a photographer is tough), I ran jpeg-recompress with find to automate it.

Since jpeg-recompress can only deal with JPG image files, for PNG image files we need to use a different tool: the pngquant. I downloaded the binary for Windows and placed it into my $PATH. To run:

# Single use.
pngquant --quality=65-80 image.png

# Sequentially. Caution! This command overwrites the existing file.
find * -type f -print -exec pngquant --quality=65-80 {} --ext=.png --force \;

By default, it outputs with the extension -fs8.png. The sequential command I write here that utilizes GNU find will overwrite existing file.

P/S: I use Git Bash on my Windows 10 machine. It is kind of cool.

NodeJS & NPM

Caution! I am using Git Bash as my terminal emulator. Pay attention to your $PATH because Git Bash assigns the bin folder inside the USER's main folder as the $PATH (in my case, it is /c/Users/Aizan Fahri/bin). This is really convenient, IMHO. To use node & npm, I just need to download the precompiled binaries and fortunately we do have official binaries.

Once the zip archive has been downloaded, you only need 4 items to be placed in your $PATH: node_modules, node.exe, npm, and npm.cmd. Close all terminal emulators before proceeding to the next step. Re-open a terminal emulator (i.e. Git Bash), and try invoking the command node -v. If it returned its version, it is good to go. Here are some cool packages that can be globally installed and run: is-up-cli for checking a website's status, public-ip-cli for figuring out your public-facing IP, gtop and vtop for fancier top, etc.

aria2c

The reason I like aria2c so much (package name on apt: aria2) is because I can use it as a very tiny torrent client. Meaning that I do not need to install a complicated software since aria2c can manage it with a relatively small footprint. This package is available on Windows 10 (use scoop).

aria2c --file-allocation=none --seed-time=0

Since the command is quite long there, put it in our alias. Now, run it with leech followed by magnet link bracketed with quote signs.

alias leech="aria2c --file-allocation=none --seed-time=0"

ImageMagick's convert

Make a simple GIF animation with 2 files:

convert -loop 0 -delay 200 screenshot.1.png screenshot.2.png out.gif

Explanations:

Git

My favorite bash function that I have in my ~/.zshrc

function pub { git push "$1" master }

This function pub takes $1 as the argument, which is the field of target remote repo.