Mastering Bash and Terminal

Quelle: https://www.blockloop.io/mastering-bash-and-terminal

If there is one tool that every developer uses regardless of language, platform, or framework it’s the terminal. If we are not compiling code, executing git commands, or scp-ing ssl certificates to some remote server, we are finding a new version of cowsay to entertain ourselves while we wait on one of the former. As much as we use the terminal it is important that we are efficient with it. Here are some ways I make my time in the terminal efficient and effective.

Assumed Settings

Some of these commands list alt as a prefix character. This is because I have manually set alt as a meta key. Without this setting enabled you have to use the esc key instead. I recommend enabling the alt key. In Mac Terminal.app this setting is Preferences > Profiles tab > Keyboard sub-tab > at the bottom „Use option as meta key.“ In iTerm2 the setting is at Preferences > Profiles tab > Keys sub-tab > at the bottom of the window set „left/right option key acts as“ to „+Esc“. In GNOME terminal Edit > Keyboard Shortcuts > uncheck „Enable menu access keys.“

I also assume you’re using bash. I know there are some cool newcomers out there like zsh and fish, but after trying others out I always found that some of my utilities were missing or ill-replaced. If you are not using bash then YMMV.

I also assume you are using at least bash version 4. If you’re on a Mac then, unless you have manually installed bash with homebrew, you are using an old version. Install bash with homebrew and include it in /etc/shells.

Repeat Commands

I spend a lot of my time in terminal repeating commands that I have previously run. One thing I noticed a lot of people do is use the up and down arrows to navigate their history of commands. This is terribly inefficient. It requires repositioning your hands and often times removing your eyes from the computer screen. Also, your history (depending on your HISTSIZE) can be very long. Using the up and down arrows is almost like searching through a terrible version of the Oxford English Dictionary which has one word per page. Instead of searching line-by-line I use search history (ctrl-r and ctrl-s).

In your terminal window, before you type any text press ctrl-r and you should see your prompt change to (reverse-i-search):. Now begin typing any part of any previous command you have executed and you will see the most recent command which matches your search. If this is not the one you want, press ctrl-r again to search incrementally. For example, if you are searching for kubectl delete pods -l=app=nginx you would type kubectl or kubectl del. You should land on that command. If, while incrementally searching backward, you pass the one you’re looking for, press ctrl-s to go the other direction and you will see your prompt change to (i-search):. Once you find the command you want press enter to execute it or move the cursor left/right to modify the command first.

NOTE ctrl-s probably won’t work by default for most terminals. You will need to add stty -ixon to your ~/.bashrc (~/.bash_profile for Mac).

Sometimes you know that the command that you want to repeat is only two or three places back in history. In these cases it is sometimes easier to move up to that command directly. But you still should not use the arrow keys. Bash has keyboard shortcuts for this too! Here is where we use ctrl-p for „previous“ or ctrl-n for „next.“ Pressing ctrl-p moves to the previous command in history (replacing the up arrow), and ctrl-n moves to the next command (replacing the down arrow).

In mose cases your history will probably be set to record duplicates. This gets pretty annoying for me so I use the following setting to make sure my history doesn’t get flooded with duplicate entries. Add this to your ~/.bashrc or ~/.bashprofile and your history will only keep the newest versions of commands. If you typed git status seven times, it will only record the latest one and delete the previous entries.

Movements

Now that we know we don’t need the up and down arrow keys, what about the left and right? Unfortunately, these keys are still needed for single character movements, but I find myself using them less often. Here are some key combinations to move your cursor a little more efficiently.

  1. ctrl-a – move the cursor to the beginning of the current line
  2. ctrl-e – move the cursor to the end of the current line
  3. alt-b – move the cursor backwards one word
  4. alt-f – move the cursor forward one word
  5. ctrl-k – delete from cursor to the end of the line
  6. ctrl-u – delete from cursor to the beginning of the line
  7. alt-d – delete the word in front of the cursor
  8. ctrl-w – delete the word behind of the cursor

The last four aren’t necessarily movements, but I use them in conjunction most of the time.

Copy / Paste

One of my favorite MacOS command line utilities is pbcopy/pbpaste. I like them so much that I created the aliases for my linux machines using xclip (shared below). These two commands use your system clipboard (also called the pasteboard, hence the names). You can pipe data to pbcopy to copy something to your system clipboard or you can pipe pbpaste to another command to paste from your clipboard. Here are some examples that I use:

In linux, I put the following in my ~/.bashrc to create the same effect.

Changing Directories

cd is one of my most used commands according to my bash history. One thing I find myself doing a lot is changing between two directories or briefly changing from directory a to directory b and then back to a. Depending on the reason I’m changing directories I will use either cd - or a combination of pushd and popd. If you type cd - and press enter, you will change to your previous working directory.

On the other hand, sometimes I know that I want to go to some directory in a different place, but I might cd a few times to get there, but I want to mark my place so that I can get back quicker. In this case, you would use pushd like this.

You can pushd multiple times to build a stack. I don’t find myself doing this much, but it’s there if you need it.

Background Processes

One of my pet peves about working with other software developers is that they almost always have ten or more terminal windows open at all times. Usually, they will have one terminal per directory they are working with (this can be avoided by using pushd, popd, and cd tricks mentioned above). But often they will have a few windows open that are running processes which have locked the window. This is difficult to work with because it requires flipping back and forth and knowing where everything is. For executing processes I like to use a mixture of some commands.

If you need to run a command indefinitely you can send it to the background by first running it and then pressing ctrl-z. This will suspend or pause the process. After it has been suspended, type bg and press enter. This will move it to a running state, but it will no longer have control of your terminal window. However, if you close the terminal that job will terminate. To avoid this, you disown the process by typing disown and pressing enter. At this point the process is no longer a child of your current terminal process. I often use this to run kubectl proxy or python -m SimpleHTTPServer.

  1. ctrl-z – move the current process to the background in a suspended state.
  2. jobs -l – list the current background processes for the current tty session.
  3. bg – tell the most recent background process to continue running in the background
  4. fg – bring the most recent background process back to the foreground
  5. disown -h – disown the most recent background job. This will remove it from your current tty session. It will not be able to be brought back to the foreground. You will have to control it either with kill or something else.

bg, fg, and disown can be used with the job number found in jobs -l. If you run jobs -l you will see the job number at the beginning of the line. If you want to bring the 2nd job to the foreground you run fg %2. If you want to disown the fourth job then you run disown -h %4, and so on. The plus sign (or minus sign) at the bigging of the line has meaning as well. A plus sign indicates that the job is the most recently used, or the one that will be targeted if you type any of the commands without a job ID. The minus sign is the second most recently used.

I use ctrl-z a lot because I use a single terminal window for vim and as my command line interface. When I’m writing code in vim and I need to get back to my shell prompt I use ctrl-z to suspend vim. NOTE this will still print stdout and stderr to your command window. If you want to change that then you can redirect to files

I modified my PS1 to show my current background job count.

Working With Files

Several times throughout the day I want to view the contents of a file. Before, I would cat the file or open it in vim. cat was annoying because it flooded my terminal history. This is when I learned to use less to open files with pagination. When you open a file with less the contents of the file become paginated and you start at page one. What’s great about less is that many of my favorite key combinations work. You can use ctrl-u to page up, ctrl-d to page down, ctrl-p to scroll up one line, ctrl-n to scroll down one line, g goes to the top of the file, G goes to the bottom of the file, and / searches the file.

While less is great for opening files, I may not know where the file is in the first place. Say I have a file named „auth.py“ but I don’t remember exactly where I put it. I could cd back and forth until I find it, or do what some people do and run start . and browse for it in a UI window (terrible workflow). Instead, I use either find, ag (silver searcher), or tree. find is great for searching by file name. You can run find . -type f -name auth.py to search the current directory for a file named „auth.py“. tree is great for listing a directory in a tree format (much like how you see it in a UI). ag is an applciation called „the silver searcher.“ It is essentially a modernized version of grep and I find myself using it quite often. It’s better than grep in that it automatically ignores commonly ignored files such as the .git directory, virtualenv, and anything listed in your .gitignore. I like silver searcher because the command line arguments are very similar to grep so my flags are generally transferrable. Note It’s best to combine these commands with less because they will likely flood your terminal history.

Choose a Few to Start With

I did not use all of these when I first started using bash, nor did I memorize them all at once. I picked up one or two here and there over the years. It’s difficult to memorize key combinations, especially when there are so many of them. Pick one or two shortcuts and focus on using them. I find myself using these commands by muscle memory, not by memorizing each keyboard shortcut. In fact, once I started to write this I had to open up the terminal and work around to remember which shortcuts I use. I hope these help you work with bash and terminal more effectively. It is easy to learn one new trick and force yourself to use it for a few days until you get used to it. Once you are comfortable with that command, pick up another one.

Veröffentlicht unter Linux | Verschlagwortet mit

Improving a workflow for importing BibTeX citations

Source: http://tex.stackexchange.com/questions/79251/improving-a-workflow-for-importing-bibtex-citations

You can set up Jabref to automatically import a reference from Firefox into the current database, but it’s somewhat arcane. Here is my solution under Linux:

  • 1) Select Options -> Preferences -> Advanced — and check „Listen for remote operation …“ I don’t think it matters which port.
  • 2) Create a small bash script (text file) named „jabref-import“ that looks like this:

Replace „~/local/jabref/JabRef-2.8.1.jar“ with the path to your Jabref .jar file on your machine. Or if you have a working executable called „jabref“, you can replace everything before the „-i“ with „jabref“. Just make sure your executable accepts command-line options (mine didn’t).

In Ubuntu 13.04, the following variant of the script works:

where which jabref searches for an executable called jabref on your current $PATH.

3) Make the file executable:

4) Make sure Jabref is already open. Go to Firefox, download a citation file. It could be a .bib or .ris or .ref or whatever. Select the „Open with…“ option in the dialog, and select the jabref-import executable that you just made. The import dialog should pop up in Jabref with your citation.

Forschungswerkstatt Skript

Skript Traffic Generierung

 

Edit Artist, Album or other fields of multiply music files at once

Software EasyTAG needed.

Select all files you want to edit, enter (for instance) the Artist info you want in the Artist field, then click the little square to the right of the field and whatever info you have entered will be copied to that field in ALL files that you have selected. Repeat for the remaining fields.

Using Sshuttle in Daily Work

Source: http://teohm.com/blog/2012/04/01/using-sshuttle-in-daily-work/

I was first introduced to sshuttle by Sooyoung (@5ooyoung) in Favorite Medium as a workaround to The Great Firewall in China.

Since then, it has become my light-weight network tunneling tool in daily work.

Install sshuttle

The installation is easy now. You can install it through Mac OSX Homebrew, or Ubuntu apt-get.

I use sshuttle to..

1. Tunnel all traffic

This is the first command I learned. It forwards all TCP traffic and DNS requests to a remote SSH server.

Just like ssh, you can use any server specified in ~/.ssh/config. The -v flag means verbose mode.

Besides TCP and DNS, currently sshuttle does not forward other requests such as UDP, ICMP ping etc.

2. Tunnel all traffic, but exclude some

You can exclude certain TCP traffic using -x option.

3. Tunnel only certain traffic

To tunnel only certain TCP traffic, specify the IP addresses or IP ranges that need tunneling.

This command comes in handy, whenever I need to test an app feature (e.g. Netflix movie streaming) which only available in certain countries, or to bypass ISP faulty caches.

4. VPN to office network

I seldom do VPN, but all you need is the remote SSH server with -NH flags turned on.

-N flag tells sshuttle to figure out by itself the IP subnets to forward, and -H flag to scan for hostnames within remote subnets and store them temporarily in /etc/hosts.

IP addresses.. troublesome?

Well, I try not to deal with IP addresses manually. So I wrote a few sshuttle helpers (tnl, tnlbut, tnlonly, vpnto) that allow me to use domain names instead of IP addresses:

Tunnel all traffic

Tunnel all traffic, but exclude some

Tunnel only certain traffic

VPN to office network


The script is available on my GitHub repo. You can load it into your ~/.bashrc. To override the default tunneling SSH server in the script:

Preventing Brute Force SSH Attacks

Many VPS customers are surprised at the number of failed SSH login attempts to their servers. By just having a listening server on the Internet, you will get dozens or even hundreds of brute force login attempts each day. Most of these attempts come from automated scripts running on other compromised machines. If you are tired of reading through the failed attempts in the logs, there are a number of things that you can to do block the attempts, or otherwise make them unsuccessful.

1- If you will always be connecting to your server from the same IP address, you can firewall off port 22 to everything EXCEPT your own IP address.

iptables -A INPUT -p tcp -d 0/0 -s YOUR.IP.GOES.HERE –dport 22 -j ACCEPT
iptables -A INPUT -p tcp -d 0/0 –dport 22 -j DROP

Then run ‚iptables-save‘

Note: if you setup IP tables this way then it may cause you to lose ssh access to your server if your IP ever changes. And it can also make access to your server by RimuHosting staff more difficult.  It will also obviously prevent you from connecting to your server except from that one source IP.

2- Run sshd on a non-standard port. Since most automated attacks only attempt to connect on port 22, this can be an effective way to hide from automated attackers. To configure this, just change the Port line in /etc/ssh/sshd_config and restart ssh

Port 1022

3- Use the AllowUsers directive in the ssh configuration to only allow certain users or IP’s. In /etc/ssh/sshd_config, you can specify a list of allowed users like this:

AllowUsers bob john root@11.22.33.44 root@99.88.77.66

This will allow users ‚bob‘ and ‚john‘ to log in from anywhere, and root is only allowed to log in from those two IP addresses.

4- Use strong passwords! Brute force attempts will try common passwords like words (or combinations of words) in a dictionary, names, and common passwords. Strong passwords generally use a combination of upper and lower-case characters, numbers, and non-alphanumeric characters.

5- Even better, don’t use passwords at all. Instead, install your public key on the server and use it to log in. If all of your users will use public keys, you can set PasswordAuthentication to ’no‘. To disable password authentication just for root, use ‚PermitRootLogin without-password‘. For Debian/Ubuntu, you’ll also need to turn off ‚UsePam‘ and ‚ChallengeResponseAuthentication‘.

6- If you need to permit logins from arbitrary addresses, consider using a program like DenyHosts or Fail2ban. They watch for failed logins and add the IP addresses of attackers to /etc/hosts.deny and/or update firewall rules to null route them. DenyHosts can also be configured to synchronize with a global database so you can proactively deny hosts that other users have blacklisted.  Keep in mind that mistyping your password when you try to log in will then probably lock you out of your VPS.

7- Use ‚hashlimit‘ in ‚iptables‘:

iptables -I INPUT -m hashlimit -m tcp -p tcp –dport 22 –hashlimit 1/min
–hashlimit-mode srcip –hashlimit-name ssh -m state –state NEW -j ACCEPT

This rule limits one connection to the SSH port from one IP address per minute.

For more information, ‚man iptables‘ and ‚iptables -m hashlimit –help‘.

8 – Use port knocking to completely hide the port your SSH server is listening too; example.  This will of course make it pretty complicated for you to log in.

If you manage to lock yourself out, you can always log in using your root password via the Console over SSH feature.

Source: https://rimuhosting.com/knowledgebase/linux/misc/preventing-brute-force-ssh-attacks

Owncloud installation on debian 8

Install owncloud via the owncloud repository.

 

Debian_8.0 owncloud-8.2.2-1.1

You can add the repository key to apt. Keep in mind that the owner of the key may distribute updates, packages and repositories that your system will trust (more information). RUN:

Run the following shell commands as root to add the repository and install from there.

After this you have to create a database user with a password and a database for owncloud.

Now you can visit http://yourdomain.com/owncloud/

There you fill in all data and finish the installation.

When you are logged in you see that owncloud recommends you to enabling caching:

Also TLS should be installed:

Therefore I followed the instructions of http://workshop.tecchannel.de/a/owncloud-8-2-unter-ubuntu-server-14-04-lts-installieren,3277676,7#s9

 

Tip:

For Update notification you can use apticron

Sources:

  • https://www.howtoforge.com/tutorial/owncloud-install-debian-8-jessie/
  • https://doc.owncloud.org/server/8.2/admin_manual/installation/source_installation.html
  • https://www.techandme.se/how-to-configure-redis-cache-in-ubuntu-14-04-with-owncloud/
  • http://workshop.tecchannel.de/a/owncloud-8-2-unter-ubuntu-server-14-04-lts-installieren,3277676,7#s9
  • http://www.rojtberg.net/687/secure-owncloud-setup/

Installation bbcp

Attention: bbcp traffic is not encrypted!

bbcp must be installed on both ends

http://moo.nac.uci.edu/~hjm/HOWTO_move_data.html

Veröffentlicht unter Linux

Install Visual Code Grepper on Ubuntu 14.04×64

VisualCodeGrepper needs the .net 4 Framework  installed. The following steps are necessary to get VisualCodeGrepper working with wine on ubuntu 14.04.