Missing Semmster Learning 学习笔记

计算机教学中缺失的一课 ：https://missing.csail.mit.edu/。

这里的笔记主要关于：命令行、shell编程、shell环境。

一、Course overview + the shell

认识shell。shell直译为壳，比喻为操作系统内核外面的一层，是我们同内核对话的一个界面。

推荐阅读：https://wangdoc.com/bash/intro

bash是最常用的shell，可以把它当作一种编程语言，命令解释器。以下学习的就是bash的相关命令。

1.1 shell命令入门

echo "hello world"
echo 'hello world'
echo hello\ world

echo的字符用双引号或单引号包含。也可以不用它们，但遇到空格时要用反斜杠转译空格。（因为空格默认为分隔符）

1.2 how system can find echo?

echo $PATH
which echo
/bin/echo "hello world"

echo其实是一段小程序，它也有自己的代码。通过$PATH这个系统变量，就能知道操作系统在哪里寻找echo的可执行文件。

也可以通过指定可执行文件的路径的方式来执行特定的可执行文件。

1.3 navigting in the shell

cd
pwd
ls和ls -l
绝对路径和相对路径

理解linux的路径是一颗树，根路径从/开始。绝对路径从根开始，相对路径是相对于当前目录。

1.4 connecting programs

programs always associated with tow stream: input stream and output stream
redirection < file and > file
when cat is not given any arguments, it prints contents from its input stream to its output stream

cat < hello.txt
| pipe command

unix系统设计的哲学：

• 程序默认从键盘接受输入，输出到屏幕。（即每个程序关联标准输入流，标准输出流）
• 通过左右箭头符号可以重定向输入输出
• 管道可以将上一个程序的输出导入到下一个程序的输入

二、Shell Tools and Scripting

shell scripting, about learning a new language:

• basic data type
• Control flow
  • If
  • case
  • while
  • for
• syntax

shell编程就是学新的编程语言，你需要知道：

• 数据类型
• 程序控制流
• 具体语法

2.1 variable

如何定义变量？直接写出变量名，紧跟着等于号，最后是值。注意中间不能有空格。

foo=bar # foo = bar is wrong

双引号和单引号在shell程序中的区别在于里面的变量是否会被解释。单引号不会解释变量。

echo "$foo" # this print bar
echo '$foo' # this print $foo

2.2 function

mcd () {
	mkdir -p "$1"
	cd "$1"
}

• $0 name of program
• $1-9 arguments to the script
• $# number of arguments
• $$ pid
• $@ all the arguments
• $? the last command’s exit status

定义函数也非常简单，xxx。

2.3 Logical command

|| 
&&
; # simplely seperate current command and the next command

或逻辑、与逻辑、单纯的分隔符。在命令行环境中也能使用。

2.4 command substitution

echo "start program at $(date)"

这就是先前讲的，双引号内的命令会被解释执行。前提是用$()包围。

2.5 Process substitution

<(cmd) 

# this will execute cmd and place the output in a temporary file and substitute the
# <() with that file's name

在Bash中，”<(cmd)”是一种称为”Process Substitution”的特殊语法，它允许将命令的输出作为文件传递给另一个命令。

具体来说，”<(cmd)”会将命令cmd的输出作为一个临时文件，并将该文件的路径作为参数传递给当前命令。

这个临时文件只存在于命令执行期间，并在命令执行完毕后自动删除。

2.6 wildcards and curly braces

• Wildcards - Whenever you want to perform some sort of wildcard matching, you can use ? and * to match one or any amount of characters respectively.

For instance, given files foo, foo1, foo2, foo10 and bar, the command rm foo? will delete foo1 and foo2 whereas rm foo* will delete all but bar.

• Curly braces {} - Whenever you have a common substring in a series of commands, you can use curly braces for bash to expand this automatically.

This comes in very handy when moving or converting files.

convert image.{png,jpg}
# Will expand to
convert image.png image.jpg

cp /path/to/project/{foo,bar,baz}.sh /newpath
# Will expand to
cp /path/to/project/foo.sh /path/to/project/bar.sh /path/to/project/baz.sh /newpath

# Globbing techniques can also be combined
mv *{.py,.sh} folder
# Will move all *.py and *.sh files


mkdir foo bar
# This creates files foo/a, foo/b, ... foo/h, bar/a, bar/b, ... bar/h
touch {foo,bar}/{a..h}
touch foo/x bar/y
# Show differences between files in foo and bar
diff <(ls foo) <(ls bar)
# Outputs
# < x
# ---
# > y

通配符的概念无需多言。？表示任意一个字符，*表示任意多个字符。

大括号的作用比较微妙。有点像for循环遍历列表，然后自动展开。

这三个符号能极大的拓展匹配，实现自动化操作。

2.7 shebang line

#!/bin/bash

Shebang line是指在脚本文件的第一行中使用特定格式的注释来指定解释器的路径。Shebang（也称为 Hashbang ）是一个由井号和叹号构成的字符串行 *#!*。

Shebang line的作用是告诉系统应该使用哪个解释器来执行脚本，从而使脚本能够正确地运行。

Some differences between shell functions and scripts that you should keep in mind are:

• Functions have to be in the same language as the shell, while scripts can be written in any language. This is why including a shebang for scripts is important.
• Functions are loaded once when their definition is read. Scripts are loaded every time they are executed. This makes functions slightly faster to load, but whenever you change them you will have to reload their definition.
• Functions are executed in the current shell environment whereas scripts execute in their own process. Thus, functions can modify environment variables, e.g. change your current directory, whereas scripts can’t. Scripts will be passed by value environment variables that have been exported using export
• As with any programming language, functions are a powerful construct to achieve modularity, code reuse, and clarity of shell code. Often shell scripts will include their own function definitions.

differences between shell functions and scripts

• functions are executed in the current shell environment
• scripts execute in their own process

执行shell脚本其实是另外开了一个进程执行，所以当前环境不受影响。

source xx.sh	

而source一个shell脚本其实是加载脚本内的变量，这会影响当前环境变量。

2.8 shell tools

find 
grep
uniq
sort
wc
awk

一些好用的小工具：

• TLDR pages

• Tree

• fasd

• autojump

• nnn

五、Command-line Environment

提升你的shell工作流。

5.1 job control

你的shell使用一种叫做信号的机制在进程间沟通。信号是一种软中断机制。

ctrl-c 传递SIGINT信号
ctrl-\ 传递SIGQUIT信号
ctrl-z 传递SIGTSTP信号，short for Terminal Stop

SIGTERM signal ask a process to exit. Using kill command to send it.

• kill
• jobs
• fg
• bg
• nohup

5.2 terminal multiplexers

tmux教学

leading key <C-b> x means you press ctrl and b, then release them together, and then press x

panes

竖直分裂一个窗口 <C-b> %
水平分裂一个窗口 <C-b> "
关闭当前窗口 exit or ctrl-d

windows - equivalent to tabs in browsers(threads in process)

<C-b> c  创建一个虚拟桌面
<C-b> p  切换上一个
<C-b> n		切换下一个

sessions - a session is an independent workspace with one or more windows

tmux  # start a new session
tmux new -s NAME # start a new session with name
tmux ls #ls current sessions
detach a session with <C-b> d
attach a session `tmux a` , with -t to specify which

5.3 Aliases

alias name="command arg1 arg2"

为你的常用命令设置别名，减少重复劳动。

使用alias可以查看设置的别名。

常见的别名设置：

todo

5.4 Dotfiles

隐藏文件，通常是各种程序的配置文件。

• bash ~/.bashrc or~/.bash_profile
• git ~/.gitconfig
• vim ~/.vimrc
• tmux ~/.tmux.conf
• ssh~/.ssh/config

关于隐藏文件，我们需要知道三件事：

• 内容
• 位置
• 管理

5.5 Remote Machines

（1）ssh远程登录

ssh user@remote_server_name

ssh远程登录十分重要。

ssh user@remote_server_name command

如果只执行一条命令，不想登录远程主机。

ls | ssh user@remote_server_name grep PATTERN

这条命令会先将本地ls的输出通过管道传送到远程机器的grep上。是不是很神奇。这就是Unix的设计哲学。

（2）如果你不想每次都输入密码，利用非对称加密算法中的公钥和私钥，就能免去麻烦。

---

Key generation

To generate a pair you can run ssh-keygen.

ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519

You should choose a passphrase, to avoid someone who gets hold of your private key to access authorized servers. Use ssh-agent or gpg-agent so you do not have to type your passphrase every time.

If you have ever configured pushing to GitHub using SSH keys, then you have probably done the steps outlined here and have a valid key pair already. To check if you have a passphrase and validate it you can run ssh-keygen -y -f /path/to/key.

Key based authentication

ssh will look into .ssh/authorized_keys to determine which clients it should let in. To copy a public key over you can use:

cat .ssh/id_ed25519.pub | ssh foobar@remote 'cat >> ~/.ssh/authorized_keys'

A simpler solution can be achieved with ssh-copy-id where available:

ssh-copy-id -i .ssh/id_ed25519 foobar@remote

---

（3）文件传输

• ssh+tee

cat localfile | ssh remote_server tee serverfile

tee命令读标准输入到一个文件中。

• scp

scp path/local_file remote_host:path/remote_file

scp可以像cp一样，将本地文件cp到远程路径

• rsync

增强的scp，不过多深入。

（4）端口转发

• 本地端口转发：发送给本地端口的请求发送到远程机器上
• 远程端口转发：远程机器监听请求，将请求转发给本地机器

ssh -L local_ip:local_port:remote_ip:remote_port user@remote_seerver

这是本地端口转发的语法，L表示本地端口转发。本地网卡端口是可以省略的，这时表示local port绑定了本地主机所有的网卡。

比如：

$ ssh -L 9999:localhost:8888 user@remote_server

通过访问本地localhost:9999就能访问远程服务器的localhost:8888服务。

了解跳板机的概念：https://developer.aliyun.com/article/1035160

5.6 Shells & Frameworks

On-my-zsh

• Syntax-highlighting

• History-substring-search

框架是件美好的事情。