2023
Journal-2023
record my journey in CS, gain from history, sometimes scandal.
12.20
awk按行号过滤
cat xxx | awk 'NR % 2 == 0'
12.19
Nginx提高反代时限
有时会遇到504 Gateway Timeout，需要修改默认的时间限制。
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
根据官方文档，这些时间限制大多默认是60s
12.18
Python multiprocessing.Queue反直觉行为
官方文档：
class multiprocessing.Queue([maxsize])
qsize()
Return the approximate size of the queue. Because of multithreading/multiprocessing semantics, this number is not reliable.
Note that this may raise NotImplementedError on platforms like macOS where sem_getvalue() is not implemented.
empty()
Return True if the queue is empty, False otherwise. Because of multithreading/multiprocessing semantics, this is not reliable.
Queue.empty和Queue.qsize都是不可靠的，可能返回不符合预期的结果。实际上只有用get_nowait然后except Empty才是可靠的。
12.15
强制挂起进程
利用SIGSTOP信号可以从操作系统层面直接暂停进程，再发送SIGCONT信号可以继续进程。目标进程对此无感，不影响子进程。
kill -s STOP <pid> # 挂起进程
kill -s CONT <pid> # 恢复进程
12.12
查看文件inode编号
$ ls -i test.cpp
1742502 test.cpp
12.11
恢复core dumped现场
ulimit -c # 查看是否限制了core文件
cat /proc/sys/kernel/core_pattern # 查看core文件处理策略
# Manjaro下为|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h，表示由systemd-coredump接管
# Ubuntu下为core，程序crash后应该会在运行目录生成名为core的文件
对于Manjaro，coredumpctl info可以查看系统中发生的coredump记录，Storage处记录了core文件保存位置，并且是由zstd压缩的
cp /var/lib/systemd/coredump/core.....zst ./core.zst
zstd --decompress core.zst
gdb a.out core # 启动调试，程序暂停在crash处
12.08
root也无法修改的文件
今天需要修改服务器的/etc/profile却提示Operation not permitted
# whoami
root
# ls -l /etc/profile
-rw-r--r-- 1 root root 582 Dec 8 18:25 /etc/profile
# cp ~/profile_new /etc/profile
cp: cannot create regular file '/etc/profile': Operation not permitted
# rm /etc/profile
rm: cannot remove '/etc/profile': Operation not permitted
挺诡异的。后来了解到实际上还有更细粒度的权限机制
# lsattr /etc/profile
--S-iad--------------- /etc/profile
其中S表示任何更改必须立即写入磁盘，a表示仅允许追加操作，i即immutable表示无法修改，d表示no dump即备份时忽略。
解决方案：
# chattr -ai /etc/profile
# cp ~/profile_new /etc/profile
# chattr +Said /etc/profile
12.07
向运行中的Python进程注入代码
利用pyrasite库
pip install pyrasite
echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope # 启用ptrace调试
pyrasite <PID> to_inject.py --verbose
to_inject.py：
import pdb
pdb.set_trace()
无sudo安装配置zsh
# Download and extract
wget http://www.zsh.org/pub/zsh-5.9.tar.gz
tar -xzf zsh-5.9.tar.gz
cd zsh-5.9
./configure --prefix=$HOME/.apt/usr # you can change
make -j32
# all tests should pass or skip
make check
make install
配置当bash父进程为sshd时自动启动zsh
P_PROC=$(ps -o comm= -p $PPID)
if [ $P_PROC = "sshd" ]; then
exec zsh
fi
11.25
git拉取远程tag
git fetch --tags <remote>
11.22
重启通知守护程序dunst
killall dunst; notify-send foo
11.19
清理pacman缓存（自动保留到近3个版本）：sudo paccache -r
11.9
systemctl --user报错Failed to connect to bus: No such file or directory
sudo service systemd-logind start
10.29
监控文件变化并执行操作
inotifywait命令
VS Code 可视化辅助插件
.diff .patch: Diff Viewer
VS Code C++配置
CMake和Makefile插件本质上是为C/C++插件提供了configuration，可以用指令C/C++: Change Configuration Provider...来选择provider
10.22
图形化蓝牙
sudo pacman -S blueman
10.19
Nginx简单配置登录认证
sudo apt-get install apache2-utils
sudo htpasswd -c /etc/nginx/.htpasswd <username>
server {
listen 80;
server_name example.com;
location / {
auth_basic "Private Property";
auth_basic_user_file /etc/nginx/.htpasswd;
proxy_pass http://your_backend;
}
}
10.18
Vim Cheat Sheet
(those may useful but not trivial)
| Action | Effect |
|---|---|
w |
jump to the start of next word |
b/e |
jump to the start/end of this word |
5w |
repeat w for 5 times |
3igo<ESC> |
insert go 3 times |
fx/Fx |
jump to the next/previous x |
% |
jump to the matching parenthese |
0/$ |
jump to the start/end of this line |
*/# |
jump to the next/previous occurrence of this word |
gg/G |
jump to the first/last line |
5G |
jump to line 5 |
[[/]] |
jump to previous/next code block |
x/X |
remove the current/previous char |
dw |
delete from the cursor to the start of next word(exclusive) |
D |
delete from the cursor to the end of line |
dfx |
delete from the cursor to the next x(inclusive) |
. |
repeat the previous command |
(v)= |
auto-format the selection block |
(v)o |
jump to the other end of selection block |
(v)/xxx&n |
search and extend the selection block |
(v)u/U |
lowercase/uppercase the selection block |
(v)i(/i[/i{ |
select text in ()/[]/{} |
ggVG |
select the whole file |
:3,10d |
delete line [3, 10] |
:1,10s/old/new/g |
execute replacement in line [1, 10] |
:%s/old/new/gc |
c means ask before replacement |
:%!xdd |
execute command xdd for current buffer |
10.17
pip自动安装某时间点之前的旧版包
pip install pypi-timemachine
pypi-timemachine 2021-1-1 # start a local pypi server in the past
pip install --index-url localhost:<port> <package>
10.11
docker使用GPU
yay -S nvidia-docker # 注意通过源码编译安装，不可选择bin包，因为nvidia发布的最新的bin包已经过时
sudo systemctl restart docker.service
docker run --gpus all <...> # 为容器启用GPU
10.10
Named Pipe
pipe要求通信的两个进程具有相同的file descriptor context，一般来说仅适用于通过fork创建的子进程和父进程之间。
如果要实现任意两个线程之间通信可以使用具名管道Named Pipe（又名FIFO）。具名管道以文件形式存在。
mkfifo <name> # create a fifo
fifo = 'fifo_name'
if not os.path.exists(fifo):
os.mkfifo(fifo)
with open(fifo, 'w') as f:
f.write('asdf') # read or write as it is a file
读写操作默认都是阻塞的，即write时会等待读者出现，反之亦然。
10.9
subprocess的一个坑
sp.check_output('my_cmd ""', shell=True)和sp.check_output(['my_cmd', '""'])是不等价的前者子进程收到的第一个参数是''（空串），而后者子进程收到的第一个参数是'""'（两个引号）
因此，cmd: list[str] = cmd_str.split()的做法是错误的。
10.7
sed文本替换
sed "s/pattern/replacement/flags" file # output to stdout
sed -i "s/pattern/replacement/flags" file # inplace mode, output to file
sed -i.bak "s/pattern/replacement/flags" file # inplace mode, create file.bak for the original file
sed "s/pattern/replacement/flags" # deal with stdin
按键显示工具
key-mon
10.6
pip并发下载
21.3版本之后支持：
pip install some-package --use-feature=fast-deps
.desktop文件设置环境变量
把Exec改成用bash执行
[Desktop Entry]
Type=Application
Name=MyApp
Icon=myapp-icon
Exec=bash -c 'export MY_ENV_VAR=some_value; /path/to/myapp'
10.1
Tmux翻页
Ctrl+[进入翻页模式，上下键&PageUP/PageDown翻页，q键退出翻页模式
统计目录下的文件类型
find . -type f | sed -n 's/..*\.//p' | sort | uniq -c | sort -nk 1
开启热点
yay -S linux-wifi-hotspot
wihotspot-gui
Git中文显示
Git会使用八进制编码非ASCII文件名，关闭引用路径功能即可正常显示
git config --global core.quotepath off
9.30
Git误删恢复
执行git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch *.pdf' --prune-empty --tag-name-filter cat -- --all清除提交的pdf文件之后不知道为什么，工作树中的pdf文件也都没了
但是git在filter-branch的时候会自动创建备份，可以借此恢复：
git reset --hard refs/original/refs/heads/master
9.29
Git代理
对于ssh协议的git仓库，http_proxy等环境变量是无效的
需在ssh配置文件中设置：
Host github.com
Hostname github.com
ForwardAgent yes
ProxyCommand nc -X connect -x <proxy_host>:<proxy_port> %h %p
9.23
PyTorch拓展张量维度
# (a, b, ..., c) -> (a, b, ..., x+c+y)
X = torch.nn.functional.pad(X, (x, y), 'constant', 0)
MatplotLib无法显示
报错：
UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown
解决方案：
pip install PyQt5
提取mp4音频到wav
ffmpeg -i input.mp4 -vn output.wav
9.21
用户级systemd自启动
报错：Unit ***.service is added as a dependency to a non-existent unit multi-user.target.
原因：multi-user.target是系统级target，不能被用户级服务关联
解决方案：使用default.target替代
[Install]
WantedBy=default.target
9.20
Selenium 登录
部分网站能检测出selenium，可以让selenium直接使用已登录的浏览器profile
# Create Profile
mkdir profile
firefox --profile ./profile
# use profile
opt.profile = './profile'
VS Code 文件模板
WPS 粗体异常
sudo pacman -U /var/cache/pacman/pkg/freetype2-2.13.0-1-x86_64.pkg.tar.zs
9.19
Proxy Pool
9.18
Git clone
fatal: transport 'file' not allowed
git config --global protocol.file.allow always
9.16
WPS Linux 导出PDF失败
报错WPS writer encounter an error while trying export to PDF
solution: yay -S libtiff5
requests乱码
可能是gzip压缩内容未被requests库解析所致
通过headers设置'Accept-Encoding': 'utf-8'即可
编码
常见中文编码：
- Unicode
- UTF-8
- UTF-16
- GB2312
- BIG5
- GBK
- GB18030
自动检测编码：charset-normalizer
9.13
VS Code出错
加载 Web 视图时出错: Error: Could not register service workers: InvalidStateError: Failed to register a ServiceWorker: The document is in an invalid state.
9.12
PDF Format
Figure对象也有可能是文本对象的父结点，提取文本时不能忽略
Threading
Python中threading共享全局变量，而multiprocessing创建的是进程，拥有独立内存空间
9.10
Matplotlib
设置横纵坐标等比例
ax.set_aspect('equal', 'box')
绘制三维图像
D = -np.array([[float(i) for i in line] for line in data])
x = np.arange(0., 4.001, 0.02)
y = np.arange(0., 5.001, 0.02)
x, y = np.meshgrid(x, y)
fig, ax = plt.subplots(subplot_kw={"projection": "3d"})
ax.plot_surface(x, y, D, cmap=cm.coolwarm)
plt.show()
启用网格
ax.grid(True)
ax.set_xticks([N(5/20*i - 2.5) for i in range(20)]) # optional
ax.set_yticks([N(5/20*i - 2.5) for i in range(20)]) # optional
绘制矩形
field = Rectangle((N(-2), N(-1)), N(4), N(2), fc='b', alpha=0.2)
ax.add_patch(field)
9.3
debtap换源
sudo sed -i "s|http://ftp.debian.org/debian/dists|https://mirrors.ustc.edu.cn/debian/dists|g" /usr/bin/debtap
sudo sed -i "s|http://archive.ubuntu.com/ubuntu/dists|https://mirrors.ustc.edu.cn/ubuntu/dists|g" /usr/bin/debtap
9.2
软路由单线多拨实现（待填坑）
8.30
dd制作镜像
sudo dd if=/dev/sda of=sdcard-backup.img bs=4M status=progress
sudo dd if=/dev/sdb bs=4M status=progress | gzip > sdcard-backup.img.gz
gunzip -c sdcard-backup.img.gz | sudo dd of=/dev/sdb bs=4M status=progress
8.29
itertools.groupby在使用前需要先sort
8.28
python拓展名
.pyc：编译的字节码
.pyd：windows上的dll格式C-extension
.pyi：类型提示文件
.pyx：cython源文件
8.27
proxy setup & IP check
alias stp="export https_proxy=http://127.0.0.1:7890;export http_proxy=http://127.0.0.1:7890;export all_proxy=socks5://127.0.0.1:7890; curl ipinfo.io; echo;"
alias unsetproxy="unset https_proxy http_proxy all_proxy; curl ipinfo.io; echo;"
shell控制流
#!/bin/zsh
while true; do
echo "forwarding..."
ssh -NL 3004:localhost:6006 cool9
echo "try re-authing..."
sleep 1
if ./frp_auth/cool9 </dev/null; then
echo "re-authed successfully"
sleep 1
else
echo "re-authing failed"
sleep 60
fi
done
if命令的行为是反的，条件返回0则执行then
#!/bin/zsh
export GIT_WORK_TREE=~/journal
export GIT_DIR=$GIT_WORK_TREE/.git
if git diff --quiet; then
echo "Already up-to-date"
else
git add $GIT_WORK_TREE -v
git commit -m "update (`date`)"
# Request to push
echo -n "Push? [Y/n]"
read ans
case $ans in
[yY] | "" )
echo "Push:"
git push
;;
[nN] )
echo "Abort."
;;
* )
echo "Invalid response."
;;
esac
fi
ssh
端口映射
远程到本机ssh -L
本机到远程ssh -R
密钥转发
eval $(ssh-agent -s)
ssh-add /path/to/your/private-key-with-passphrase
ssh -A user@remote-server
START
难绷，去年3月份到现在直接完全忘记写。血亏。一定要坚持下去啊！