baby sword‘s blog baby sword‘s blog
首页
  • java基础
  • java进阶
大数据
  • mysql

    • mysql索引
    • mysql日志
  • redis

    • 单机下的redis
    • 集群下的redis
  • Spring
  • springboot
  • RPC
  • netty
  • mybatis
  • maven
  • 消息队列
  • kafka
  • zookeeper
  • rocketmq
  • 七大设计原则
  • 创建型模式
  • 结构型模式
  • 行为型模式
  • SpringCloud

    • eureka
  • SpringCloud Alibaba

    • nacos
  • 计算机网络
  • 操作系统
  • 算法
  • 个人项目
  • 个人面试面经
  • 八股记忆
  • 工作积累
  • 逻辑题
  • 面试

    • 百度后端实习二面
GitHub (opens new window)

zhengjian

不敢承担失去的风险,是不可能抓住梦想的
首页
  • java基础
  • java进阶
大数据
  • mysql

    • mysql索引
    • mysql日志
  • redis

    • 单机下的redis
    • 集群下的redis
  • Spring
  • springboot
  • RPC
  • netty
  • mybatis
  • maven
  • 消息队列
  • kafka
  • zookeeper
  • rocketmq
  • 七大设计原则
  • 创建型模式
  • 结构型模式
  • 行为型模式
  • SpringCloud

    • eureka
  • SpringCloud Alibaba

    • nacos
  • 计算机网络
  • 操作系统
  • 算法
  • 个人项目
  • 个人面试面经
  • 八股记忆
  • 工作积累
  • 逻辑题
  • 面试

    • 百度后端实习二面
GitHub (opens new window)
  • mysql

    • mysql索引

    • mysql日志

    • 其他

      • mysql与mongo对比
      • 数据库变量的设置
      • 《题解sql45题》
      • bufferPool详解
      • 如何解决死锁
        • Per second averages calculated from the last 18 seconds
        • BACKGROUND THREAD
        • srvmasterthread loops: 5 srvactive, 0 srvshutdown, 35264 srv_idle srvmasterthread log flush and writes: 0
        • SEMAPHORES
        • OS WAIT ARRAY INFO: reservation count 3 OS WAIT ARRAY INFO: signal count 3 RW-shared spins 0, rounds 0, OS waits 0 RW-excl spins 0, rounds 0, OS waits 0 RW-sx spins 0, rounds 0, OS waits 0 Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
        • LATEST DETECTED DEADLOCK
        • * WE ROLL BACK TRANSACTION (1)
        • TRANSACTIONS
        • ---TRANSACTION 297275, ACTIVE 93 sec 5 lock struct(s), heap size 1128, 5 row lock(s), undo log entries 2 MySQL thread id 9, OS thread handle 29160, query id 76 localhost 127.0.0.1 root
        • FILE I/O
        • I/O thread 0 state: wait Windows aio (insert buffer thread) I/O thread 1 state: wait Windows aio (log thread) I/O thread 2 state: wait Windows aio (read thread) I/O thread 3 state: wait Windows aio (read thread) I/O thread 4 state: wait Windows aio (read thread) I/O thread 5 state: wait Windows aio (read thread) I/O thread 6 state: wait Windows aio (write thread) I/O thread 7 state: wait Windows aio (write thread) I/O thread 8 state: wait Windows aio (write thread) I/O thread 9 state: wait Windows aio (write thread) Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] , ibuf aio reads:, log i/o's:, sync i/o's: Pending flushes (fsync) log: 0; buffer pool: 0 1111 OS file reads, 530 OS file writes, 191 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
        • INSERT BUFFER AND ADAPTIVE HASH INDEX
        • Ibuf: size 1, free list len 0, seg size 2, 0 merges merged operations: insert 0, delete mark 0, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 1 buffer(s) Hash table size 34679, node heap has 1 buffer(s) Hash table size 34679, node heap has 2 buffer(s) Hash table size 34679, node heap has 4 buffer(s) 6.94 hash searches/s, 2.78 non-hash searches/s
        • LOG
        • Log sequence number 730947248 Log buffer assigned up to 730947248 Log buffer completed up to 730947248 Log written up to 730947248 Log flushed up to 730947248 Added dirty pages up to 730947248 Pages flushed up to 730947248 Last checkpoint at 730947248 86 log i/o's done, 0.00 log i/o's/second
        • BUFFER POOL AND MEMORY
        • Total large memory allocated 136974336 Dictionary memory allocated 408115 Buffer pool size 8191 Free buffers 6945 Database pages 1238 Old database pages 475 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 0, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 1088, created 150, written 330 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 1238, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0]
        • ROW OPERATIONS
        • 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=7760, Main thread ID=2356 , state=sleeping Number of rows inserted 4, updated 0, deleted 1, read 1 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 19, updated 348, deleted 0, read 7206 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 10.78 reads/s
      • 关于缓存一致性与innodb bufferpool的一点思考
      • mysql读写分离和分库分表
  • redis

  • MongoDB

  • 后端存储实战

  • 数据库
  • mysql
  • 其他
xugaoyi
2023-07-05
目录

如何解决死锁

https://z.itpub.net/article/detail/7B944ED17C0084CF672A47D6E938B750

# 什么是死锁

死锁是并发系统中常见的问题,同样也会出现在数据库MySQL的并发读写请求场景中。当两个及以上的事务,双方都在等待对方释放已经持有的锁或因为加锁顺序不一致造成循环等待锁资源,就会出现“死锁”。常见的报错信息为 Deadlock found when trying to get lock...。

举例来说 A 事务持有 X1 锁 ,申请 X2 锁,B事务持有 X2 锁,申请 X1 锁。A 和 B 事务持有锁并且申请对方持有的锁进入循环等待,就造成了死锁。

img

如上图,是右侧的四辆汽车资源请求产生了回路现象,即死循环,导致了死锁。

从死锁的定义来看,MySQL 出现死锁的几个要素为:

  1. 两个或者两个以上事务
  2. 每个事务都已经持有锁并且申请新的锁
  3. 锁资源同时只能被同一个事务持有或者不兼容
  4. 事务之间因为持有锁和申请锁导致彼此循环等待

# InnoDB 锁类型

为了分析死锁,我们有必要对 InnoDB 的锁类型有一个了解。img

MySQL InnoDB 引擎实现了标准的行级别锁:共享锁( S lock ) 和排他锁 ( X lock )

  1. 不同事务可以同时对同一行记录加 S 锁。
  2. 如果一个事务对某一行记录加 X 锁,其他事务就不能加 S 锁或者 X 锁,从而导致锁等待。

如果事务 T1 持有行 r 的 S 锁,那么另一个事务 T2 请求 r 的锁时,会做如下处理:

  1. T2 请求 S 锁立即被允许,结果 T1 T2 都持有 r 行的 S 锁
  2. T2 请求 X 锁不能被立即允许

如果 T1 持有 r 的 X 锁,那么 T2 请求 r 的 X、S 锁都不能被立即允许,T2 必须等待 T1 释放 X 锁才可以,因为 X 锁与任何的锁都不兼容。共享锁和排他锁的兼容性如下所示:img

# 间隙锁( gap lock )

间隙锁锁住一个间隙以防止插入。假设索引列有2, 4, 8 三个值,如果对 4 加锁,那么也会同时对(2,4)和(4,8)这两个间隙加锁。其他事务无法插入索引值在这两个间隙之间的记录。但是,间隙锁有个例外:

  1. 如果索引列是索引,那么只会锁住这条记录(只加行锁),而不会锁住间隙。
  2. 对于联合索引且是索引,如果 where 条件只包括联合索引的一部分,那么依然会加间隙锁。

# next-key lock

next-key lock 实际上就是 行锁+这条记录前面的 gap lock 的组合。假设有索引值10,11,13和 20,那么可能的 next-key lock 包括:

(负无穷,10],(10,11],(11,13],(13,20],(20,正无穷)

在 RR 隔离级别下,InnoDB 使用 next-key lock 主要是防止幻读问题产生。

# 意向锁( Intention lock )

InnoDB 为了支持多粒度的加锁,允许行锁和表锁同时存在。为了支持在不同粒度上的加锁操作,InnoDB 支持了额外的一种锁方式,称之为意向锁( Intention Lock )。意向锁是将锁定的对象分为多个层次,意向锁意味着事务希望在更细粒度上进行加锁。意向锁分为两种:

  1. 意向共享锁( IS ):事务有意向对表中的某些行加共享锁
  2. 意向排他锁( IX ):事务有意向对表中的某些行加排他锁

由于 InnoDB 存储引擎支持的是行级别的锁,因此意向锁其实不会阻塞除全表扫描以外的任何请求。表级意向锁与行级锁的兼容性如下所示:img

# 插入意向锁( Insert Intention lock )

插入意向锁是在插入一行记录操作之前设置的一种间隙锁,这个锁释放了一种插入方式的信号,即多个事务在相同的索引间隙插入时如果不是插入间隙中相同的位置就不需要互相等待。假设某列有索引值2,6,只要两个事务插入位置不同(如事务 A 插入3,事务 B 插入4),那么就可以同时插入。

# 锁模式兼容矩阵

横向是已持有锁,纵向是正在请求的锁:img

# 阅读死锁日志

在进行具体案例分析之前,咱们先了解下如何去读懂死锁日志,尽可能地使用死锁日志里面的信息来帮助我们来解决死锁问题。

后面测试用例的数据库场景如下:MySQL 5.7 事务隔离级别为 RR

表结构和数据如下:

create table `student`(`id` int not null AUTO_INCREMENT,
`stuno` int DEFAULT NULL,
`score` int DEFAULT NULL,
PRIMARY key(`id`),
KEY `idx_stuno`(`stuno`));

insert into student(stuno,score) VALUES(2,80),(5,98),(6,77);
1
2
3
4
5
6
7

img

测试用例如下:img

通过执行show engine innodb status 可以查看到近一次死锁的日志。

# 日志分析如下:

  1. ***** (1) TRANSACTION: TRANSACTION 2322, ACTIVE 6 sec starting index read

事务号为2322,活跃 6秒,starting index read 表示事务状态为根据索引读取数据。常见的其他状态有:img

mysql tables in use 1 说明当前的事务使用一个表。

locked 1 表示表上有一个表锁,对于 DML 语句为 LOCK_IX

LOCK WAIT 2 lock struct(s), heap size 1136, 1 row lock(s)
1

LOCK WAIT 表示正在等待锁,2 lock struct(s) 表示 trx->trx_locks 锁链表的长度为2,每个链表节点代表该事务持有的一个锁结构,包括表锁,记录锁以及自增锁等。本用例中 2locks 表示 IX 锁和lock_mode X (Next-key lock)

1 row lock(s) 表示当前事务持有的行记录锁/ gap 锁的个数。

MySQL thread id 37, OS thread handle 140445500716800, query id 1234 127.0.0.1 root updating
1

MySQL thread id 37 表示执行该事务的线程 ID 为 37 (即 show processlist; 展示的 ID )

delete from student where stuno=5 表示事务1正在执行的 sql,比较难受的事情是 show engine innodb status 是查看不到完整的 sql 的,通常显示当前正在等待锁的 sql。

***** (1) WAITING FOR THIS LOCK TO BE GRANTED:

RECORD LOCKS space id 11 page no 5 n bits 72 index idx_stuno of table cw****.****student trx id 2322 lock_mode X waiting

RECORD LOCKS 表示记录锁, 此条内容表示事务 1 正在等待表 student 上的 idx_stuno 的 X 锁,本案例中其实是 Next-Key Lock 。

事务2的 log 和上面分析类似:

  1. ***** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id 11 page no 5 n bits 72 index idx_stuno of table cw****.****student trx id 2321 lock_mode X
1

显示事务 2 的 insert into student(stuno,score) values(2,10) 持有了 a=5 的 Lock mode X

| LOCK_gap,不过我们从日志里面看不到事务2执行的 delete from student where stuno=5;

这点也是造成 DBA 仅仅根据日志难以分析死锁的问题的根本原因。

  1. ***** (2) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 11 page no 5 n bits 72 index idx_stuno of table cw****.****student trx id 2321 lock_mode X locks gap before rec insert intention waiting
1

表示事务 2 的 insert 语句正在等待插入意向锁 lock_mode X locks gap before rec insert intention waiting ( LOCK_X + LOCK_REC_gap )

# 经典案例分析

# 案例一:事务并发 insert 键冲突

表结构和数据如下所示:imgimg测试用例如下:img日志分析如下:

  1. 事务 T2 insert into t7(id,a) values (26,10) 语句 insert 成功,持有 a=10 的 排他行锁( Xlocks rec but no gap )
  2. 事务 T1 insert into t7(id,a) values (30,10), 因为T2的条 insert 已经插入 a=10 的记录,事务 T1 insert a=10 则发生键冲突,需要申请对冲突的索引加上S Next-key Lock( 即 lock mode S waiting ) 这是一个间隙锁会申请锁住(,10],(10,20]之间的 gap 区域。
  3. 事务 T2 insert into t7(id,a) values (40,9)该语句插入的 a=9 的值在事务 T1 申请的 gap 锁4-10之间, 故需事务 T2 的第二条 insert 语句要等待事务 T1 的 S-Next-key Lock 锁释放,在日志中显示 lock_mode X locks gap before rec insert intention waiting 。

# 案例一:先 update 再 insert 的并发死锁问题

表结构如下,无数据:img测试用例如下:img死锁分析: 可以看到两个事务 update 不存在的记录,先后获得间隙锁( gap 锁),gap 锁之间是兼容的所以在update环节不会阻塞。两者都持有 gap 锁,然后去竞争插入意向锁。当存在其他会话持有 gap 锁的时候,当前会话申请不了插入意向锁,导致死锁。

# 如何尽可能避免死锁

  1. 合理的设计索引,区分度高的列放到组合索引前面,使业务 SQL 尽可能通过索引定位更少的行,减少锁竞争。
  2. 调整业务逻辑 SQL 执行顺序, 避免 update/delete 长时间持有锁的 SQL 在事务前面。
  3. 避免大事务,尽量将大事务拆成多个小事务来处理,小事务发生锁冲突的几率也更小。
  4. 以固定的顺序访问表和行。比如两个更新数据的事务,事务 A 更新数据的顺序为 1,2;事务 B 更新数据的顺序为 2,1。这样更可能会造成死锁。
  5. 在并发比较高的系统中,不要显式加锁,特别是是在事务里显式加锁。如 select … for update 语句,如果是在事务里(运行了 start transaction 或设置了autocommit 等于0),那么就会锁定所查找到的记录。
  6. 尽量按主键/索引去查找记录,范围查找增加了锁冲突的可能性,也不要利用数据库做一些额外额度计算工作。比如有的程序会用到 “select … where … order by rand();”这样的语句,由于类似这样的语句用不到索引,因此将导致整个表的数据都被锁住。
  7. 优化 SQL 和表设计,减少同时占用太多资源的情况。比如说,减少连接的表,将复杂 SQL 分解为多个简单的 SQL。

# ===================================== 2023-07-05 18:56:02 0x6ea8 INNODB MONITOR OUTPUT

# Per second averages calculated from the last 18 seconds

# BACKGROUND THREAD

# srv_master_thread loops: 5 srv_active, 0 srv_shutdown, 35264 srv_idle srv_master_thread log flush and writes: 0

# SEMAPHORES

# OS WAIT ARRAY INFO: reservation count 3 OS WAIT ARRAY INFO: signal count 3 RW-shared spins 0, rounds 0, OS waits 0 RW-excl spins 0, rounds 0, OS waits 0 RW-sx spins 0, rounds 0, OS waits 0 Spin rounds per wait: 0.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx

# LATEST DETECTED DEADLOCK

2023-07-05 18:55:28 0x2144 *** (1) TRANSACTION: TRANSACTION 297276, ACTIVE 37 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 2 lock struct(s), heap size 1128, 1 row lock(s) MySQL thread id 10, OS thread handle 23232, query id 69 localhost 127.0.0.1 root updating delete from student where stuno=5

*** (1) HOLDS THE LOCK(S): RECORD LOCKS space id 174 page no 5 n bits 72 index idx_stuno of table test.student trx id 297276 lock_mode X waiting Record lock, heap no 3 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 4; hex 80000005; asc ;; 1: len 4; hex 80000002; asc ;;

*** (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 174 page no 5 n bits 72 index idx_stuno of table test.student trx id 297276 lock_mode X waiting Record lock, heap no 3 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 4; hex 80000005; asc ;; 1: len 4; hex 80000002; asc ;;

*** (2) TRANSACTION: TRANSACTION 297275, ACTIVE 59 sec inserting mysql tables in use 1, locked 1 LOCK WAIT 5 lock struct(s), heap size 1128, 4 row lock(s), undo log entries 2 MySQL thread id 9, OS thread handle 29160, query id 73 localhost 127.0.0.1 root update insert into student(stuno,score) VALUES (2,10)

*** (2) HOLDS THE LOCK(S): RECORD LOCKS space id 174 page no 5 n bits 72 index idx_stuno of table test.student trx id 297275 lock_mode X Record lock, heap no 3 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 4; hex 80000005; asc ;; 1: len 4; hex 80000002; asc ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 174 page no 5 n bits 72 index idx_stuno of table test.student trx id 297275 lock_mode X locks gap before rec insert intention waiting Record lock, heap no 3 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 4; hex 80000005; asc ;; 1: len 4; hex 80000002; asc ;;

# *** WE ROLL BACK TRANSACTION (1)

# TRANSACTIONS

Trx id counter 297282 Purge done for trx's n:o < 297281 undo n:o < 0 state: running but idle History list length 2 LIST OF TRANSACTIONS FOR EACH SESSION: ---TRANSACTION 283479652005272, not started 0 lock struct(s), heap size 1128, 0 row lock(s) ---TRANSACTION 283479652002944, not started 0 lock struct(s), heap size 1128, 0 row lock(s) ---TRANSACTION 283479652002168, not started 0 lock struct(s), heap size 1128, 0 row lock(s) ---TRANSACTION 283479652001392, not started 0 lock struct(s), heap size 1128, 0 row lock(s) ---TRANSACTION 297281, ACTIVE 28 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 2 lock struct(s), heap size 1128, 1 row lock(s) MySQL thread id 10, OS thread handle 23232, query id 80 localhost 127.0.0.1 root updating delete from student where stuno=5 ------- TRX HAS BEEN WAITING 28 SEC FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 174 page no 5 n bits 72 index idx_stuno of table test.student trx id 297281 lock_mode X waiting Record lock, heap no 3 PHYSICAL RECORD: n_fields 2; compact format; info bits 32 0: len 4; hex 80000005; asc ;; 1: len 4; hex 80000002; asc ;;


# ---TRANSACTION 297275, ACTIVE 93 sec 5 lock struct(s), heap size 1128, 5 row lock(s), undo log entries 2 MySQL thread id 9, OS thread handle 29160, query id 76 localhost 127.0.0.1 root

# FILE I/O

# I/O thread 0 state: wait Windows aio (insert buffer thread) I/O thread 1 state: wait Windows aio (log thread) I/O thread 2 state: wait Windows aio (read thread) I/O thread 3 state: wait Windows aio (read thread) I/O thread 4 state: wait Windows aio (read thread) I/O thread 5 state: wait Windows aio (read thread) I/O thread 6 state: wait Windows aio (write thread) I/O thread 7 state: wait Windows aio (write thread) I/O thread 8 state: wait Windows aio (write thread) I/O thread 9 state: wait Windows aio (write thread) Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] , ibuf aio reads:, log i/o's:, sync i/o's: Pending flushes (fsync) log: 0; buffer pool: 0 1111 OS file reads, 530 OS file writes, 191 OS fsyncs 0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s

# INSERT BUFFER AND ADAPTIVE HASH INDEX

# Ibuf: size 1, free list len 0, seg size 2, 0 merges merged operations: insert 0, delete mark 0, delete 0 discarded operations: insert 0, delete mark 0, delete 0 Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 0 buffer(s) Hash table size 34679, node heap has 1 buffer(s) Hash table size 34679, node heap has 1 buffer(s) Hash table size 34679, node heap has 2 buffer(s) Hash table size 34679, node heap has 4 buffer(s) 6.94 hash searches/s, 2.78 non-hash searches/s

# LOG

# Log sequence number 730947248 Log buffer assigned up to 730947248 Log buffer completed up to 730947248 Log written up to 730947248 Log flushed up to 730947248 Added dirty pages up to 730947248 Pages flushed up to 730947248 Last checkpoint at 730947248 86 log i/o's done, 0.00 log i/o's/second

# BUFFER POOL AND MEMORY

# Total large memory allocated 136974336 Dictionary memory allocated 408115 Buffer pool size 8191 Free buffers 6945 Database pages 1238 Old database pages 475 Modified db pages 0 Pending reads 0 Pending writes: LRU 0, flush list 0, single page 0 Pages made young 0, not young 0 0.00 youngs/s, 0.00 non-youngs/s Pages read 1088, created 150, written 330 0.00 reads/s, 0.00 creates/s, 0.00 writes/s Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000 Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s LRU len: 1238, unzip_LRU len: 0 I/O sum[0]:cur[0], unzip sum[0]:cur[0]

# ROW OPERATIONS

# 0 queries inside InnoDB, 0 queries in queue 0 read views open inside InnoDB Process ID=7760, Main thread ID=2356 , state=sleeping Number of rows inserted 4, updated 0, deleted 1, read 1 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s Number of system rows inserted 19, updated 348, deleted 0, read 7206 0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 10.78 reads/s

# END OF INNODB MONITOR OUTPUT

编辑 (opens new window)
上次更新: 2024/02/22, 14:03:19
bufferPool详解
关于缓存一致性与innodb bufferpool的一点思考

← bufferPool详解 关于缓存一致性与innodb bufferpool的一点思考→

最近更新
01
spark基础
02-22
02
mysql读写分离和分库分表
02-22
03
数据库迁移
02-22
更多文章>
Theme by Vdoing | Copyright © 2019-2024 Evan Xu | MIT License
  • 跟随系统
  • 浅色模式
  • 深色模式
  • 阅读模式