11 Tips for Detecting and Responding to Intrusions on Linux

![[Pasted image 20240510093043.png]]
With the increasing popularity of open-source products, it is crucial for a backend engineer to be able to clearly identify whether an abnormal machine has been compromised. Based on my personal work experience, I have compiled several common scenarios of machines being hacked for reference.

This article is first published in the medium MPP plan. If you are a medium user, please follow me in medium. Thank you very much.

Background: The following scenarios are observed on CentOS systems and are similar for other Linux distributions.

1. Intruders May Delete Machine Logs

Check if log information still exists or has been cleared using the following commands:

Image

2. Intruders May Create a New File for Storing Usernames and Passwords

Check /etc/passwd and /etc/shadow files for any alterations using the following commands:

Pasted image 20240510085544

3. Intruders May Modify Usernames and Passwords

Examine the contents of /etc/passwd and /etc/shadow files for any changes using the following commands:

Pasted image 20240510085559

4. Check Recent Successful and Last Unsuccessful Login Events on the Machine

Refer to the log “/var/log/lastlog” using the following commands:

Pasted image 20240510085611

5. Use who to View All Currently Logged-in Users on the Machine

Refer to the log file “/var/run/utmp”:

Pasted image 20240510085623

6. Use last to View Users Logged in Since Machine Creation

Refer to the log file “/var/log/wtmp”:

Pasted image 20240510085717

7. Use ac to View Connection Time (in Hours) for All Users on the Machine

Refer to the log file “/var/log/wtmp”:

Pasted image 20240510085812

8. If Abnormal Traffic is Detected

Use “tcpdump” to capture network packets or “iperf” to check traffic.

9. Review the /var/log/secure Log File

Attempt to identify information about intruders using the following commands:

Pasted image 20240510085837

10. Identify Scripts Executed by Abnormal Processes

a. Use the top command to view the PID of abnormal processes:

Pasted image 20240510085848

b. Search for the executable file of the process in the virtual file system directory:

Pasted image 20240510085857

11. File Recovery After Confirming Intrusion and Deletion of Important Files

  1. When a process opens a file, even if it’s deleted, it remains on the disk as long as the process keeps it open. To recover such files, use lsof from the /proc directory.

  2. Most lsof information is stored in directories named after the process’s PID, such as /proc/1234, containing information for PID 1234. Each process directory contains various files providing insight into the process’s memory space, file descriptor list, symbolic links to files on disk, and other system information. lsof uses this and other kernel internal state information to generate its output.

Using the information above, you can retrieve the data by examining /proc/<PID>/fd/<descriptor>.

For example, to recover /var/log/secure, follow these steps:

a. Check /var/log/secure, confirming its absence:

Pasted image 20240510085921

b. Use lsof to check if any process is currently accessing /var/log/secure:

Pasted image 20240510085930

c. From the information above, PID 1264 (rsyslogd) has opened the file with a file descriptor of 4. It’s marked as deleted. Therefore, you can check the corresponding information in /proc/1264/fd/4:

Pasted image 20240510085938

d. You can recover the data by redirecting it to a file using I/O redirection:

Pasted image 20240510085950

e. Confirm the existence of /var/log/secure again. This method is particularly useful for many applications, especially log files and databases.

Pasted image 20240510090000

The above is the method I summarized for dealing with Linux intrusion. It can generally handle most problems. If you encounter an unresolved issue, it is best to seek advice from a professional IT operations and maintenance engineer.

I may not have written it completely correctly, so if you have different opinions, please leave a comment and let me know.