8

I am using iptables with ipset on an Ubuntu server firewall. I am wondering if there is a command for importing a file containg a list of ip's to ipset. To populate an ipset, right now, I am adding each ip with this command:

ipset add manual-blacklist x.x.x.x

It would be very helpfull if I can add multiple ip's with a single command, like importing a file or so.

At command

 for ip in `cat /home/paul/ips.txt`; do ipset add manual-blacklist $ip;done

I get this response

resolving to IPv4 address failed to parse 46.225.38.155

for each ip in ips.txt

I do not know how to apply it.

byte00
  • 83
  • 1
  • 1
  • 5

4 Answers4

16

You can use ipset save/restore commands.

ipset save manual-blacklist

You can run above command and see how you need to create your save file.

Example output:

create manual-blacklist hash:net family inet hashsize 1024 maxelem 65536 
add manual-blacklist 10.0.0.1 
add manual-blacklist 10.0.0.2

And restore it with below command. 

ipset restore -! < ips.txt

Here we use -! to ignore errors mostly because of duplication.

ibrahim
  • 1,067
  • 8
  • 17
  • 2
    It must be noted that `restore` works blazingly fast compared to `add` when adding lots of records. On my PC it took 17 sec to `add` 10000 lines, and only 0.05 sec to `restore` – pumbo Dec 05 '18 at 05:53
  • I think you should not use `save`, because it generates `Error in line 1: Set cannot be created: set with the same name already exist`. Additionally, there is no-sort on the output (`-s -sorted` doesn't work on `restore`) – acgbox Nov 24 '22 at 21:34
  • 1
    @acgbox You will not get error if you add -! as I mentioned – ibrahim Nov 25 '22 at 22:20
1

If your list is geting really realy big for example 200 000 lines or even more and you have enough memory in server to support this, it is nice to run this whole oneliner cycle with nice:

nice -n 5 bash -c "for IP in \$(cat textfile.txt); do ipset add <setname> \$IP -exist timeout <seconds>; done"

Then your other services can step up from bash and you do not get any networking or mysql or apache or other services lag because bash is using all the resources.

AgentDeus
  • 11
  • 1
  • 1
    If you have to add many lines, it's better to use `restore` command, which works very fast as opposed to adding the lines one by one – pumbo Dec 05 '18 at 05:49
1

I know this is a few years old but why not just make a simple shell script with all the IP addreses and then run that shell script. I add hundreds of IP addresses this way.

Example: (done in Centos 8 using vi as the editor)

vi manual-blacklist.sh

Then in the manual-blacklist.sh file add all your IP addresses as listed below: (Press the i key to insert your lines)

ipset add manual-blacklist 123.45.67.111
ipset add manual-blacklist 123.45.67.112
ipset add manual-blacklist 123.45.67.113

And so on until all your IP addresses are added. To speed things up I use Excel to make my lists of IP addresses and then export it as a text file and then do a copy and paste into my shell script file.

Save the file. Press ESC key and then :wq! and then press the Enter key to save the file.

Then to run it use the following command:

./manual-blacklist.sh

All your IP addresses will be added to your manual-blacklist ipset file. And if it notices any duplicates it will ignore them as they were already added prior.

Then save this file: (I save it to my home directory and the etc directory)

ipset save > /etc/ipset.conf
ipset save > /home/username/ipset.conf

And to restore:

ipset restore -f /etc/ipset.conf

The -f switch specifies a filename to print into instead of stdout (list or save commands) or read from instead of stdin (restore command).

Hope this helps.

Greenonline
  • 1,759
  • 7
  • 16
  • 21
TechWanderer
  • 11
  • 2
  • Hello! As a conclusion, I am currently using the restore version. I prepare my file in excel and then copy / paste in txt and restore. It works. Thanks for this solution too. – byte00 Apr 01 '22 at 04:37
0

Try this command:

for ip in $(cat </file.txt>); do ipset -A <set-name> $ip;done

If you still get the a error then check your text file for spaces and unreadable\weird characters in a advanced text editor (Notepad++, SublimeText).Delete the spaces and unreadable\weird characters and try again.

ell
  • 47
  • 10