Remove https characters in a text file

Question

I need to remove all http and https from txt file.

Like this:

http://ac.tecnicasdeinvasao.com
http://go.tecnicasdeinvasao.com
http://lp.tecnicasdeinvasao.com
https://ac.tecnicasdeinvasao.com
http://secreto.tecnicasdeinvasao.com
https://go.tecnicasdeinvasao.com
https://lp.tecnicasdeinvasao.com
https://secreto.tecnicasdeinvasao.com

To

ac.tecnicasdeinvasao.com
go.tecnicasdeinvasao.com
lp.tecnicasdeinvasao.com
ac.tecnicasdeinvasao.com
secreto.tecnicasdeinvasao.com
go.tecnicasdeinvasao.com
lp.tecnicasdeinvasao.com
secreto.tecnicasdeinvasao.com

I tryed using sed but without success.

Seamus · Accepted Answer · 2021-11-18T01:01:45.853

I prefer awk to sed, so here's what I'd do:

awk allows you to define custom field separators, which makes your problem fairly straightforward:

Assuming the file containing the full URLs is tstfile.txt, declare the field separator (-F) as //, and then print the second field ($2):

$ awk -F'//' '{print $2}' tstfile.txt
ac.tecnicasdeinvasao.com
go.tecnicasdeinvasao.com
lp.tecnicasdeinvasao.com
ac.tecnicasdeinvasao.com
secreto.tecnicasdeinvasao.com
go.tecnicasdeinvasao.com
lp.tecnicasdeinvasao.com
secreto.tecnicasdeinvasao.com

If you want the results in a file, you can use a redirection >somefile.txt.

And if your distro uses the GNU version of awk, a.k.a. gawk, you have the option of updating your input file in-place:

$ awk -i inplace -F'//' '{print $2}' tstfile.txt
$

Now, the contents of tstfile.txt will be exactly as shown above, which may save you a step or two - depending on your end objective. Other options, including saving the original file under a different file name are covered in this answer.

@Chead: Generally speaking, `awk` treats each *line* as a `record`, and each `record` has one or more `fields`. A `field separator` tells `awk` what the field boundaries are. In this case, because we declared `//` as the `field separator`, the characters to the left of the `field separator` is the first field, the chars to the right are the second field. As `awk` reads the line/record, each `field` is stored in `$1` (first record), `$2` (2nd record), etc. This worked because we declared `//` as the `field separator`, and printed the second field: `$2`. Lots of good tutorials on `awk` online. — Seamus, Nov 17 '21 at 02:59

AlexPixel · Answer 2 · 2021-11-17T02:47:13.527

Because you asked with sed

With sed easier to read:

sed  -E 's/http.+[/]//'

With sed easier to write:

sed -E 's/http.+\///'

Both do the same. sed use the format of /select/replace/, because of this is harder to read with \ backlash.

-E is to use regular expression "newer" versions
s is for find and replace a string
.+ has 2 meanings, the . dot is for any character, in this case the s and the + sign is for selecting everything else after the .
if a character has special meaning in regex world, you need to escape them, usually with \ backlash. In this case, you require telling sed to select everything until the last /, so you require escaping it like this \/ but is harder to read \///, the last // is from the format of /select/replace/ you can also escape with brackets [/], and because you want to remove what you have selected then you write nothing --> //

and because everything starts with h(is the same), so you can make it even shorter

sed -E 's/.+\///'  
sed -E 's/.+[/]//'

this means --> select everything from the beginning of the TEXT until the last / --> whateverGoesHere/

Note that in cases where the search pattern or replacement string contain `/`, it is easiest to instruct `sed` to use a different command argument separator, e.g. `sed -E 's|http.+/||'` or `sed -E 's,http.+/,,'` — AdminBee, Nov 18 '21 at 08:58
man you were so good to explain sed! now i understood! thanks so much :) — Lucas Fernandes, Nov 18 '21 at 14:28

score 1 · Answer 3 · answered Nov 17 '21 at 01:20

Try this:

awk '{sub(/https?:\/\//,"");print}' file.txt > outfile.txt

awk loops through the input lines performing the program on each line. I didn't specify any regular expression to match, so the code in curly braces is executed on each line. The sub function matches the regular expression between slashes and replaces it with the quoted empty string. The question mark in the regular expression makes the "s" optional

Remove https characters in a text file

3 Answers3

Because you asked with sed