forum.vdsworld.com :: View topic - Find URL

forum.vdsworld.com Forum Index

forum.vdsworld.com
Visit VDSWORLD.com

FAQ

Usergroups

Register

Log in to check your private messages

Find URL
Goto page 1, 2 Next

forum.vdsworld.com Forum Index -> General Help

View previous topic :: View next topic

Author

Message

Rubes_sw
Valued Contributor

Valued Contributor

Joined: 11 Jun 2001
Posts: 625
Location: Northern Ireland

Posted: Wed Apr 14, 2004 5:31 pm Post subject: Find URL

Can someone give an example scripts on scanning a html file and creating a list of URLs on that page/file? Nathan

Send private message

Send e-mail

Visit poster's website

SnarlingSheep
Professional Member

Professional Member

Joined: 13 Mar 2001
Posts: 759
Location: Michigan

Posted: Wed Apr 14, 2004 6:21 pm Post subject:

Reply with quote

I'm not sure whether you want to list all links, or all actual URLs - whether they are in plain text or a link.
Below is a simple example for listing links. Being simple, it also lists things in a link such as "target=_blank".
Hopefully it helps you.

Code:

LIST CREATE,1
LIST CREATE,2

LIST ADD,1,<title>Test</title>
LIST ADD,1,<body>
LIST ADD,1,Text here<br>
LIST ADD,1,Links: <a href=@CHR(34)http://www.vdsworld.com@CHR(34)>Visit VDSWorld</a> - <a href=@CHR(34)http://forum.vdsworld.com@CHR(34)>Visit VDSWorld Forum</a><br>
LIST ADD,1,E-mail: <a href=@CHR(34)mailto:webmaster@CHR(64)site.com@CHR(34)>webmaster@CHR(64)site.com</a>
LIST ADD,1,</body>

%%line = @MATCH(1,<a href=)
while @GREATER(%%line,-1)
%%text = @ITEM(1)
%%pos = @POS(<a href=,%%text)
while @GREATER(%%pos,0)
REM Strip beginning for easier position check
%%text = @STRDEL(%%text,1,@FADD(%%pos,7))
%%pos = @POS(>,%%text)
REM Add URL to results list
LIST ADD,2,@SUBSTR(%%text,1,@PRED(%%pos))
REM Strip beginning again
%%text = @STRDEL(%%text,1,%%pos)
%%pos = @POS(<a href=,%%text)
wend
REM Move to next occurrence pf "<a href="
LIST SEEK,1,@SUCC(@INDEX(1))
%%line = @MATCH(1,<a href=)
wend

REM Strip everything but domain

REM Reuse List 1
LIST CLEAR,1
%%line = 0
repeat
%%text = @ITEM(2,%%line)
%%pos = @POS("://",%%text)
REM If there is no "//" it should be a local file or e-mail link
if @GREATER(%%pos,0)
%%text = @STRDEL(%%text,1,@FADD(%%pos,2))
REM Check for ending slash, quote, space or end of string
%%pos = @POS(/,%%text)
if @GREATER(%%pos,0)
   LIST ADD,1,@SUBSTR(%%text,1,@PRED(%%pos))
else
   %%pos = @POS(@CHR(34),%%text)
   if @GREATER(%%pos,0)
   LIST ADD,1,@SUBSTR(%%text,1,@PRED(%%pos))
   else
   %%pos = @POS(@CHR(32),%%text)
   if @GREATER(%%pos,0)
      LIST ADD,1,@SUBSTR(%%text,1,@PRED(%%pos))
   else
      LIST ADD,1,@SUBSTR(%%text,1,@LEN(%%text))
   end
   end
end
end
%%line = @SUCC(%%line)
until @EQUAL(%%line,@COUNT(2))

REM Show stripped results
INFO ""@CR()@TEXT(1)

_________________
-Sheep
My pockets hurt...

Last edited by SnarlingSheep on Wed Apr 14, 2004 7:11 pm; edited 1 time in total

Send private message

Send e-mail

Rubes_sw
Valued Contributor

Valued Contributor

Joined: 11 Jun 2001
Posts: 625
Location: Northern Ireland

Posted: Wed Apr 14, 2004 6:29 pm Post subject:

works great, All i need now is to: strip everything and leave: www.domainname.com with no " or / Nathan Thanks SnarlingSheep

Send private message

Send e-mail

Visit poster's website

FreezingFire
Admin Team

Joined: 23 Jun 2002
Posts: 3508

Posted: Wed Apr 14, 2004 6:44 pm Post subject:

I could use a code like what Nathan is asking about too. Not sure quite how to do it. _________________ FreezingFire VDSWORLD.com Site Admin Team

Send private message

Visit poster's website

Rubes_sw
Valued Contributor

Valued Contributor

Joined: 11 Jun 2001
Posts: 625
Location: Northern Ireland

Posted: Wed Apr 14, 2004 7:04 pm Post subject:

Yep just to clarify: "http://www.selfcateringreservations.com/" target="_blank" So your left with http://www.selfcateringreservations.com Thanks anyone

Send private message

Send e-mail

Visit poster's website

SnarlingSheep
Professional Member

Professional Member

Joined: 13 Mar 2001
Posts: 759
Location: Michigan

Posted: Wed Apr 14, 2004 7:12 pm Post subject:

Reply with quote

Rubes_sw wrote:

Yep just to clarify:

"http://www.selfcateringreservations.com/" target="_blank"

So your left with

http://www.selfcateringreservations.com

Thanks anyone Very Happy

Very Happy

lol, you changed it after I updated my code above..I thought you wanted to strip everything but the domain(www.vdsworld.com)
Oh well, hopefully you can use it anyway.
_________________
-Sheep
My pockets hurt...

Send private message

Send e-mail

Rubes_sw
Valued Contributor

Valued Contributor

Joined: 11 Jun 2001
Posts: 625
Location: Northern Ireland

Posted: Wed Apr 14, 2004 7:21 pm Post subject:

Works Great ! Your a string king Nathan

Send private message

Send e-mail

Visit poster's website

SnarlingSheep
Professional Member

Professional Member

Joined: 13 Mar 2001
Posts: 759
Location: Michigan

Posted: Wed Apr 14, 2004 7:24 pm Post subject:

Reply with quote

Rubes_sw wrote:

Works Great !

Your a string king Very Happy

Very Happy

Nathan

In the confusion, I forgot to mention that Mac was a consultant on the domain stripping Wink

Wink

_________________
-Sheep
My pockets hurt...

Send private message

Send e-mail

FreezingFire
Admin Team

Joined: 23 Jun 2002
Posts: 3508

Posted: Wed Apr 14, 2004 9:01 pm Post subject:

Thanks a lot to everyone that helped!!!! Awesome _________________ FreezingFire VDSWORLD.com Site Admin Team

Send private message

Visit poster's website

FreezingFire
Admin Team

Joined: 23 Jun 2002
Posts: 3508

Posted: Wed Apr 14, 2004 9:07 pm Post subject:

Reply with quote

The code goes a little over my head, do you know how to add in support
for URLs that link to another page on the same site? For example,
if you had:

Code:

<a href="index.php">

I want to use this code for the open source web crawler I'm starting...
I'll be sure to give everyone who helps ample credit. Smile

Smile

_________________
FreezingFire
VDSWORLD.com
Site Admin Team

Send private message

Visit poster's website

SnarlingSheep
Professional Member

Professional Member

Joined: 13 Mar 2001
Posts: 759
Location: Michigan

Posted: Wed Apr 14, 2004 11:02 pm Post subject:

Reply with quote

I think this is more along the lines of what you are looking for FF:

Code:

LIST CREATE,1
LIST CREATE,2

LIST ADD,1,<title>Test</title>
LIST ADD,1,<body>
LIST ADD,1,Text here<br>
LIST ADD,1,<a href=@CHR(34)http://www.test.net/files/test/index.html@CHR(34) class="test">Test</a><br>
LIST ADD,1,Links: <a href=@CHR(34)index.php@CHR(34) target=@CHR(34)_blank@CHR(34)>Visit VDSWorld</a> - <a href=@CHR(34)http://forum.vdsworld.com@CHR(34)>Visit VDSWorld Forum</a><br>
LIST ADD,1,E-mail: <a href=@CHR(34)mailto:webmaster@CHR(64)site.com@CHR(34)>webmaster@CHR(64)site.com</a>
LIST ADD,1,</body>

repeat
%%found = @MATCH(1,<a href=)
if @GREATER(%%found,0)
%%text = @ITEM(1)
repeat
%%pos = @POS(<a href=,%%text)
   if @GREATER(%%pos,0)
REM Set %%pos to the end of "<a href="
%%pos = @FADD(%%pos,7)
   %%text = @STRDEL(%%text,1,%%pos)
   %%pos = @POS(@CHR(32),%%text)
   %%pos2 = @POS(>,%%text)
   REM Check if there is a space in the href line
   if @GREATER(%%pos,0)
      REM Check which comes first, space or >
   if @GREATER(%%pos2,%%pos)
   LIST ADD,2,@SUBSTR(%%text,1,@PRED(%%pos))
   %%text = @STRDEL(%%text,1,%%pos)
   else
   LIST ADD,2,@SUBSTR(%%text,1,@PRED(%%pos2))
      %%text = @STRDEL(%%text,1,%%pos2)
      end
   REM If no space, end URL at >
   else
      LIST ADD,2,@SUBSTR(%%text,1,@PRED(%%pos2))
   %%text = @STRDEL(%%text,1,%%pos2)
   end
   end
until @EQUAL(%%pos,0)
end
if @NOT(@EQUAL(@SUCC(@INDEX(1)),@COUNT(1)))
LIST SEEK,1,@SUCC(@INDEX(1))
end
until @NOT(@EQUAL(%%found,1))

REM Reuse Lists
LIST CLEAR,1
LIST ASSIGN,1,2
LIST CLEAR,2

GOSUB StripQuotes
INFO ""@CR()@TEXT(2)
exit

:StripQuotes
%x = 0
repeat
%%text = @ITEM(1,%x)
repeat
   %%pos = @POS(@CHR(34),%%text)
   if @GREATER(%%pos,0)
   %%text = @STRDEL(%%text,%%pos,%%pos)
   end
until @EQUAL(%%pos,0)
LIST ADD,2,%%text
%x = @SUCC(%x)
until @EQUAL(%x,@COUNT(1))
exit

Let me know if I need to explain anything.
_________________
-Sheep
My pockets hurt...

Send private message

Send e-mail

vtol
Valued Contributor

Valued Contributor

Joined: 05 Feb 2004
Posts: 656
Location: Eastern Indiana

Posted: Thu Apr 15, 2004 12:16 am Post subject:

Guess this proves the 'No Shirt, No Shoes, No Service' is not allways true In reverse order of coarse Whatta' nice guy that sheep

Send private message

Visit poster's website

FreezingFire
Admin Team

Joined: 23 Jun 2002
Posts: 3508

Posted: Thu Apr 15, 2004 12:32 am Post subject:

EXCELLENT, EXCELLENT, and MORE EXCELLENT AWESOME WORK Thanks a lot. This motivates me a lot more to start work on that internet crawler since I at least have a starting point. I'll disect the code this weekend when I can sit down and actually stare at the code. Then I'll ask some questions. Again, thanks a lot You helped me a lot. I was never good at string stuff. _________________ FreezingFire VDSWORLD.com Site Admin Team

Send private message

Visit poster's website

SnarlingSheep
Professional Member

Professional Member

Joined: 13 Mar 2001
Posts: 759
Location: Michigan

Posted: Thu Apr 15, 2004 12:39 am Post subject:

I refuse to be responsible for any strokes you suffer during this enjoyment period _________________ -Sheep My pockets hurt...

Send private message

Send e-mail

vtol
Valued Contributor

Valued Contributor

Joined: 05 Feb 2004
Posts: 656
Location: Eastern Indiana

Posted: Thu Apr 15, 2004 12:47 am Post subject:

Reply with quote

vtol wrote:

Shocked

Laughing

Laughing

Laughing

Laughing

Send private message

Visit poster's website

Display posts from previous:

	forum.vdsworld.com Forum Index -> General Help	All times are GMT Goto page 1, 2 Next
Page 1 of 2

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum

@vdsworld

Powered by phpBB © 2001, 2005 phpBB Group