Python Urllib Usage: Urlopen, UrlParse

Use urllib: open an HTML file from an Internet site and parse a URL.

Urllib. Data is often stored on the Internet. In these locations a Python program cannot access it directly. It must instead make an external request. With urllib we make these requests: we download external files.

Urlopen. There are some complexities to opening a website in Python. First we must import, from the urllib library, the urlopen method. This is the first line of the Python file. The program next calls urlopen().

Note: The argument is the location of the web page. We next call decode() on the line. This fixes some of the data.

Tip: The last argument to print is end="". This fixes some programs with double line breaks at the end of lines.

Python program that reads lines from Internet site from urllib.request import urlopen # Print first four lines of this site. i = 0 for line in urlopen(""): # Decode. line = line.decode() # Print. print(i, line, end="") # See if past limit. if i == 3: break i += 1 Output 0 <!doctype html><html><head><link rel=canonical 1 href=><link rel=stylesheet 2 href=1><title>Dot Net Perls</title><meta 3 name=description

Parse. Web locations usually begin in http or https. These are called URLs or URIs. In Python we use the urllib.parse module to access the urlparse type. And here we parse a URL—we choose the English Wikipedia.

Then we access some fields from the parsed URL object result. The scheme is the "http" part, with no punctuation. The netloc is the domain itself: it has periods but no leading or trailing punctuation.

And: The path is the location on the domain. We use it on the root page here, so the path is simply a forward-slash "/."

Tip: There are more fields on the ParseResult. You can just print the ParseResult and all the fields will be printed. This helps discovery.

Python program that uses urlparse from urllib.parse import urlparse # Parse this url. result = urlparse("") # Get some values from the ParseResult. scheme = result.scheme loc = result.netloc path = result.path # Print our values. print(scheme) print(loc) print(path) Output http /

Summary. Python finds heavy usage on the Internet. It works both on the server and on desktop systems. It can fetch external files or web pages. But the complexity of Python programs increases when external files are necessary.Standard Library:

Because: External files cause errors. Sometimes they are not found. Other times they are in an invalid format.

Dot Net Perls
© 2007-2020 Sam Allen. Every person is special and unique. Send bug reports to