Beautiful Soup – Troubleshooting

Error Handling

In Beautiful Soup, there are two main types of errors that need to be handled. These errors don’t originate from your script, but rather from the structure of the snippet, which results in the Beautiful Soup API throwing an error.

These two main errors are: −

AttributeError

This error occurs when the dot notation doesn’t find a sibling tag for the current HTML tag. For example, you might have encountered this error because the cost-key function is missing an “anchor tag” because it traverses and expects an anchor tag.

KeyError

This error occurs if a required HTML tag attribute is missing. For example, if we don’t have a data-pid attribute in a snippet, the pid-key function will throw a key-error.

To avoid the two errors listed above, when parsing a result, it will be bypassed to ensure that a malformed fragment is not inserted into the database –

except(AttributeError, KeyError) as er:
pass

diagnose()

Whenever we find ourselves having trouble understanding what BeautifulSoup is doing with our document or HTML, simply pass it to the diagnose() function. When passing a document file to the diagnose() function, we can display a list of different parsers that process the document.

The following is an example demonstrating the use of the diagnose() function.

from bs4.diagnose import diagnose

with open("20 Books.html", encoding="utf8") as fp:
data = fp.read()

diagnose(data)

Output

Parsing Errors

There are two main types of parsing errors. When you feed your document into Beautiful Soup, you might get an exception like HTMLParseError. You might also get unexpected results where Beautiful Soup’s parse tree looks very different from what you expected from parsing the document.

None of these parsing errors are caused by Beautiful Soup. They are caused by the external parsers we use (html5lib, lxml), as Beautiful Soup doesn’t include any parser code. One way to resolve these parsing errors is to use another parser.

from HTMLParser import HTMLParser

try:
   from HTMLParser import HTMLParseError
except ImportError, e:
   # From python 3.5, HTMLParseError is removed. Since it can never be
   # thrown in 3.5, we can just define our own class as a placeholder.
   class HTMLParseError(Exception):
      pass

HTML Tutorial”>HTML parser can cause two of the most common parsing errors: HTMLParser.HTMLParserError: malformed start tag and HTMLParser.HTMLParserError: bad end tag. The solution is to use another parser, typically lxml or html5lib.

Another common unexpected behavior is failing to find a tag you know is in the document. However, when you run find_all() , it returns [] or find() returns None .

This may be because Python’s built-in HTML parser sometimes skips tags it doesn’t understand.

XML Parser Error

By default, the BeautifulSoup package parses documents as HTML. However, it’s very easy to use and handles unformatted XML in a very elegant way using BeautifulSoup4.

To parse a document as XML, you need the lxml parser. You simply pass “xml” as the second argument to the BeautifulSoup constructor —

soup = _BeautifulSoup(markup, "lxml-xml")

Or

soup = _BeautifulSoup(markup, "xml")

A common XML parsing error is –

AttributeError: 'NoneType' object has no attribute 'attrib'

This can happen when using the find() or findall() functions and some elements are missing or undefined.

Other Parsing Errors

Here are some other parsing errors we’ll discuss in this section.

Environment Issues

In addition to the parsing errors mentioned above, you may encounter other parsing issues, such as environment issues. Your script may work on one operating system but not another, or work in one virtual environment but not another, or not work outside of a virtual environment. All of these issues may be caused by different parser libraries being available in the two environments.

It is recommended to understand or check the default parser in your current working environment. You can check the default parser available in your current working environment or explicitly pass the desired parser library as the second argument to the BeautifulSoup constructor.

Case Insensitivity

Because HTML tags and attributes are case-insensitive, all three HTML parsers convert tag and attribute names to lowercase. However, if you want to preserve mixed-case or uppercase tags and attributes, it is best to parse the document as XML.

UnicodeEncodeError

Let’s look at the following code snippet –

soup = BeautifulSoup(response, "html.parser")
print (soup)

Output

UnicodeEncodeError: 'charmap' codec can't encode character 'u011f'

The above problem can be caused by two main reasons. You may be trying to print a Unicode character that your console doesn’t know how to display. Second, you may be trying to write to a file and the Unicode character you pass in isn’t supported by your default encoding.

One way to solve the above problem is to encode the response text/characters before making the soup to achieve the desired result, as shown below.

responseTxt = response.text.encode('UTF-8')

KeyError: [attr]

This is caused by accessing tag[‘attr’] when the tag in question does not have the attr attribute defined. The most common errors are: “KeyError: ‘href'” and “KeyError: ‘class'”. If you are not sure whether attr is defined, use tag.get(‘attr’).

for item in soup.fetch('a'):
try:
if (item['href'].startswith('/') or "tutorialspoint" in item['href']):
(...)
except KeyError:
pass # or some other fallback action

AttributeError

You may encounter an AttributeError, as shown below.

AttributeError: 'list' object has no attribute 'find_all'

This error occurs because you expect find_all() to return a single tag or string. However, soup.find_all returns a Python list of elements.

All you need to do is iterate over this list and retrieve the data from each element.