When a Request Fails
A web request can fail three ways, no connection, an error status code or no answer in time, and a program handles each with requests' own exceptions, raise_for_status() turning a 4xx or 5xx answer into an HTTPError, a friendly message instead of a traceback, and one more try when the network may come back.
Journeyman · Python · REST API · Web Development · Updated
After this lesson you can
- Turn an error status into an exception with raise_for_status(), and catch HTTPError, ConnectionError and Timeout from requests
- Give a friendly message when a request fails, and try once more only when the failure might pass, such as no connection or no answer in time
Key takeaways
- With no network, requests.get raises requests.exceptions.ConnectionError, and the program stops with a traceback unless it catches it.
- An error status such as 404 or 500 raises nothing by itself. response.raise_for_status() raises an HTTPError for any code from 400 to 599.
- When the server takes longer than the timeout, requests raises requests.exceptions.Timeout.
- Catch each one with try and except, tell the user what happened in plain words, and try once more only when waiting might help, after no connection or no answer in time.
The forecast program works at home. Now run it on a laptop at the campground, where the Wi-Fi comes and goes, or on the morning the Weather Service is busy, and it stops with forty lines of traceback. Every program that calls a web API meets this sooner or later, and the difference between a tool people trust and one they give up on is what it does then.
The example is the Pine Ridge Campground forecast from Getting Data with requests, and the three ways its request can fail.
No connection
To fail on purpose without unplugging anything, ask a server that doesn't exist. A domain ending in .invalid never reaches a computer:
import requests
headers = {"User-Agent": "PythonCourseExample/1.0 (example@example.com)"}
response = requests.get("https://api.weather.invalid/points/44.8783,-122.6534", headers=headers, timeout=10)
The traceback is long, because the error passes through several layers of requests' own code. Its last line, on Windows, says what happened:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.weather.invalid', port=443): Max retries exceeded with url: /points/44.8783,-122.6534 (Caused by NameResolutionError("HTTPSConnection(host='api.weather.invalid', port=443): Failed to resolve 'api.weather.invalid' ([Errno 11001] getaddrinfo failed)"))
A ConnectionError means the request never reached the server: no network, a wrong address, or a server that's down. requests raises its own, requests.exceptions.ConnectionError, an exception (Python's report of an error while a program runs), which try and except can catch, as Handling Errors with try and except shows.
An error answer: raise_for_status()
Sometimes the server answers, and the answer is bad news. Ask for a point the Weather Service doesn't forecast:
import requests
headers = {"User-Agent": "PythonCourseExample/1.0 (example@example.com)"}
response = requests.get("https://api.weather.gov/points/0,0", headers=headers, timeout=10)
print(response.status_code)
print(response.json()["title"])
404
Data Unavailable For Requested Point
Nothing stopped: a 404 (not found) or a 500 (the server broke) is still an answer, so requests raises no exception for it. A program that goes on to read ["properties"]["forecast"] stops with a KeyError that says nothing about the real problem. response.raise_for_status() checks the status for you: it does nothing for a code under 400, and raises HTTPError for any code from 400 to 599:
import requests
headers = {"User-Agent": "PythonCourseExample/1.0 (example@example.com)"}
response = requests.get("https://api.weather.gov/points/0,0", headers=headers, timeout=10)
response.raise_for_status()
print("This line never runs.")
Traceback (most recent call last):
File "C:\Users\Sam\python-course\camping\status_check.py", line 6, in <module>
response.raise_for_status()
~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "C:\Users\Sam\python-course\camping\.venv\Lib\site-packages\requests\models.py", line 1167, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.weather.gov/points/0,0
Call it right after every request, before you read the JSON.
No answer in time: Timeout
timeout=10 waits at most 10 seconds. When the server takes longer, requests gives up and raises requests.exceptions.Timeout. To see it happen, this program sets a timeout far too short for any real server, a thousandth of a second:
import requests
headers = {"User-Agent": "PythonCourseExample/1.0 (example@example.com)"}
url = "https://api.weather.gov/points/44.8783,-122.6534"
try:
response = requests.get(url, headers=headers, timeout=0.001)
except requests.exceptions.Timeout:
print("The weather service took too long to answer.")
The weather service took too long to answer.
A timeout is how long a program waits for an answer before it gives up. Ten seconds suits a forecast; a person watching a spinner would rather have a message than wait a minute.
A friendly message, and one more try
A dropped connection or a slow answer may be gone a few seconds later, so it's worth one more try. A 404 won't fix itself: the address is wrong, and asking again gives the same answer. This version of get_json catches each failure with its own except and its own message:
import time
import requests
headers = {"User-Agent": "PythonCourseExample/1.0 (example@example.com)"}
def get_json(url):
"""Return the API's JSON answer, or None when there isn't one."""
for attempt in [1, 2]:
try:
response = requests.get(url, headers=headers, timeout=10)
response.raise_for_status()
return response.json()
except requests.exceptions.HTTPError as error:
print("The weather service said no:", error)
return None
except requests.exceptions.Timeout:
print(f"No answer in time, try {attempt} of 2.")
except requests.exceptions.ConnectionError:
print(f"No connection, try {attempt} of 2.")
if attempt == 1:
time.sleep(2)
print("Couldn't reach the weather service. Check your connection and try again.")
return None
print(get_json("https://api.weather.gov/points/0,0"))
print(get_json("https://api.weather.invalid/points/44.8783,-122.6534"))
points = get_json("https://api.weather.gov/points/44.8783,-122.6534")
print(points["properties"]["forecast"])
The weather service said no: 404 Client Error: Not Found for url: https://api.weather.gov/points/0,0
None
No connection, try 1 of 2.
No connection, try 2 of 2.
Couldn't reach the weather service. Check your connection and try again.
None
https://api.weather.gov/gridpoints/PQR/107,74/forecast
time.sleep(2) waits two seconds before the second try. The HTTPError returns at once, with no second try. The working address returns on its first try. All three exceptions share a parent, requests.exceptions.RequestException, so except requests.exceptions.RequestException: catches every way a request can fail, which is handy when the program does the same thing for each, as Saved Copies: Working Offline does.
Check yourself
2 questions to try on the lesson page.
So what
A request that might fail needs a plan for each way it fails: raise_for_status() for a bad answer, an except for each exception, words a person understands, and a second try only when waiting might help.