Mastering Python Request Timeout: Essential Tips for Better Requests

All Python applications that use resources from external sources face server downtime challenges. If external servers that your app uses to fetch data from go down, it may freeze if you don't set a Python request timeout parameter.

To avoid this, we will show how to set connect and read timeouts and handle potential issues. By the end of the article, you will learn about timeout errors and what values are best for optimal performance in different circumstances.

If you are web scraping with Python, then defining Python request timeout parameters will prevent your scraper from being stuck on a single unresponsive server.

Setting the Python environment

You will need three things to start using the timeout parameter within your applications. Firstly, download Python for your operating system and install it, if you do not have it already. Then, you must install the Python Requests library.

Open your Terminal and type the following line.

		
			python -m pip install requests
		
	

If you are using Python 3, use the following line instead.

		
			python3 -m pip install requests
		
	

The third step is optional, but we strongly recommend coding in an integrated development environment (IDE), like Microsoft's Visual Studio Code or PyCharm, specifically designed for Python.

What are timeouts in Python’s requests library

A 'connection timed out' error, which indicates the server failed to respond to your browser, is a typical connection timeout example. Whenever your browser communicates with a server that does not respond, it, after some time, issues a timeout error message. The time interval that your browser waits for the server to respond is the timeout parameter.

The Python Requests library lets you set two timeout parameters. You will use the connect timeout parameter to define how long your application must try to connect to a server. Meanwhile, the read timeout is a time interval that your app waits for the server to return the requested data. You can also use a single timeout parameter that will set the same value for both connect and read timeouts.

How to set timeouts in Python’s Requests library

Setting timeout values significantly improves your application's performance. Firstly, it quickly informs you, or the user, about connection issues. Simultaneously, suppose your app has looped on trying to connect to a server that does not respond. In that case, it continuously uses device resources (CPU time, network sockets, allocated memory, etc) on a futile task.

Dealing with connection timeout errors is particularly important when web scraping, because your application will be contacting dozens of servers. If it hangs over a failed connection, your scraper will work more slowly, grabbing less data than intended.

Now, we will show you how to set a timeout parameter for GET HTTP requests using Visual Studio Code IDE. Firstly, create a new Python file and save it at your chosen location. Don't forget to add the .py file extension at the end to make sure you work within a Python environment.

In the newly opened file, copy the following code.

		
			import requests
res = requests.get(
 "https://marsproxies.com/proxies/residential-proxies/",
timeout=10
)
print(res.text)
		
	

If you need to specify the exact path, use the following command. Suppose you saved the file in the C:\Users\Jane\LearnPython directory, then type this line.

		
			cd C:\Users\Jane\LearnPython
		
	

The Terminal has now switched to where you saved your file. You can now execute the code using this command.

		
			Python filename.py
		
	

We have successfully grabbed the whole HTML document of our residential proxy landing page. But let's spice things up a bit and specify that the GET request timeout occurs within a very short 0.001 timeframe.

The Terminal now displays readtimeout error

		
			requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='marsproxies.com', port=443): Read timed out. (read timeout=0.001)

		
	

Although the initial connection was successful, 0.001 seconds is not nearly enough to grab and return the required data from the server. Simultaneously, it demonstrates that setting overly short timeouts is unproductive, as it does not allow the servers to fulfil clients' demands in time.

Setting tuple timeout values

Most of the time, you will be setting two timeout values (connect and read) to avoid timeout exception errors due to incorrectly specified parameters.

For example, you may want a shorter connect timeout of 1 to 3 seconds to prevent your app from contacting an offline server repeatedly. However, a few seconds may not be enough for HTTP requests to gather online data, especially if it also includes images and video content. In this case, you can set connect timeouts for both scenarios using this code.

		
			import requests
res = requests.get(
 "https://marsproxies.com/proxies/residential-proxies/",
   timeout=(1, 5)
)
print(res.text)
		
	

As you can see, Python syntax makes specifying both timeouts very easy. You simply fill in the connect and read values for the timeout parameter and separate them with a comma.

Keep in mind that the default timeout value is always undefined, which means that your app will keep hanging on GET, POST, and other requests indefinitely if you do not specify otherwise.

Handling POST requests

We have shown how to set timeout values for sending GET HTTP requests, but you will often deal with POST requests also. For example, you may need to log in to a specific website to access required data.

First things first, review the ethical web scraping guidelines, as you don't want to grab personally identifiable or copyrighted information protected by a password. Once you have verified everything is in order, use the following code as an example for POST requests.

		
			import requests
res = requests.post(
"https://dashboard.marsproxies.com/login/",
 data={"username": "test", "password": "pass"},
 timeout=10
)
print(res.text)
		
	

Keep in mind that logging in will require a more complex code to handle CSRF tokens as well as cookies to keep you logged in. However, the code snippet above shows only how to set a timeout limit for POST HTTP requests.

Handling timeout exceptions

Setting up timeout limits is crucial, but handling timeout errors is just as important. Firstly, you must be aware of what causes the issue, and searching for read timeouts in hundreds of Terminal lines is inefficient. Use the following code to specify the error code.

		
			import requests
try:
 res = requests.get(
  "https://marsproxies.com/proxies/residential-proxies/",
timeout=0.001
    )
 print(res.text)
except requests.Timeout as e:
  print(e)
		
	

We are setting an extremely low value to demonstrate how a timeout occurs. In reality, you will use a longer maximum wait time for the server to fetch your data. Let's see what's going on in the Terminal.

Instead of displaying dozens of lines, you now get a single line that indicates the issues. Let's customize some more, change the 'print(e)' part to the following.

		
			print("The request failed due to slow server responses.")
		
	

You can write whatever you like within the parentheses. This way, you can specify error messages for each request, so that you can easily find the source of the issue. Notice that we have also switched to the 'try' and 'except' functions, which are a popular and efficient way of handling timeout issues.

Best practices for timeout management

The majority of modern applications communicate with external databases and API's to enrich them with content. But that also makes them susceptible to temporary network issues. If you do not set the timeout correctly, your app will often freeze or crash, losing users. Here are a few tips to solve such issues.

Use the timeout parameter

Although self-explanatory, some developers tend to forget that the starting timeout does not have a limit. So, for a while, the application may be running perfectly fine, until a timeout occurs. Then, the app loops over the failed request method indefinitely or until the underlying issue is solved, which can make it freeze and crash.

Setting correct timeout values

In our examples, we have set extremely low timeout values to demonstrate how the errors occur. In real apps, you don't want to do that. Typically, your connect value will be shorter than the read timeout value, because connecting to the server is much faster than grabbing data from it.

The exact timeout parameters heavily depend on the network structure, targeted data size, your geographical location, server hardware and software specifics, etc. The following timeout values are approximations to give you a general idea.

• Web scraping. 1 to 5 seconds connection and 5 to 30 seconds read values give enough time for your scraper to fetch data. You can also implement a retry mechanism that would increment the read timeout value by a few seconds with every retry, which solves temporary network issues.
• APIs. APIs are typically faster and more reliable data exchange tools. In this case, you can set 1-3 connection and 2-10 seconds timeout values and lower them if the requested data is compact.
• Internal databases. You can set a low 1-second connect and 1-3 seconds read values for communicating with resources on the same network to make your application faster.

Setting global timeouts

Sometimes you may need to set a global timeout value for all network requests that your app makes. You can use the requests.Session() object that lets you specify chosen parameters for each HTTP request made.

For example, the Session object is widely used to manage cookies to include one automatically with each subsequent request and keep the user logged in. Here's a simplistic code example of how to set global timeout values.

		
			import requests
session = requests.Session()
session.timeout = 5
try:
 res = session.get("https://marsproxies.com/")
    print(f"Success: {res.status_code}")
except requests.exceptions.Timeout:
   print("Failed: Timed out")
try:
res = session.get("https://marsproxies.com/proxies/residential-proxies/")
    print(f"Success: {res.status_code}")
except requests.exceptions.Timeout:
    print("Failed: Timed out")
		
	

In this example, we have specified a global 5-second timeout and issued two GET requests to our website pages. The result either prints a 200 status code for a successful connection, or the 'Failed: Timed out' error.

Monitor and adjust timeouts

Regularly checking your app performance to spot existing or potential troubles is best. We recommend defining your P95 and P99 latency percentiles. The first one marks the request duration under which 95% of requests succeed, while the second marks the 99% success rate. The best practice is to set the timeout just above these values to give some room for normal network performance swings.

Common timeout issues and how to avoid them

Although setting timeouts is relatively easy, you may still run into some trouble. Here are three scenarios you should know off.

No timeout values

The most obvious issue is not setting timeouts at all. As we have stated previously, the default value is undefined, so your app will continue making HTTP requests indefinitely, consuming resources, slowing it down, or even crashing.

Improper exception handling

You may experience app stutter or downtime even with correctly set timeouts. Double-check whether you set the exceptions correctly within the 'try' and 'except' blocks. Make sure you also use the 'requests.exceptions' function to identify and solve errors.

Overly aggressive timeouts

Setting short timeouts may be tempting to make your app work faster, but that will likely produce false server downtime positives. If you give your app a 0.01 read timeout, it will fail to grab your data, not because it cannot reach the server, but because there's not enough time to retrieve the information.

Final words

In this guide, you have learned to set timeout values using GET or POST request methods. Also, we have briefly explained how to handle timeout exceptions. If you find this guide useful, keep an eye on our blog as we will continue adding new Python material, like coding retry requests.

We also recommend checking out our web scraping guide if you're using timeouts to grab online data. And if you have a suggestion or want to ask our proxy specialists a question, join our growing Discord community.