What Are HTTP Headers? 2025 Guide With Examples

Key Considerations:

• HTTP headers are metadata in web communications, sent as key value pairs in every HTTP request and response, informing how to handle the request/response.
• They enable the web client and server to exchange crucial information (e.g., content type, caching, authentication) beyond the main data payload, ensuring the correct content is delivered and displayed.
• Proper use of headers improves performance (through caching and compression), enhances security (via special response headers for HTTPS, content policies, etc.), and aids in debugging and API development.
• Web developers can inspect and modify headers using tools like browser devtools, cURL, or API clients. Understanding what HTTP headers are and how they work is essential for web scraping, API integration, and secure web applications.

HTTP headers are a core component of the HTTP protocol, consisting of additional information that travels with each request and response. They are essentially lines of text formatted as key value pairs (e.g., Header-Name: header value) included in the headers section of an HTTP message.

These headers allow the client and server to pass instructions or metadata about the request/response. In other words, headers are the behind-the-scenes data that describe or direct how to handle the main data in an HTTP message.

Why HTTP headers are important

Headers serve many purposes that facilitate communication between a web browser (or other client) and a server. They can define caching behavior, facilitate authentication, manage session state, and perform content negotiation, among other functions.

Thanks to headers, the client and server can, for example, agree on the data format (HTML, JSON, etc.), handle caching mechanisms to improve efficiency, and include credentials or tokens for secure access, and ensure encrypted and authenticated response over HTTPS. This makes headers vital in both everyday browsing and API interactions.

Because headers carry such important metadata, they play a crucial role in various scenarios:

• Web APIs

When you call a REST API, you often include headers like Content-Type (to tell the server you're sending JSON, for instance) or Authorization (to pass an API key or token). The server’s response headers might indicate rate limit info or CORS permissions. Without headers, robust API communication would be very difficult.

• Browsers

Your browser automatically sends many request header fields (User-Agent, Accept, etc.) with each page request, and servers return response headers (like cookies, content type, caching directives). These headers ensure the HTTP request is processed correctly (for example, the server knows what language or content type you prefer) and that the HTTP response is rendered correctly by the browser.

• Debugging & development

Inspecting HTTP headers is a common debugging practice. If a web page isn’t behaving correctly (e.g., fonts not loading due to cross-origin issues), developers check headers like Access-Control-Allow-Origin and content types. Tools like browser devtools or proxy monitors let you view headers to diagnose problems. In short, headers are often the first place to look when something goes wrong in web communications.

In summary, HTTP headers are important because they enable richer and more controlled communication between clients and servers beyond just raw data. They ensure that both parties understand context (what format the data is, how to cache it, how to secure it, etc.), making the web work seamlessly.

How HTTP headers work

HTTP is a request-response protocol: a client (such as a browser) sends an HTTP request, and the web server returns an HTTP response. Headers are an integral part of this exchange. In an HTTP message (whether request or response), the general format is: a start-line, followed by one or more header lines, an empty line, and then the message body (if any).

Each header line consists of a case-insensitive name, followed by a colon :, and a value. For example, a header might look like Content-Type: text/html. Header names are not case-sensitive (so Content-Type and content-type are treated the same), though by convention they are often capitalized.

In HTTP/2 and later, header names are automatically sent in lowercase by the protocol, but the semantics remain the same. Multiple headers can be present, each on its own line.

After the headers, a blank line signifies the end of the header section, and any message body (payload) comes after. The message body is the actual content being transmitted (HTML of a page, JSON data, image bytes, etc.), while headers describe or affect this content.

To illustrate, consider a simple example of an HTTP request and response with headers:

GET /index.html HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/117.0.0.1 Safari/537.36
Accept: text/html,application/xhtml+xml
Accept-Language: en-US,en;q=0.9

This is an HTTP request from a browser trying to fetch /index.html from example.com. It starts with a request line (GET /index.html HTTP/1.1), and then includes several headers:

• Host: example.com specifies the host (domain) the request is targeting.

• User-Agent: Mozilla/5.0 (Windows NT 10.0; ... Safari/537.36 identifies the client software and version (here, a web browser and OS details).

• Accept: text/html,... indicates the content types the client can accept (HTML, XHTML, etc.).

• Accept-Language: en-US,en;q=0.9 indicates preferred languages (U.S. English in this case).

After the headers, there’s an empty line. (In this example, there is no message body because GET requests typically don’t have a body.)

Now, the server’s HTTP response might look like:

HTTP/1.1 200 OK
Date: Fri, 24 Oct 2025 16:00:00 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 3472
Connection: keep-alive
Set-Cookie: sessionId=abc123; HttpOnly; Path=/

<html>
  ... [HTML content of the page] ...
</html>

The response start-line is HTTP/1.1 200 OK (indicating a successful request with status code 200). Following that are headers:

• Date: Fri, 24 Oct 2025 16:00:00 GMT – the date/time the response was sent by the server.

• Content-Type: text/html; charset=UTF-8 – the media type of the response body (HTML text in UTF-8 encoding).

• Content-Length: 3472 – the size of the message body in bytes (here the HTML page is 3472 bytes long, i.e., the length of the response payload).

• Connection: keep-alive – indicates the connection should be kept open (so the client can make any following requests on the same single transport level connection without reconnecting).

• Set-Cookie: sessionId=abc123; HttpOnly; Path=/ – a cookie the server is setting in the client’s browser (for session tracking).

After these headers, a blank line follows, and then the actual HTML content of the page is in the message body. The browser will use the header info to determine how to interpret the bytes in the body (as HTML text in this case, thanks to the Content-Type header), how long the body is, and how to handle connection reuse.

Notice that some headers are request headers (sent from client to server) and others are response headers (sent from server to client).

Some headers, like Content-Type, can appear in both requests and responses (for example, a client sending a POST request will include Content-Type and Content-Length to describe its request body). Others are specific to one direction (e.g., Set-Cookie is only in responses, Authorization only in requests).

It’s also worth noting that certain headers are considered end-to-end (they must be forwarded to the final recipient by proxies), while others are hop-by-hop (relevant only for a single transport level connection and not forwarded by intermediaries).

For instance, the Connection header is hop-by-hop: it controls whether to keep the current TCP connection alive, and proxies will consume this header rather than forwarding it. This prevents proxies from inadvertently keeping connections open to the wrong destinations.

In practice, this means that headers listed in the Connection header (like Keep-Alive or proxy-specific instructions) are removed or handled by proxies, whereas end-to-end headers (like Content-Type or Cache-Control) are passed along unchanged to the final recipient.

Understanding how HTTP headers are structured and transmitted helps developers work effectively with HTTP. Whether you’re debugging an API call, setting up caching, or inspecting why a web page is behaving a certain way, looking at the headers often reveals what’s happening behind the scenes.

General HTTP headers

General headers apply to both requests and responses, providing general information or controlling aspects of the connection. They are not specific to either the request or the response alone. In older HTTP specifications these were called "general header fields." Here are a few important general headers:

• Cache-Control

This header contains directives for caching that intermediaries (like proxies) and browsers should follow. It is used in both requests and responses to control how caching is done.

• Date

The Date header represents the date and time at which the message was originated by the server (or client). In a response, it’s the server’s timestamp for when the response was generated.

• Connection

This header controls whether the network connection stays open after the current request/response completes. Common values are keep-alive or close.

These general headers ensure that both HTTP request and HTTP response messages can manage fundamental aspects like caching and connection behavior consistently.

Request headers

Request headers are the headers the client (browser or other HTTP client) sends with an HTTP request to provide more information about what it wants or about itself. They inform the server about the context of the request. Here are some common request header fields:

• Host

This header specifies the host (domain name or IP) and optionally the port number of the server to which the request is being sent, e.g. Host: example.com. The Host header is required in virtually every HTTP/1.1 request.

• User-Agent

This header identifies the client software making the request. It typically includes the application name and version, operating system, and other details. For example, a web browser’s user agent string might be "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/117.0.0.1 Safari/537.36".

• Accept

The Accept header tells the server which content types the client can process, in order of preference. For example: Accept: text/html, application/xhtml+xml, application/xml;q=0.9, /;q=0.8. This means “I prefer HTML or XHTML, but will accept XML or anything else if necessary.”

• Accept-Language

This header specifies the preferred languages for the response, so the server can select an appropriate locale. For example: Accept-Language: en-US,en;q=0.9,fr;q=0.8 means the user prefers American English, but can accept generic English or French to some extent.

• Authorization

The Authorization header contains credentials used to authenticate the user agent with the server. This is how a client proves its identity or access rights to access a protected resource. Commonly, it’s used for APIs or sites that require login. For example, Authorization: Basic dXNlcjpwYXNzd29yZA== carries a base64-encoded username:password for HTTP Basic Auth, and Authorization: Bearer <token> carries an OAuth or JWT bearer token for API authentication.

• Referer

(Yes, it’s intentionally misspelled in the spec; sometimes seen as "Referrer".) This header indicates the URL of the page that referred the request to the server. In other words, if you clicked a link on Page A that led you to Page B, the request for Page B includes Referer: <URL of Page A>.

These are just a few of the many request headers available. Other notable ones include Cookie (sending cookies to the server), Accept-Encoding (what compression formats the client can handle, like gzip or br), DNT (do-not-track preference), If-Modified-Since / If-None-Match (for conditional requests using caching, to ask the server if the currently requested page has changed), and X-Requested-With (often set to XMLHttpRequest by AJAX calls).

Each request header serves a specific purpose to help the server understand the request context and meet the client’s needs.

Response headers

Response headers are sent by the server back to the client as part of the HTTP response. They provide additional information about the server’s response, the payload, or the server itself. Essentially, response headers accompany the status code and message body to tell the client how to interpret the data and what to do next. Some common response headers include:

• Server

The Server header identifies the software and version running on the server that handled the request (for example, Server: Apache/2.4.54 (Ubuntu) or Server: nginx/1.21.3)

• Set-Cookie

This header is used by a server to deliver cookies to the user-agent. Cookies are key-value data that the server wants the client to store and send back on later requests. For example: Set-Cookie: sessionId=abc123; Path=/; HttpOnly.

• Location

The Location header redirects the client to a different URL. It usually appears in HTTP response headers for status codes indicating a redirection (3xx series) or for the creation of a resource. For example, if the client requests a page that has moved, the server might respond HTTP/1.1 301 Moved Permanently with Location: https://www.new-site.com/newpage.

• Access-Control-Allow-Origin

This is part of Cross-Origin Resource Sharing (CORS) protocol. It indicates which origins (domains) are allowed to access the resource. For example, Access-Control-Allow-Origin: * means any origin can access, whereas Access-Control-Allow-Origin: https://example.com would allow only code from that domain to access the response.

• Retry-After

This header tells the client how long to wait before making a new request to the server. It is typically found in responses that indicate a temporary condition, like 503 Service Unavailable or 429 Too Many Requests

These response headers (and others like Content-Encoding, Cache-Control, Expires, Pragma, Content-Disposition, etc.) provide crucial context about the response. They can direct the client to take further action (redirect, retry later), provide info about the data (type, length, language), or handle state (cookies).

Next, we’ll look more closely at two specific categories of headers often found in responses: representation headers and security headers.

Representation headers

Representation headers describe the characteristics of the resource or payload being transferred, especially in responses. They tell the client about the content in the message body, such as its type, size, or encoding. Some key representation headers include:

• Content-Type

Indicates the media type of the resource in the message body. This header tells the client how to interpret the bytes of the response (or request) payload.

• Content-Length

Specifies the size of the message body, in bytes. For example, Content-Length: 3472 tells the recipient that the message body is 3472 bytes long.

• Content-Encoding

Indicates what content coding (compression) has been applied to the body. Common values are gzip, br (Brotli), or deflate.

• Content-Language

Describes the natural language of the intended audience for the content

• Last-Modified

The date and time at which the server believes the resource was last changed.

• ETag

An opaque identifier for a specific version of a resource. ETag stands for “entity tag”. It’s often a hash or fingerprint of the content. For example: ETag: "33a64df551425fcc55e4d42a148795d9f25f89d8"

Representation headers ensure that the client knows exactly what it’s getting (or sending) and can make informed decisions about caching and processing.

For example, the browser knows from Content-Type how to display content, from how much data to expect, and from Last-Modified/ETag whether it can use a cached version on subsequent requests.

Security headers

Security headers are response headers that instruct browsers to implement security features or restrictions. They help protect websites against threats by controlling browser behavior. Here are some critical security-related HTTP headers:

• Strict-Transport-Security (HSTS)

This header tells the browser that the site should only be accessed over HTTPS for a specified duration.

• X-Frame-Options

This header protects against clickjacking by controlling whether your content can be embedded in an iframe on another site.

• Content-Security-Policy (CSP)

A powerful header that specifies where content (scripts, images, styles, etc.) can be loaded from for your page.

• X-Content-Type-Options

This header prevents MIME type sniffing. Usually set to X-Content-Type-Options: nosniff

• Referrer-Policy

This header controls how much referrer information is sent when navigating from your site to another or between pages.

When properly configured, these security headers significantly harden a web application against common attacks. They are easy to implement and have wide browser support.

As best practice, web admins should regularly review and update these response headers (e.g., updating CSP rules as the site changes, renewing HSTS, etc.) to keep security up-to-date. In fact, scanning tools (and SEO/security checkers) often check for the presence of these headers as indicators of a secure site setup.

HTTP headers and proxies in web scraping

HTTP headers are especially important in the context of web scraping, and when using proxies, they require careful handling. When scraping websites with an automated script or bot, you want your HTTP requests to appear as similar as possible to those of a real user. This is where headers come in. Using the right headers can make your requests look legitimate and avoid detection or blocking.

Key headers to use with proxies

If you are routing your scraping traffic through an HTTP proxy or a series of proxies, you should ensure certain headers are present or properly set:

• User-Agent

This is the most crucial header for web scraping. Many websites will outright block requests with no user agent or an obvious bot agent. By setting a realistic string (e.g., imitating a Chrome or Firefox browser on a common OS), you make your scraper blend in with regular traffic.

• Referer

Including a Referer header can sometimes help your requests appear more natural. For example, if scraping page X, which is usually navigated to from page Y on the site, setting Referer: Y might reduce suspicion.

• Accept-Language

To mimic real browsers, you can send an Accept-Language header (e.g., en-US,en;q=0.9). This isn’t strictly required, but real browsers send it, and it might influence site content (like localization).

• Cookie

If you need to log in or maintain a session during scraping, you’ll deal with cookies. After the server sets cookies via Set-Cookie in a prior response, your scraper should send those back in a Cookie header on subsequent requests.

• Proxy-Authorization

When you use a paid proxy service or any proxy server that requires authentication, you may need to send Proxy-Authorization: Basic <credentials> in your request. This header is consumed by the proxy (not forwarded to the target site) and is used to authenticate you to the proxy server.

• Connection

When using some proxies, you might notice additional hop-by-hop headers like Connection: keep-alive or proxy-specific connection directives. Generally, you shouldn’t mess with these too much in scraping; use the default keep-alive to gain performance (the proxy can reuse connections to you and possibly to the target).

Using these headers appropriately helps you avoid blocks. For instance, setting appropriate Accept headers can prevent immediate blocks or bans by not triggering obvious bot filters. Similarly, including cookies and referers as expected makes your scraping pattern appear more like a normal browsing session.

Best practices for web scraping

When scraping with headers and proxies, consider the following best practices:

• Rotate and vary headers

Don’t use the exact same User-Agent string for months on end for all requests. Bots are often identified by unchanging or outdated user agents. Use a pool of realistic user agent strings and rotate them. Similarly, vary the Accept-Language or other optional headers if possible (within reasonable, realistic values).

• Headers consistency

Ensure the headers you send make sense together. For example, don’t claim to be a Chrome on Windows user agent but then send an Accept-Language: fr-CA (Canadian French) if typically a U.S. Windows Chrome user wouldn’t have that locale. Consistency in payload (body content) and headers is key. Also, if you pretend to be a browser, include the common headers that a browser would send (Accept, Accept-Encoding, etc.), not just User-Agent.

• Mind proxy leakage

A poorly configured proxy might add headers like Via or X-Forwarded-For which can reveal that you’re using a proxy or even expose your real IP. Ideally, use proxies that don’t add such headers or use HTTPS to the proxy so the headers are hidden from the target. If you control the proxy setup (like with requests in Python or Node.js), ensure it’s truly anonymous.

• Respect robots and rate limits

Even with the best headers, if you scrape too fast or forbidden sections, you can get blocked. Some sites use subtle header-based traps (like checking if a human would normally have a certain cookie or header at that stage). Try to emulate a real session: load images/CSS if needed (or at least pretend to with Accept headers), obey Retry-After if you get one, and do not bombard the site with excessive parallel requests through the proxy.

• Test without a proxy first

Sometimes, issues come from proxies rather than your header setup. If something isn’t working, test the request directly without the proxy to see if headers are correct. Then introduce the proxy. This can pinpoint if the proxy is stripping or altering headers. For example, some free proxies might remove or rewrite the User-Agent header, so avoid those for serious scraping.

By carefully setting HTTP headers and using quality proxies, you can scrape data more reliably and easily bypass IP bans or fingerprinting techniques that sites employ. In essence, headers (combined with proxies) are your disguise to appear as a regular user browsing the web.

Tools to inspect HTTP headers

Working with HTTP headers is much easier if you know how to inspect them. Several tools can help you view and debug headers for requests and responses:

• Browser developer tools

Modern browsers (Chrome, Firefox, Edge, Safari) have built-in developer tools. In the Network tab of these tools, you can see each request your browser makes and examine its request and response headers. For example, in Chrome, you can press F12 (or right-click and Inspect), go to the Network panel, reload the page, and click on a specific resource.

• cURL (command line)

The cURL command is a versatile tool for making HTTP requests from the terminal. You can use it to inspect headers in various ways.

• Postman and API clients

Postman is a popular API testing tool with a GUI. It allows you to create HTTP requests, add whatever headers you want, send the request, and inspect the response headers and body. It’s very useful for developing and testing web APIs.

• Online header checker tools

There are online services that let you enter a URL, and they will fetch it and show you the headers. Examples include web-sniffer, httpstatus.io, or various SEO analyzers that list response headers (and sometimes flag security headers presence).

• Proxy interception tools (advanced)

Tools like Fiddler, Wireshark, or OWASP ZAP can intercept and display HTTP traffic, including headers.

Best practices for using HTTP headers

To wrap up, here are some best practices when working with HTTP headers in your applications or servers:

• Don’t expose sensitive data

Avoid putting confidential information in headers. Remember that headers can sometimes be logged or exposed. For example, sending a password or sensitive token as a header (unless it’s part of a standard scheme like Authorization) is not a good idea. Even in Authorization headers, use secure tokens (like Bearer tokens or API keys) rather than raw credentials when possible.

• Use standardized headers when possible

Before inventing a custom header, check if a standard header already meets your needs. The HTTP spec and IANA registry list many headers. Using standard headers ensures better compatibility (clients and intermediaries know how to handle them).

• Regularly update security-related headers

Security headers (like those discussed: HSTS, CSP, etc.) should be reviewed periodically. For example, if your site now supports HTTPS fully, enable HSTS and update its max-age to a high value.

• Follow RFC guidelines and specifications

This might sound obvious, but many issues arise from non-conformant headers. Adhere to the HTTP standards for how to format header values (e.g., date formats, list separators), and only use allowed characters.