Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better User-Agent header for RSS feed requests #93

Closed
jpco opened this issue Sep 28, 2024 · 3 comments · Fixed by #94
Closed

Better User-Agent header for RSS feed requests #93

jpco opened this issue Sep 28, 2024 · 3 comments · Fixed by #94
Labels
enhancement New feature or request good first issue Good for newcomers storyprovider

Comments

@jpco
Copy link

jpco commented Sep 28, 2024

The default User-Agent for the python requests library leads to 403 errors from certain blogs -- some googling indicates this might be a Cloudflare protection against (lazy) web scrapers. Just setting something random like headers={'User-Agent': 'goosepaper/0.0'} for the requests.get() call at rss.py:39 causes those blogs to work better.
@jpco
Copy link
Author

jpco commented Sep 28, 2024

Oh, for repro purposes, a good feed is https://civileats.com/feed/

@j6k4m8
Copy link
Owner

j6k4m8 commented Sep 30, 2024

This is a great idea, thank you!

@j6k4m8 j6k4m8 added enhancement New feature or request good first issue Good for newcomers storyprovider labels Sep 30, 2024
j6k4m8 added a commit that referenced this issue Sep 30, 2024
@j6k4m8
Add User-Agent header for RSS feed requests
cc99630 
Fixes #93
@j6k4m8 j6k4m8 mentioned this issue Sep 30, 2024
Merged
@j6k4m8 j6k4m8 closed this as completed in #94 Oct 4, 2024
@j6k4m8 j6k4m8 closed this as completed in 525f310 Oct 4, 2024
@j6k4m8
Copy link
Owner

j6k4m8 commented Oct 4, 2024

Fixed in #94!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue Good for newcomers storyprovider
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add User-Agent header for RSS feed requests j6k4m8/goosepaper
2 participants
@j6k4m8 @jpco