I wrote about my DuckDuckGo (and Bing) shadow ban in a previous post. It made the rounds, so in the interest of transparency (and so I remember for next time this happens), here’s steps I’ve taken to fix this, what I know so far, and what I know…
Edit: …and stay for some good news at the end.
Steps I’ve taken so far
Let’s look at what I’ve done and create a paper trail for our future selves:
- I filed a Support Ticket (REQ00047859) with Bing
- I verified my domain with Bing through a DNS record on Jan 7th, 2023
- I fixed my
<meta>description crawl errors on Jan 14th, 2023 - I removed some extra whitespace before my
<!DOCTYPE html>on Jan 17th, 2023 - I clicked every “Re-index” button in the site I could find
- Scott Hanselman offered to upstream this to people directly at Bing, so I sent him some screenshots.
First of all, Scott is the best guy. While I’m chuffed and thankful for his offer to help, I have to say “knowing a guy” as a remediation step isn’t my favorite fix. I’ll be happy if I show up, but it sort of ruins the science. I know Bing has a hard job filtering 2 billion spam sites and 42 real sites, so it’s not suprising a small site like mine would get swept up in some algorithmic gatekeeping. But hopefully something comes out of this, both for developers like me who made an oopsie and for Bing to help refine their tooling (or error messaging?) becuase I know a handful of people in the same boat as me.
What I know so far: Other people have this problem
After posting, other people reached out to me and said they were having the same problem with their small personal sites.
The anecdata I’ve received here is fascinating. Kev Quirk’s situation is actually a success story because after about a month his site started showing up. Tom O’Dwyer’s situation is less great, he was flat out told by support his site “did not meet the standards set by Bing”. Daniel Aleksandersen was the person who told me about my <!DOCTYPE html> whitespace issue and gave me three or four scenarios he’s A/B tested where Bing fails on valid HTML; based on what he was saying, Bing seems pretty particular about HTML, which is unfortunate because even HTML isn’t particular about HTML. Based on what every one has been saying, the rule of thumb is that if you fall out of the index, it takes about 4~6 weeks to get back in.
It’s also worth noting that my site is missing in other search engines, like Ecosia and Kagi, which I assume also use the Bing index.
What Bing Webmaster Tools is telling me
As I said above, I fixed my <meta> description crawl errors by adding a bunch of redundant descriptions. I don’t think this is good SEO, but at a minimum I’ve cleared all “Errors” from the Bing Webmaster Tools Site Scan. I’ll go back through and make this better but I’m trying to see if I can trigger a successful indexing.
After errors, the sole lead I have is when I go to the URL Inspection section and enter my homepage URL, I get the following information: Bing discovered on 24 Dec 2015 but original attempt at a crawl errors out with the following error…
The inspected URL is known to Bing but has some issues which are preventing indexation. We recommend you to follow Bing Webmaster Guidelines to increase your chances of indexation.
Okay, another clue! In my next post, I’ll take a good hard look at if I’m adhering to the Bing Webmaster Guidelines.
🚨 Breaking news
Update: Before posting I checked the same URL Inspection page and now my domain is getting ✅ green checkmarks for indexing. I’m also seeing empty sections of the Bing Webmasters Tools light up with data. And most importantly, when I search for my domain “daverupert.com” in Bing and DuckDuckGo I see a single post from 2015 in the first page of results. Things are happening! 🎉