I can't recall a book that I gave up on completely but there was one that came really close.
It all started when my Mom had recommended a book by author Dean Koontz called Phantoms. The set up was interesting, an entire towns population had disappeared seemingly overnight; like a modern day Roanoke. But early in the book there was a glaring continuity error that really turned me off. The characters were investigating the town and discovered evidence that the local sheriff had fired three shots at something before he disappeared, because there were three shell casings on the ground in the sheriff's station. They also found his service revolver nearby.
Now I'll admit that I'm a bit of a firearms nerd and tend to notice all kinds of errors in movies and TV shows (you shouldn't hear the hammer cocking when someone has a Glock, if the gun is out of ammo the slide should be locked to the rear, pre Civil War settlers didn't use double action revolvers, ect). However I don't think it takes all that much firearm knowledge or research to know that the brass shells don't simply eject out of a revolver. If sheriff only got three shots off before being taken and dropping the revolver then those three shell casings would still be in the cylinder, not laying on the ground.
It was a relatively small error but it bugged me. However I pushed on, finished the book and really enjoyed the rest of it, so much that I went to the local book store and bought another Dean Koontz book to see if this was a new author for me to read. This next book was called TicTok.
Once again it started out with a pretty intriguing premise. A writer comes home and finds a strange doll on his door step. He takes it inside to study, wonder where it came from and why. Then the doll starts moving and changing. Unfortunately things start to go a bit off the rails after that.
What I thought was going to be a creepy story suddenly became very silly. I figured that maybe this was another case where the author didn't know how reality works so I kept reading and waiting for it to get better; but it didn't. Things went from silly to ridiculous to completely absurd. I was getting more and more frustrated as I read and eventually just put the book down.
About a week or two later a friend asked if I had finished the book. He was a big Stephen King fan and was curious if he would like Dean Koontz. I started telling him about the book and describing how it went from creepy to stupid. As I was describing some of the nonsensical events it sounded more like something you might see in a slapstick, screwball comedy movie. That's when it occurred to me that the book wasn't supposed to be horror, it was a comedy!
So I went back and started reading the book again with the new understanding that I was supposed to be laughing at the absurdity, not afraid to turn the page. With that in mind I was able to finish the book and appreciate it for what it was trying to be. I also decided that Dean Koontz was not the right author for me and have avoided his books ever since.