Without using the help of Pinyin I was taught Chinese by my family and could do the memorizing of many words. The thing that had me worried is that while I could read without pinyin in some parts, I couldn't in others. If I was given a children's storybook, I would read it aloud, knowing each lashes of the word. If I was given a difficult adult book, I'm not able to easily read it without the pinyin on the top.
Is there a way to solve this problem?
TL;DR:

You need a bigger vocabulary to increase reading proficiency. Based on your comments, it sounds like you know a lot of vocabulary (perhaps orally), but you have trouble recognizing those characters in writing. So, instead of a bigger vocabulary, you need to increase your character recognition. Point #3 below + Anki will help you do that.
Pronunciation and vocabulary size are correlated, so having good pronunciation and knowing the pronunciation of the words is important.
I wrote a Python script that calculates word frequencies in
a given Chinese text that may be useful in pre-learning the vocabulary for a reading project. YMMV. Or, search online for a Chinese word frequency counter like this one.

Like L Parker mentioned in his comment, there are many reasons why you have difficulty reading a &quot;difficult adult book&quot;. Is that adult book fiction or nonfiction? If it's nonfiction, does it contain a lot of jargon from a specific field? How familiar are you with the content?
In general, one popular area of research into reading comprehension is that of lexical coverage and lexical thresholds — i.e. what percentage of the vocabulary of a given text do you need to know in order to be able to comprehend it? Laufer (2013) seems like a relatively recent, short, and decent introduction to this problem:

&quot;Most researchers agree that only when readers possess a critical mass of second language (L2) knowledge (lexical and grammatical) can general reading skills, such as distinguishing between main and peripheral information, ... operate most efficiently ... vocabulary knowledge is a good predictor of reading proficiency, if not the best&quot; (Laufer, 2013, p. 868, emphasis my own).

In other words, vocabulary size is key. Laufer (2013) summarizes research which suggests a lexical threshold of 95% to 98% coverage as optimal, but there are many factors that would determine what threshold you'd need for your own situation.
Since you mention pinyin, here's a side note about pronunciation:
Learners with unstable pronunciation have trouble putting words into working memory (Ellis and Beaton, 1993  (PDF)), and Papagno, Valentine, &amp; Baddeley (1991) showed that verbal working memory and vocabulary size are correlated. So, pronunciation is critical for vocabulary in the long term, and Chinese has the problem of not easily being able to derive pronunciation from the written form.
However, I don't have any citations about pinyin and reading in Chinese, but I imagine that learning the pronunciation of characters in context through ruby pinyin may not be recommended as a learning strategy, because you may find that you associate the sound and meaning more with the pinyin than with the character itself.
As an L2 learner of Chinese myself, I created a Python script to count all the Chinese words (not characters) in a text and sort them according to frequency. It also takes a stop list of words to remove the words you already know. It's not very flexible, but if you know how to run a Python script, it may be helpful to you. It helped me when I decided to read a translation of 1984.
There are probably also user-friendly online solutions you could put your text into, like this one I found with a Duckduckgo search.
Hope that helps!